Kubernetes The Hard Way Part 3

kubernetes.

This is the continue blog from kubernetes the hard way part 2. in this third part i will write on preparation before bootstrapping the kubernetes instances component (etcd, worker, controller, etc). this will cover the 05-kubernetes-configuration-file, 06-data-encrpytion-key and 07-bootstrapping-etcd

Generating Kubernetes Configuration Files for Authentication

There will be 3 main kubectl commands that will be used to set the required kubeconfig file.

# sets up the cluster configuration in the kubeconfig file.
kubectl config set-cluster

# defines the authentication details for a specific user or service account
kubectl config set-credentials

# defines the connection details for a specific user or service account to a specific cluster
kubectl config set-context

# sets the default "connection" for kubectl to use when running commands.
kubectl config use-context

example .kubeconfig file

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: xxx
      server: https://:443
    name: kubernetes-the-hard-way
contexts:
  - context:
      cluster: kubernetes-the-hard-way
      user: system:kube-proxy
    name: default
current-context: default
kind: Config
preferences: {}
users:
  - name: system:kube-proxy
    user:
      client-certificate-data: xxx
      client-key-data: xxx

Client Authentication Configs

First section, i will try to orchestrate a required kubeconfig for the kubernetes cluster. such as:

1. controller manager This component use to runs control plane component, such as Node Controller, Replication Controller & Service Controller
2. kubelet This is an agent that runs on each node in the cluster, responsible for managing node resources such as pod to then communication with control plane
3. kube-proxy This component runs on each node and provides network proxying and load balancing for services in the cluster.
4. scheduler This component is responsible for scheduling pods on nodes in the cluster. It takes into account factors like resource availability, node affinity, and pod priority when making scheduling decisions.

Retrieve the DNS of load-balancer

Before setting the required kubeconfig, we need to retrieve the DNS from load-balancer that already created before. we can see in the load-balancer page or using this command

# Retrieve DNS
KUBERNETES_PUBLIC_ADDRESS=$(aws elbv2 describe-load-balancers \
  --load-balancer-arns ${LOAD_BALANCER_ARN} \
  --output text --query 'LoadBalancers[0].DNSName')

Kubelet Configuration

First is to set the kubelet using kubectl. as explaine before, kubelet are agents that run on each node in the cluster. They are responsible for managing containers and communicating with the control plane. Since now i have 3 worker, so using this script can help to generate for all 3 worker

for instance in worker-0 worker-1 worker-2; do
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://${KUBERNETES_PUBLIC_ADDRESS}:443 \
    --kubeconfig=${instance}.kubeconfig

  kubectl config set-credentials system:node:${instance} \
    --client-certificate=${instance}.pem \
    --client-key=${instance}-key.pem \
    --embed-certs=true \
    --kubeconfig=${instance}.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:node:${instance} \
    --kubeconfig=${instance}.kubeconfig

  kubectl config use-context default --kubeconfig=${instance}.kubeconfig
done

Kube-proxy Configuration

kube-proxy maintains network rules on each node, enabling communication between pods and services. this script help to generate required kubeconfig for the kube-proxy

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${KUBERNETES_PUBLIC_ADDRESS}:443 \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials system:kube-proxy \
  --client-certificate=kube-proxy.pem \
  --client-key=kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

This will generate kube-proxy.kubeconfig

Kube-controller Configuration

Next is kube-controller. The kube-controller-manager is responsible for running controllers that maintain the desired cluster state (e.g., managing deployments, replicasets).

# server will be using 127.0.0.1:6443 because the controller-manager connects to the API server running on the same machine (localhost).
kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig

This will generate kube-controller-manager.kubeconfig

Kube-scheduler Configuration

The kube-scheduler assigns pods to nodes based on resource availability and other constraints.

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
  --client-certificate=kube-scheduler.pem \
  --client-key=kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-scheduler \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig

This will generate kube-scheduler.kubeconfig

Admin Configuration

This kubeconfig is for cluster administrators, granting them full access to the Kubernetes API.

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=admin.kubeconfig

kubectl config set-credentials admin \
  --client-certificate=admin.pem \
  --client-key=admin-key.pem \
  --embed-certs=true \
  --kubeconfig=admin.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=admin \
  --kubeconfig=admin.kubeconfig

kubectl config use-context default --kubeconfig=admin.kubeconfig

This will generate admin.kubeconfig

Distribute the Kubernetes Configuration Files

Now when the kubeconfig is ready, time to move it to the aws instance worker & controller. this script will help for scp process

Worker Instance

for instance in worker-0 worker-1 worker-2; do
  external_ip=$(aws ec2 describe-instances --filters \
    "Name=tag:Name,Values=${instance}" \
    "Name=instance-state-name,Values=running" \
    --output text --query 'Reservations[].Instances[].PublicIpAddress')

  scp -i kubernetes.id_rsa \
    ${instance}.kubeconfig kube-proxy.kubeconfig ubuntu@${external_ip}:~/
done

Controller Instance

for instance in controller-0 controller-1 controller-2; do
  external_ip=$(aws ec2 describe-instances --filters \
    "Name=tag:Name,Values=${instance}" \
    "Name=instance-state-name,Values=running" \
    --output text --query 'Reservations[].Instances[].PublicIpAddress')

  scp -i kubernetes.id_rsa \
    admin.kubeconfig kube-controller-manager.kubeconfig kube-scheduler.kubeconfig ubuntu@${external_ip}:~/
done

Generating the Data Encryption Config and Key

Now since we already moved all required kubeconfig and it cert, how to protect it? we will now perform encryption process for encrypting Kubernetes Secrets. This is a critical security practice to protect sensitive data stored within the cluster.

Generate encryption key

This command generates a random 32-byte key using /dev/urandom (a source of strong random data) and then encodes it using Base64. Base64 encoding makes the key suitable for storing in text files like the configuration file.

ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

Now create new file encryption-config.yaml for the encryption config file

kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}

Copy the file to each controller instance

for instance in controller-0 controller-1 controller-2; do
external_ip=$(aws ec2 describe-instances --filters \
    "Name=tag:Name,Values=${instance}" \
 "Name=instance-state-name,Values=running" \
 --output text --query 'Reservations[].Instances[].PublicIpAddress')

scp -i kubernetes.id_rsa encryption-config.yaml ubuntu@${external_ip}:~/
done

Bootstrapping ETCD Cluster

What is a ETCD Cluster

etcd is a distributed key-value store. Think of it as a reliable database that can be shared across multiple machines. It's designed to be highly available, fault-tolerant, and consistent.

Why ETCD is crucial for kubernetes cluster

Kubernetes uses etcd as its primary data store. Essentially, everything Kubernetes needs to know about the state of cluster is stored in etcd. This includes:

1. Cluster State: Which nodes are part of the cluster, their health status, available resources, etc.
2. Application Configurations: Deployments, Services, Pods – all the information defining how your applications should run.
3. Secrets: Sensitive information like passwords, API keys, and certificates.
4. Networking Information: Network policies, service endpoints, and other network configurations.

Without etcd, Kubernetes would have no way to maintain a consistent view of the cluster. It wouldn't know what applications are running, where they're running, or how to manage them.

Login to controller instance

Now we need to retrieve the external_ip to be allowed ssh into the instance. this script help to obtain the 3 instance controller public_ip

for instance in controller-0 controller-1 controller-2; do
  external_ip=$(aws ec2 describe-instances --filters \
    "Name=tag:Name,Values=${instance}" \
    "Name=instance-state-name,Values=running" \
    --output text --query 'Reservations[].Instances[].PublicIpAddress')

  echo ssh -i kubernetes.id_rsa ubuntu@$external_ip
done

Bootstrapping an etcd Cluster Member

Download and Install the etcd Binaries

As per Oct 2024, the LTS of ETCD binaries is 3.5.16

# Download the binary
wget -q --show-progress --https-only --timestamping \
  "https://github.com/etcd-io/etcd/releases/download/v3.5.16/etcd-v3.5.16-linux-amd64.tar.gz"

# Extract & Install
tar -xvf etcd-v3.5.16-linux-amd64.tar.gz
sudo mv etcd-v3.5.16-linux-amd64/etcd* /usr/local/bin/

Configure ETCD server

## Creates necessary directories for etcd data and configuration.
sudo mkdir -p /etc/etcd /var/lib/etcd

## owner of the file or directory full control
sudo chmod 700 /var/lib/etcd

## Copies the TLS certificates and key to the etcd configuration directory.
sudo cp ca.pem kubernetes-key.pem kubernetes.pem /etc/etcd/

After that, we need to retrieve internal IP for each of the controller, we can use this command:

INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)

This command uses curl to query the AWS Instance Metadata Service for the instance's private IPv4 address and stores that address in the INTERNAL_IP variable. This is a standard way to obtain the internal IP address of an EC2 instance programmatically.

Also the IP address 169.254.169.254 is a special address reserved for link-local addressing. Specifically, it's used by the AWS Instance Metadata Service (IMDS).

Each etcd member must have a unique name within an etcd cluster. Set the etcd name to match the hostname of the current compute instance:

ETCD_NAME=$(curl -s http://169.254.169.254/latest/user-data/ \
  | tr "|" "\n" | grep "^name" | cut -d"=" -f2)
echo "${ETCD_NAME}"

And for etcd systemd, need also to be created.

cat <<EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --cert-file=/etc/etcd/kubernetes.pem \\
  --key-file=/etc/etcd/kubernetes-key.pem \\
  --peer-cert-file=/etc/etcd/kubernetes.pem \\
  --peer-key-file=/etc/etcd/kubernetes-key.pem \\
  --trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\
  --advertise-client-urls https://${INTERNAL_IP}:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster controller-0=https://10.0.1.10:2380,controller-1=https://10.0.1.11:2380,controller-2=https://10.0.1.12:2380 \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

This script creates a systemd unit file for the etcd service. Systemd is a system and service manager for Linux; unit files define how services should be started, stopped, and managed. This particular script configures etcd to run as a service, enabling it to start automatically on boot and restart if it fails.

Start & Verification

We can confirm the installation is succeed or not by start & verify it

Start ETCD Server

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd

Verify ETCD Server

sudo ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.pem \
  --cert=/etc/etcd/kubernetes.pem \
  --key=/etc/etcd/kubernetes-key.pem

Output should be similiar like this

bbeedf10f5bbaa0c, started, controller-2, https://10.0.1.12:2380, https://10.0.1.12:2379, false
f9b0e395cb8278dc, started, controller-0, https://10.0.1.10:2380, https://10.0.1.10:2379, false
eecdfcb7e79fc5dd, started, controller-1, https://10.0.1.11:2380, https://10.0.1.11:2379, false