DevOps Garfield

Reference: k0sctl installation

1. Required prep - Ensure unique system ID. K0s cluster deployment will fail if aren't.

   sudo systemd-machine-id-setup

2. Disable SELinux because k0sctl doesn't seem to like it

   sudo setenforce 0

3. System prep ( RPM-based distros )
   1. Download the k0sctl binary:
      1. URL: https://github.com/k0sproject/k0sctl/releases
   2. Download the k0s binary:
      1. URL: https://github.com/k0sproject/k0s/releases
4. Copy the binaries into /usr/local/bin and set execute permissions

   sudo cp k0sctl* /usr/local/bin/k0sctl
   sudo chmod u+x /usr/local/bin/k0sctl
   sudo cp k0s-v* /usr/local/bin/k0s
   sudo chmod u+x /usr/local/bin/k0s

1. Login as root as required by the commands we're using

   sudo su -

2. Create the directory

   mkdir -p /etc/k0s

3. Copy or generate the configuration file
   1. If you already have a configuration file, copy it into the current directory
   2. If not, copy the following content and update for the IPs or hostnames of your nodes:

      apiVersion: k0sctl.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s-cluster
        user: admin
      spec:
        hosts:
        - ssh:
            address: 10.0.0.1  # Update this entry to specify hostname or IP address
            user: root
            port: 22
            keyPath: null      # Update this entry to specify the key path
          role: controller
        - ssh:
            address: 10.0.0.2  # Update this entry to specify hostname or IP address
            user: root
            port: 22
            keyPath: null      # Update this entry to specify the key path
          role: worker
        options:
          wait:
            enabled: true
          drain:
            enabled: true
            gracePeriod: 2m0s
            timeout: 5m0s
            force: true
            ignoreDaemonSets: true
            deleteEmptyDirData: true
            podSelector: ""
            skipWaitForDeleteTimeout: 0s
          concurrency:
            limit: 30
            uploads: 5
          evictTaint:
            enabled: false
            taint: k0sctl.k0sproject.io/evict=true
            effect: NoExecute
            controllerWorkers: false

4. Generate or copy your ssh keys to each target. If you're not sure how, have a look at this site: https://www.ssh.com/academy/ssh/keygen
5. Open the firewall ports on the control plane node(s)

   # Open Kubernetes API server port
   firewall-cmd --permanent --add-port=6443/tcp
   
   # Open Kubelet API port
   firewall-cmd --permanent --add-port=10250/tcp
   
   # Open default k0s internal networking ports (BGP and VXLAN)
   firewall-cmd --permanent --add-port=179/tcp
   firewall-cmd --permanent --add-port=4789/udp
   
   # Reload firewall to apply changes
   firewall-cmd --reload

6. Open the firewall ports on the worker node(s)

   # Open Kubelet API port and networking tunnels
   firewall-cmd --permanent --add-port=10250/tcp
   firewall-cmd --permanent --add-port=179/tcp
   firewall-cmd --permanent --add-port=4789/udp
   firewall-cmd --permanent --add-port=8132-8133/tcp
   
   # Reload firewall to apply changes
   firewall-cmd --reload

7. Create the cluster

   k0sctl apply --config /etc/k0s/k0sctl.yaml

8. If you want to use kubectl, lens, or other tools outside the cluster nodes, then you will need a “kubeconfig” file. You can generate the file with the following command

   k0s kubeconfig > ~/.kubeconfig

9. Don't forget to set the proper permission. This file contains credentials ( certs ) that allow access without entering any information

   chmod 600 ~/.kubeconfig

10. Find the ports that need to be open

    grep -i port /etc/k0s/k0s.yaml

1. Watch the cluster deployment as they start ( to quit, press CTRL+C )

   watch -n1 'k0s kubectl get all -A; echo; k0s kubectl get node'

Note that you should always have an odd number of controll nodes. Therefore, you should have 1 or 3 or 5 control nodes, depending on the cluster size. For a non-production environment, start wtih 1. For production environments, start with 3. Monitor the kube-apiserver performance. If it starts to show hi CPU utilization, increase the number of CPUs on the controll node(s).

1. On the existing controller node
   1. Create the controller node token

      k0s token create --role=controller --expiry=1h > token-file

   2. Transfer the new token file to each worker node

      scp token-file user@<new-controller>:~/

   3. Transfer the k0s.yaml config file to the new controller node(s)

      scp k0s.yaml user@<new-controller>:~/

1. Check the status of the cluster

   k0s status
   
   Version: v1.32.4+k0s.0
   Process ID: 109946
   Role: controller
   Workloads: false
   SingleNode: false

1. Check the status of the cluster

   k0s status
   
   Version: v1.32.4+k0s.0
   Process ID: 91481
   Role: worker
   Workloads: true
   SingleNode: false
   Kube-api probing successful: true
   Kube-api probing last error:  

1. Run the following command:

   kubectl get nodes

It should look something like this

[garfield@k8s01 ~]$ kubectl get nodes
NAME                       STATUS   ROLES           AGE   VERSION
k8s01.home.mygarfield.us   Ready    control-plane   17h   v1.31.1
k8s02.home.mygarfield.us   Ready    <none>          10m   v1.31.1
k8s03.home.mygarfield.us   Ready    <none>          6s    v1.31.1