{{tag>projects Linux Kubernetes k8s kubeadm installation setup}}

[[kubernetes|What is Kubernetes ( k8s )?]]

{{ :projects:k0sctl-logo.jpg:?600 }}

==== Preperation process - All nodes ====

Reference: [[https://docs.k0sproject.io/stable/k0sctl-install/|k0sctl installation]]

  - Required prep - Ensure unique system ID. K0s cluster deployment will fail if aren't.<code>
sudo systemd-machine-id-setup
</code>
  - Disable SELinux because k0sctl doesn't seem to like it<code>
sudo setenforce 0
</code>
  - System prep ( RPM-based distros )
    - Download the k0sctl binary:
      - URL: [[https://github.com/k0sproject/k0sctl/releases|https://github.com/k0sproject/k0sctl/releases]]
    - Download the k0s binary:
      - URL: [[https://github.com/k0sproject/k0sctl/releases|https://github.com/k0sproject/k0s/releases]]
  - Copy the binaries into /usr/local/bin and set execute permissions<code>
sudo cp k0sctl* /usr/local/bin/k0sctl
sudo chmod u+x /usr/local/bin/k0sctl
sudo cp k0s-v* /usr/local/bin/k0s
sudo chmod u+x /usr/local/bin/k0s
</code>

==== Initialize cluster - Control nodes only ====
  - Login as root as required by the commands we're using<code>
sudo su -
</code>
  - Create the directory<code>
mkdir -p /etc/k0s
</code>
  - Copy or generate the configuration file
    - If you already have a configuration file, copy it into the current directory
    - If not, copy the following content and update for the IPs or hostnames of your nodes:<code>
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
  user: admin
spec:
  hosts:
  - ssh:
      address: 10.0.0.1  # Update this entry to specify hostname or IP address
      user: root
      port: 22
      keyPath: null      # Update this entry to specify the key path
    role: controller
  - ssh:
      address: 10.0.0.2  # Update this entry to specify hostname or IP address
      user: root
      port: 22
      keyPath: null      # Update this entry to specify the key path
    role: worker
  options:
    wait:
      enabled: true
    drain:
      enabled: true
      gracePeriod: 2m0s
      timeout: 5m0s
      force: true
      ignoreDaemonSets: true
      deleteEmptyDirData: true
      podSelector: ""
      skipWaitForDeleteTimeout: 0s
    concurrency:
      limit: 30
      uploads: 5
    evictTaint:
      enabled: false
      taint: k0sctl.k0sproject.io/evict=true
      effect: NoExecute
      controllerWorkers: false
</code>
  - Generate or copy your ssh keys to each target. If you're not sure how, have a look at this site: [[https://www.ssh.com/academy/ssh/keygen]]
  - Open the firewall ports on the control plane node(s)<code>
# Open Kubernetes API server port
firewall-cmd --permanent --add-port=6443/tcp

# Open Kubelet API port
firewall-cmd --permanent --add-port=10250/tcp

# Open default k0s internal networking ports (BGP and VXLAN)
firewall-cmd --permanent --add-port=179/tcp
firewall-cmd --permanent --add-port=4789/udp

# Tell Fedora to trust all traffic on the container network bridges
firewall-cmd --permanent --zone=trusted --add-interface=cni0
firewall-cmd --permanent --zone=trusted --add-interface=kube-router

# Allow pods to masquerade/NAT out to the local network
firewall-cmd --permanent --zone=trusted --add-masquerade

firewall-cmd --permanent --zone=trusted --add-source=10.244.0.0/16
firewall-cmd --permanent --zone=trusted --add-source=10.96.0.0/12

# Reload firewall to apply changes
firewall-cmd --reload
</code>
  - Disable the firewall completely **on the worker nodes** since k8s doesn't like it - Ref: [[https://docs.k0sproject.io/head/networking/#firewalld-k0s|k0s networking - firewalld-k0s]]<code>
sudo systemctl stop firewalld && sudo disable firewalld
</code>
  - Create the cluster<code>
k0sctl apply --config /etc/k0s/k0sctl.yaml
</code>
  - If you want to use kubectl, lens, or other tools outside the cluster nodes, then you will need a "kubeconfig" file. You can generate the file with the following command<code>
k0s kubeconfig > ~/.kubeconfig
</code>
  - Don't forget to set the proper permission. This file contains credentials ( certs ) that allow access without entering any information<code>
chmod 600 ~/.kubeconfig
</code>
  - Find the ports that need to be open<code>
grep -i port /etc/k0s/k0s.yaml
</code>


==== Verify the cluster setup/deployment - Control node ====
  - Watch the cluster deployment as they start ( to quit, press CTRL+C )<code>
watch -n1 'k0s kubectl get all -A; echo; k0s kubectl get node'
</code>

==== Add additional controller nodes to the cluster ====
Note that you should always have an odd number of controll nodes. Therefore, you should have 1 or 3 or 5 control nodes, depending on the cluster size. For a non-production environment, start wtih 1. For production environments, start with 3. Monitor the kube-apiserver performance. If it starts to show hi CPU utilization, increase the number of CPUs on the controll node(s).
  - On the existing controller node
    - Create the controller node token<code>
k0s token create --role=controller --expiry=1h > token-file
</code>
    - Transfer the new token file to each worker node<code>
scp token-file user@<new-controller>:~/
</code>
    - Transfer the k0s.yaml config file to the new controller node(s)<code>
scp k0s.yaml user@<new-controller>:~/
</code>

==== Check the status of the cluster - Control node output ====
  - Check the status of the cluster<code>
k0s status

Version: v1.32.4+k0s.0
Process ID: 109946
Role: controller
Workloads: false
SingleNode: false
</code>

==== Check the status of the cluster - Worker node output ====
  - Check the status of the cluster<code>
k0s status

Version: v1.32.4+k0s.0
Process ID: 91481
Role: worker
Workloads: true
SingleNode: false
Kube-api probing successful: true
Kube-api probing last error:  
</code>

==== User setup process ====


==== Pod network setup process ====


==== Highly Available ( multi-node control plane ) cluster setup process ====


==== Add worker nodes to the cluster setup ====


==== Verify worker nodes have successfully joined the cluster ====
  - Run the following command: <code>
kubectl get nodes
</code>

It should look something like this <code>
[garfield@k8s01 ~]$ kubectl get nodes
NAME                       STATUS   ROLES           AGE   VERSION
k8s01.home.mygarfield.us   Ready    control-plane   17h   v1.31.1
k8s02.home.mygarfield.us   Ready    <none>          10m   v1.31.1
k8s03.home.mygarfield.us   Ready    <none>          6s    v1.31.1
</code>