System Environment

Server Hardware Requirements

The K3s Server runs the Kubernetes API, the scheduler, and other cluster management components. As the number of nodes and Pods grows, the CPU and memory usage of the Server grows as well.

Use the following configuration as a reference according to the cluster scale:

Node Configuration

The current environment uses one K3s Server and two K3s Agents:

Keep the following in mind when configuring the nodes:

• Docker versions may differ between nodes; for example, bmc and sub12 use 20.10.24+dfsg1 while sub11 uses 26.1.3.
• Use the same K3s version for the Server and the Agents; this environment uses v1.36.4+k3s1 on all nodes.
• Adjust the node IPs for your own network, and make sure the nodes can communicate with each other.

Check Node Networking

All nodes must use static IP addresses, and the nodes must be able to reach each other. K3s uses the node IP as the internal cluster address. If the address changes, the node may fail or may be unable to join the cluster.

Confirm that the address is statically configured

# Show the node IPv4 address
ip -4 addr

# Show the default route
ip route show default

• valid_lft forever after the address means the address is static. A countdown such as valid_lft 3512sec means the address comes from DHCP and must be changed to a static address first.
• A default route showing proto static means the gateway is also statically configured.

Sample output from a real run:

inet 172.16.100.176/16 brd 172.16.255.255 scope global MGMT
   valid_lft forever preferred_lft forever

default via 172.16.0.1 dev MGMT proto static metric 100

Once the address is confirmed to be static, test connectivity from the bmc node to the other nodes:

ping -c 4 172.16.100.177
ping -c 4 172.16.100.178

If the nodes cannot communicate, or the address is not static, configure it first as described in Network Settings.

Check Firewall Ports

Make sure the firewall and other security policies do not block the traffic K3s requires, and confirm that these ports are not used by other programs and have no conflicting port mapping rules.

K3s uses the following ports by default:

Check the System Time

The system time of all nodes should stay in sync.

A large time offset may cause TLS certificate validation failures and prevent nodes from joining the cluster. It also makes log timestamps from different nodes inconsistent, which complicates troubleshooting.

Check the current time:

date

If the time is not synchronized, configure it first as described in Time Manager.

Check the Kernel Configuration

After configuring networking, ports, and time, confirm that the Linux kernel meets the requirements of K3s. K3s relies on kernel capabilities such as namespaces, cgroups, networking, and file systems. These capabilities are determined by the kernel configuration and cannot be added by installing packages, so check them before installing K3s and rebuild the kernel if necessary.

K3s ships with a check-config subcommand that checks the kernel configuration, cgroups, iptables, and other requirements automatically, so you do not have to compare the items one by one. Download the k3s binary for your architecture (this environment uses arm64) and run the check:

curl -sfLO https://github.com/k3s-io/k3s/releases/download/v1.36.4%2Bk3s1/k3s-arm64
chmod +x k3s-arm64
sudo ./k3s-arm64 check-config

If K3s is already installed on the node, use the installed version instead:

sudo k3s check-config

The output lists the result of each check, with an overall conclusion on the last line:

System:
- /usr/sbin iptables v1.8.9 (legacy): ok
- swap: disabled
- routes: default CIDRs 10.42.0.0/16 or 10.43.0.0/16 already routed

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
...

Storage Drivers:
- "overlay":
  - CONFIG_OVERLAY_FS: enabled

STATUS: pass

STATUS: pass means the kernel meets the requirements of K3s. If it reports fail, adjust the kernel configuration according to the items marked missing, rebuild the kernel, and flash it again. Missing items under Optional Features are optional capabilities (for example, encrypted networking).

Docker

K3s uses its bundled containerd as the container runtime by default. This environment uses Docker instead, enabled with docker: true, so every K3s node must have Docker installed.

Installation

For the installation steps, see Docker Installation.

After the installation, run the following commands to confirm that Docker is installed correctly and that the service is running:

docker --version
sudo systemctl is-active docker

Expected output:

Docker version 20.10.24+dfsg1, build 297e128
active

Migrating the data directory

Firefly devices enable overlayroot by default. In that case, / consists of a read-only root file system (/root-ro) and a writable layer located in /userdata/rootfs_overlay, and only /userdata is a separate ext4 partition on the device.

Run the following command to confirm the current file system layout:

mount | grep overlayroot

Docker stores its data in /var/lib/docker by default, which is located in the writable layer of overlayfs. The Docker overlay2 storage driver cannot be created directly on top of overlayfs: startup may fail with failed to mount overlay: invalid argument, fall back to the deprecated devicemapper driver, or even fail to start.

Therefore, move the Docker data directory to the separate /userdata partition, for example /userdata/docker.

Create the data directory:

sudo mkdir -p /userdata/docker

Edit /etc/docker/daemon.json:

sudo tee /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "data-root": "/userdata/docker",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  },
  "registry-mirrors": [
    "https://docker.m.daocloud.io",
    "https://docker.1ms.run",
    "https://docker-0.unsee.tech"
  ],
  "insecure-registries": [
    "172.16.55.55:5000"
  ],
  "experimental": true
}
EOF

Key configuration items:

If you do not use a private registry, remove the insecure-registries entry.

If your network does not need Docker Hub mirrors, remove the registry-mirrors entry.

Restart and verify

Restart Docker after the configuration:

sudo systemctl restart docker
sudo docker info

Confirm the following in the output:

Docker Root Dir: /userdata/docker

If Docker Root Dir shows /userdata/docker, the Docker data directory configuration has taken effect.

Containerd

The system containerd stores its data in /var/lib/containerd by default, which is also on overlayfs. The containerd overlay snapshotter has the same limitation as the Docker overlay2 driver and cannot be created on top of overlayfs, so its data directory must also be placed on the ext4 partition /userdata.

K3s ships with its own containerd (embedded in the k3s process, independent of the system containerd). The system containerd only supports Docker, so its CRI plugin must be disabled to avoid two CRI implementations conflicting with each other.

Modify the configuration

Create the directories:

sudo mkdir -p /etc/containerd
sudo mkdir -p /userdata/containerd

Edit /etc/containerd/config.toml:

sudo tee /etc/containerd/config.toml >/dev/null <<'EOF'
disabled_plugins = ["cri"]

root = "/userdata/containerd"
EOF

Key configuration items:

Restart and verify

Restart containerd after the configuration:

sudo systemctl restart containerd
sudo systemctl is-active containerd

Confirm the following in the output:

active

Restrict the LightDM Realtime Scheduling Permission

If the device also runs a graphical desktop environment, LightDM may use a high realtime scheduling priority.

In some environments this prevents K3s from setting up CPU cgroups when creating containers, and Pods may stay in the CreateContainerError state.

Only apply the configuration below if you actually run into this problem after installing K3s.

Edit the LightDM service:

sudo systemctl edit lightdm.service

Add:

[Service]
RestrictRealtime=yes
CPUSchedulingPolicy=other
CPUSchedulingPriority=0

Save the file, then reload the configuration and restart LightDM:

sudo systemctl daemon-reload
sudo systemctl restart lightdm.service

Check the K3s Pod status again after the restart.

If LightDM is not installed, or if K3s runs normally, this configuration is not required.