I have some K3s nodes for my pet projects using cheap VMs that come with 20 GB drives. As you know with containers and other stuff your VMs disk space can be depleted quickly. K3s has many proper default configurations but I’d suggest you the following:
- Edit /etc/systemd/journald.conf and set SystemMaxUse=200M in order to restrict the maximum disk space taken by the log files. I had more than 2 GB in journals after many months.
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.
[Journal]
SystemMaxUse=200M
Then restart journal service:
sudo systemctl restart systemd-journald
- Reduce thresholds for K8s Kubelet garbage collector. As suggested in https://github.com/k3s-io/k3s/issues/1900 I’ve reduced image-gc-high-threshold and image-gc-low-threshold.
According to https://kubernetes.io/docs/concepts/architecture/garbage-collection/#container-image-lifecycle:
Disk usage above the configured HighThresholdPercent value triggers garbage collection, which deletes images in order based on the last time they were used, starting with the oldest first. The kubelet deletes images until disk usage reaches the LowThresholdPercent value.
So I’ve edited my systemd k3s service to start with those thresholds:
sudo systemctl edit --full k3s.service
ExecStart=/usr/local/bin/k3s server --kubelet-arg=image-gc-high-threshold=50 --kubelet-arg=image-gc-low-threshold=45
And I’ve restarted that service
sudo systemctl restart k3s.service
That’s it!