Rancher: Node disk is running full within 24 hours

 

Problem: Node disk is running full within 24 hours

Cause: predict_linear(node_filesystem_free_bytes{mountpoint!~"^/etc/(?:resolv.conf|hosts|hostname)$"}[6h], 3600 * 24) < 0

Solution:

https://forums.rancher.com/t/rancher-monitoring-is-falsely-alerting-on-var-lib-lxcfs-running-out-of-disk-space/14513

predict_linear(node_filesystem_files_free{mountpoint!~"^/(?:etc/resolv.conf|etc/hosts|etc/hostname|var/lib/lxcfs)$"}[6h], 3600 * 24)