標籤: Docker

前幾天在做 aptitude upgrade 時，系統噴儲存空間不夠的錯誤訊息，用 df 追了一下，發現是 inode 被吃光，網路上找了一下統計 inode 使用量的 script 來掃整個分割區：

find / -type d -print0 | xargs -0 -n1 count_files | sort -n

...
1794 ./var/lib/docker/aufs/diff/04d42c6fb6b72464ab397cc0cd67d8600f7bc0964ff7c9bb54392ec3eb53a13e/home/git/gitlab/app/assets/images/emoji
1794 ./var/lib/docker/aufs/diff/04d42c6fb6b72464ab397cc0cd67d8600f7bc0964ff7c9bb54392ec3eb53a13e/home/git/gitlab/public/assets/emoji
1794 ./var/lib/docker/aufs/diff/04d42c6fb6b72464ab397cc0cd67d8600f7bc0964ff7c9bb54392ec3eb53a13e/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/gemojione-3.0.1/assets/png
1794 ./var/lib/docker/aufs/diff/04d42c6fb6b72464ab397cc0cd67d8600f7bc0964ff7c9bb54392ec3eb53a13e/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/gemojione-3.0.1/assets/svg
1794 ./var/lib/docker/aufs/diff/bb44ab5db5ec7f5e3fe0bde806002e887cf11cf9aa598ce0453083f80fc10ff9/home/git/gitlab/app/assets/images/emoji
1794 ./var/lib/docker/aufs/diff/bb44ab5db5ec7f5e3fe0bde806002e887cf11cf9aa598ce0453083f80fc10ff9/home/git/gitlab/public/assets/emoji
1794 ./var/lib/docker/aufs/diff/bb44ab5db5ec7f5e3fe0bde806002e887cf11cf9aa598ce0453083f80fc10ff9/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/gemojione-3.0.1/assets/png
1794 ./var/lib/docker/aufs/diff/bb44ab5db5ec7f5e3fe0bde806002e887cf11cf9aa598ce0453083f80fc10ff9/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/gemojione-3.0.1/assets/svg
1794 ./var/lib/docker/aufs/diff/fcfbf2f22b171407b0a9d657131c0c4b55aadd50c49a26abfc40cd404ab02298/home/git/gitlab/app/assets/images/emoji
1794 ./var/lib/docker/aufs/diff/fcfbf2f22b171407b0a9d657131c0c4b55aadd50c49a26abfc40cd404ab02298/home/git/gitlab/public/assets/emoji
1794 ./var/lib/docker/aufs/diff/fcfbf2f22b171407b0a9d657131c0c4b55aadd50c49a26abfc40cd404ab02298/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/gemojione-3.0.1/assets/png
1794 ./var/lib/docker/aufs/diff/fcfbf2f22b171407b0a9d657131c0c4b55aadd50c49a26abfc40cd404ab02298/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/gemojione-3.0.1/assets/svg
1966 ./var/lib/docker/aufs/diff/04d42c6fb6b72464ab397cc0cd67d8600f7bc0964ff7c9bb54392ec3eb53a13e/home/git/gitlab/public/assets
1966 ./var/lib/docker/aufs/diff/bb44ab5db5ec7f5e3fe0bde806002e887cf11cf9aa598ce0453083f80fc10ff9/home/git/gitlab/public/assets
1971 ./var/lib/docker/aufs/diff/fcfbf2f22b171407b0a9d657131c0c4b55aadd50c49a26abfc40cd404ab02298/home/git/gitlab/public/assets
2271 ./var/lib/docker/image/aufs/distribution/diffid-by-digest/sha256
2271 ./var/lib/docker/image/aufs/distribution/v2metadata-by-diffid/sha256
3291 ./var/lib/dpkg/info
4025 ./var/lib/docker/aufs/diff/fcfbf2f22b171407b0a9d657131c0c4b55aadd50c49a26abfc40cd404ab02298/home/git/gitlab/tmp/cache/assets/sprockets/v3.0
4029 ./var/lib/docker/aufs/diff/04d42c6fb6b72464ab397cc0cd67d8600f7bc0964ff7c9bb54392ec3eb53a13e/home/git/gitlab/tmp/cache/assets/sprockets/v3.0
4029 ./var/lib/docker/aufs/diff/bb44ab5db5ec7f5e3fe0bde806002e887cf11cf9aa598ce0453083f80fc10ff9/home/git/gitlab/tmp/cache/assets/sprockets/v3.0

看起來兇手是 docker，而且吃掉的量還不小。

查了一下資料，其實是自己在刪除 images 和 volumes 時有漏參數，導致有檔案沒有被清乾淨，inode 被吃光沒有回收回來。參考這篇「Docker、AUFS、inode耗尽和一个小工具」整理，可以透過以下 script 把無用的檔案清空：

docker images -qf dangling=true | xargs docker rmi

docker volume ls -qf dangling=true |xargs docker volume rm

這二個 script 跑完以後，我的 inode usage 從 99% 瞬間降到 17%，看來以後真的要小心。

其他參考資料：

• 關於我常用的 Docker 小提示 – by toomore
• tripflex/inodes – 一個統計 inode 使用量並視覺化列表的小工具