删了文件磁盘未回收

查看磁盘空间已经爆满, 删除了几百G的日志文件, 通过du -sch或者du -d1 -h /data一步步查找都没有大文件占用了, 但是磁盘空间还是没有恢复.

 df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G   24K   16G   1% /dev/shm
tmpfs            16G  1.6G   14G  11% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/vda1       200G   23G  178G  12% /
tmpfs            16G  7.7M   16G   1% /tmp
tmpfs           3.1G     0  3.1G   0% /run/user/0
/dev/vdb        500G  500G   20K 100% /data

查看仍在占用已经删除文件的进程

[root@10 logs]# sudo lsof | grep -i "Deleted" | grep sql
java       152911                              root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152913 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152914 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152915 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152916 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152917 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152918 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152919 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
java       152911  152920 java                 root  307w      REG             253,16 276980159190  805315939 /data/emr/ranger/logs/ranger_admin_sql.log.2023-09-25 (deleted)
...

删除所有关联的进程

sudo lsof | grep -i "Deleted" | grep sql |  awk '{print $2}' | xargs kill -9 

磁盘空间恢复

[root@10 logs]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G   24K   16G   1% /dev/shm
tmpfs            16G  1.6G   14G  11% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/vda1       200G   23G  178G  12% /
tmpfs            16G  7.8M   16G   1% /tmp
tmpfs           3.1G     0  3.1G   0% /run/user/0
/dev/vdb        500G  145G  356G  29% /data

参考文档: https://serverfault.com/questions/232525/df-in-linux-not-showing-correct-free-space-after-file-removal

The file is still locked by the process opening it. To free up space, do these steps:

Run sudo lsof | grep deleted and see which process is holding the file. Example result:

$ sudo lsof | grep deleted
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
cron     1623 root    5u   REG   0,21        0 395919638 /tmp/tmpfPagTZ4 (deleted)

Kill the process using sudo kill -9 {PID}. In above sample, the PID is 1623.

$ sudo kill -9 1623

Run df to check if space is already freed up. If it's still full, maybe you need to wait a few seconds and check again.