Recently, I encountered a problem. When I redeployed a service in docker, I failed to start the service after executing the SH start.sh command or docker compose up – D command, and reported a device or resource busy error.

Then I docker PS a look, good guy, not only failed to start, the service is not available yet
this is because when docker is deployed, all the original things will be RM repeated and then rebuilt. Well, the deletion was successful, but the deployment was not started successfully.

Take a look at the error reported at that time:

failed to remove root filesystem for 9b13c255d03757c456651506ada5566a5b5e5a87b851e3621dcejf823: remove /var/lib/docker/overlay/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d/merged: device or resource busy

Device or resource busy, which means that the device or resource is busy and you can’t delete it.

Then I went to the/var/lib/docker/overlay/9b13c255d03757c756651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d/merged directory. It was empty and could not be deleted manually.

terms of settlement

1. Find the processes occupying this directory

Command: grep ID/proc/*/mounts
examples are as follows:
grep 9b13c255d0375757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d/proc/*/mounts

After executing the command, it is obvious that two processes are using it:

/proc/25086/mounts:overlay /data0/docker/overlay2/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d /merged overlayrw,relatime,lowerdir=/data0/docker/overlay2/l/F4BQHNVPHGRKWXFRUIHXUMAHZ2:/data0/docker/overlay2/l/.....
/proc/25065/mounts:overlay /data0/docker/overlay2/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d /merged overlayrw,relatime,lowerdir=/data0/docker/overlay2/l/......

2. Kill the process

Command: Kill – 9 process ID
examples are as follows:
kill – 9 25086
kill – 9 25065

Note: it is important to see whether some processes can be deleted directly. If they are deleted all at once, it will be bad in case other services are affected.

3. Continue to delete the directory. The operation is successful

Command: RM – RF error file path
example:
RM – RF/var/lib/docker/overlay/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d/merged

After deleting the directory file, re execute the SH start.sh or docker compose up – D command to start the service, and the start is successful!