updated the nv driver to 450 last week, and then found that nvidia-docker could not start, error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown.

USES the command:

$ nvidia-container-cli -k -d /dev/tty info

also gives an error:

-- WARNING, the following logs are for debugging purposes only --

I1001 02:01:39.488895 21811 nvc.c:281] initializing library context (version=1.0.7, build=b71f87c04b8eca8a16bf60995506c35c937347d9)
I1001 02:01:39.488944 21811 nvc.c:255] using root /
I1001 02:01:39.488952 21811 nvc.c:256] using ldcache /etc/ld.so.cache
I1001 02:01:39.488959 21811 nvc.c:257] using unprivileged user 1000:1000
W1001 02:01:39.490269 21812 nvc.c:186] failed to set inheritable capabilities
W1001 02:01:39.490334 21812 nvc.c:187] skipping kernel modules load due to failure
I1001 02:01:39.490648 21813 driver.c:133] starting driver service
E1001 02:01:39.490932 21813 driver.c:197] could not start driver service: load library failed: libcuda.so.1: cannot open shared object file: no such file or directory
I1001 02:01:39.491071 21811 driver.c:233] driver service terminated successfully
nvidia-container-cli: initialization error: driver error: failed to process request

so the corresponding version of cuda1

is upgraded

$ apt install libcuda1-450

and then back to normal