Remove the two directories /usr/local/cuda and /Library/Frameworks/CUDA.framework.
If you just want to uninstall the toolkit, remove those two directories and reinstall the driver.
If you just want to uninstall the toolkit, remove those two directories and reinstall the driver.