Bits of Experience
In this page I'll share some tips and tricks regarding various, e.g., tools, that I use daily.
Tensorboard
While there are other tools for experiment tracking (e.g., Weights & Biases, MLFlow, Neptune, ...), I still use Tensorboard for tracking my ML experiments. It's great for tracking the evolution of your models over steps/epochs. Unfortunately, there are some caveats:
I can't see every logged image/audio/sample
This can be fixed by using the --samples_per_plugin option.
tensorboard --logdir={$DIRECTORY} --samples_per_plugin='images=200, audio=200, scalars=10000'Some hyperparameters are not visible in the HPARAMS table
It can happen especially if you're using a single Tensorboard server to show experiments with varying number of hyperparameters. See the open issue. Unfortunately this requires a fix via Tensorboard. Until then, you should be mindful not mixing experiment logs of models with a huge difference in their logged hyperparameters.
I can't access Tensorboard from another computer
Without the --bind_all option, requests coming outside of the allowed IP range (default should be localhost) will be ignored. You would also need to open a port default: 6006 in the firewall.
Server Management
NVIDIA drivers got updated... again...
Normally, one has to reboot the server after a driver update. If you don't want to reboot, you can try to restart the NVIDIA driver without rebooting. This can be done by running the following commands:
sudo systemctl stop gdm # alternatively: sudo systemctl isolate multi-user.target
sudo systemctl stop nvidia-persistenced
sudo fuser -k /dev/nvidia* # careful: kills ALL processes using the GPU
sudo modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia
lsmod | grep -i nvidia # optional verification: should return nothing
sudo modprobe nvidia_drm modeset=1
sudo systemctl start nvidia-persistenced