A small Bash tool that SSHes into a list of machines, runs
nvidia-smi (and a bit of ps/who), and tells you which machines are in
use and by which users.
Also includes a dependency-free web GUI.
If you are sharing GPU resources with others across multiple machines without a scheduling system, you often need to figure out which machines are free, and who's using them. This gives an insightful, easily readable overview of that.
- Local: bash,ssh, and standard coreutils.
- Remote:
- SSH access (key-based auth is recommended so there are no password prompts)
- nvidia-smi(Hosts without- nvidia-smican still report logged-in users and top CPU processes.)
Help wanted: The macOS
.commandhas not yet been verified on real hardware. If you try one, please report back (or open a PR) so this note can be updated.
git clone https://github.com/lucafossen/who-gpu.git
cd who-gpu
./install.sh # installs the `who-gpu` command
./install.sh --icon # also adds a desktop iconUninstall with ./uninstall.sh (add --purge to also drop your preferences).
You could also skip the installer entirely and run ./who-gpu.sh directly.
Your install options are saved at ~/.config/who-gpu/config in plain text, and is safe to edit. Re-running
the installer lets you choose again.
who-gpu --updateNew versions are mentioned once a day, in the terminal and as a dashboard badge. Disable by setting UPDATE_CHECK=0.
who-gpu never updates itself.
Run the guided setup (also runs on first-time launch):
who-gpu --setupIt asks what a plain who-gpu should open, then scans your ~/.ssh/config,
shows every host with its current
probe state, and lets you toggle each one on or off. Flipping a host on adds
a #probe marker, flipping it off removes one. Your config is backed up
(timestamped) before any change.
who-gpu --webOpens a dashboard in your browser straight away and fills it in as each machine
answers, then keeps running until you Ctrl-C. Click a machine for full output (either the who-gpu --full view or plain nvidia-smi).
List view in the top bar swaps the cards for one row per machine under a column header:
Also featured: Grouping, sorting and all your choices remembered by the browser.
The Web GUI is serverless and dependency-free: the webpage and data is just a file on disk that the CLI tool rewrites, so it works everywhere the CLI does.
Files live in ~/.local/share/who-gpu/ (or $XDG_DATA_HOME/who-gpu/) and
stay there after you quit, so you can reopen the last probe (clearly marked
stale).
Only one dashboard runs per output directory: a second --web (or a second
click on the desktop icon) exits with a message pointing at the one already
running, instead of two engines fighting over the same page.
--web reuses one SSH connection per host (ControlMaster) instead of logging
in on every refresh. Where that isn't supported (notably Git Bash) every
refresh is a full login, so who-gpu refreshes less often (default every 60s) to go easy on the
hosts. You can also manually set an interval (e.g. WHO_GPU_INTERVAL=30 as an environment variable,
or INTERVAL=30 in ~/.config/who-gpu/config).
At startup, or by running who-gpu --setup, you will be prompted to choose which machines to probe.
To track this, who-gpu will add #probe comments to tag your ssh config file entries, like this:
Host gpu-node-1
HostName 10.0.0.1
User alice
#probeYou can of course also edit this manually.
If you don't want to use a config file, you can use:
-
The command line (you can use bash brace-expansion): who-gpu gpu-node-{1..8} who-gpu alice@192.168.10.14 192.168.10.15
-
A hosts file: one host per line ( #comments / blanks ignored). Pass it with-f, or use--no-ssh-configto fall back to~/.who-gpu-hosts. Seehosts.example.who-gpu -f myhosts.txt
By default who-gpu prints the compact summary (or the dashboard, with
DEFAULT_MODE=web) and takes hosts from ~/.ssh/config. The flags below
change that:
Only one of --summary, --full or --web are to be used at a time.
Environment overrides: WHO_GPU_HOSTS (fallback hosts file path),
WHO_GPU_SSH_CONFIG (ssh config path), WHO_GPU_OUT (where --web writes,
default ~/.local/share/who-gpu), WHO_GPU_INTERVAL (seconds between --web probe
cycles; unset means 10, or 60 when SSH connections can't be reused),
WHO_GPU_NO_MUX (set to 1 to disable --web SSH connection reuse).
--json dumps JSON for scripting; it's what --web is
built on.
- CLI Summary mode (default): per host: busy/total GPUs, the usernames running GPU processes, and who's logged in.
- CLI Full mode (--full): per host: uptime/load, logged-in users, per-GPU utilization and memory, each GPU process mapped to its owning username, and the top CPU processes.
- Web GUI (--web): one card per machine, grouped by availability, with busy/total GPUs, GPU users, per-GPU utilization bars, and on click the full breakdown plus the plainnvidia-smitable.
- Uses ssh -o BatchMode=yes. Hosts without working key auth show up as failed.
- Probes run in parallel, so one dead host won't hold up the rest.
- It reads GPU process owners via nvidia-smi+ps.
- If you are using this and would like to see any features or changes implemented, don't hesitate to open an issue (or a PR)!