Deployment and recovery

Use the right server

Neither protocol authenticates clients. Use only trusted hosts. Do not expose either service to the public internet. An encrypted SSH connection protects the tunnel; it does not protect a separate public listener.

The Python server accepts --bind and --port; retaining the default loopback bind is the simplest deployment. The C++ server currently has no equivalent bind-address option, so restrict its port through networking controls.

Prepare an existing GPU host

For the PyTorch device, follow the quickstart to install and deploy with deploy_opserver.sh. Inspect ~/opserver.log on the GPU host if startup fails. The server virtual environment is ~/opvenv; set RGPU_OPSERVER_VENV or RGPU_OPSERVER_LOG on the host to override those locations.

For the C++ path, scripts/deploy_server.sh user@host copies and builds the server on an existing host. The repository also contains cloud rental scripts; these create billable resources and are not needed if you already have a GPU machine.

Use an existing tunnel

For the PyTorch device, keep this command running in a separate client terminal, using your server's SSH destination and options:

ssh -N -L 9720:127.0.0.1:9720 -p 2222 -i ~/.ssh/gpu_key user@gpu-host

In your workload environment, run the quickstart's smoke.py through that tunnel:

rgpu-run --server 127.0.0.1:9720 python smoke.py

For a process started without rgpu-run, set RGPU_OPSERVER=127.0.0.1:9720 in its environment. Port 9713 and RGPU_SERVER belong to the separate CUDA shim.

Test locally

After installing rGPU as in the quickstart, start a CPU server in one terminal:

rgpu-opserver --device cpu

Run the quickstart's smoke.py in another terminal:

rgpu-run --server 127.0.0.1:9720 python smoke.py

Both terminals should use the same Python environment. The expected output is 8.0; this checks the protocol without establishing CUDA support or GPU performance.

On an Apple Silicon Mac, use rgpu-opserver --device mps to execute on the local Apple GPU instead. The workload still selects device="rgpu"; selecting device="mps" in the workload bypasses rGPU.

Share one GPU

One rgpu-opserver accepts any number of clients. Each connection gets its own session, and one client cannot reach another's tensors. The clients share the GPU by time-slicing, just as two local processes would.

For workloads that should not affect each other, run one server per workload on its own port, and cap each server's share of GPU memory:

rgpu-opserver --port 9720 --memory-fraction 0.5 &
rgpu-opserver --port 9721 --memory-fraction 0.5 &

Without a cap, a workload can take all of the GPU's memory, and the others then fail with out-of-memory errors. The cap applies to the whole server process, not to each session, which is why each workload needs its own server. Separate servers also stop the workloads competing for one Python process's CPU time.

rGPU does not schedule GPU time between workloads. For fair shares or hardware isolation, use NVIDIA's own tools underneath: run the MPS daemon (nvidia-cuda-mps-control -d) on the GPU host, or partition a MIG-capable GPU and start each server on its own partition with CUDA_VISIBLE_DEVICES.

Share a server only with people you would give a shell on the GPU host. There is no authentication, so anyone who can reach the port can run any operation on the GPU. Give each person their own SSH tunnel, and never expose the port.

Recovering a connection

The server retains disconnected session state for a grace period, default 120 seconds. The clients can attempt to reconnect to retained state. This handles some transport interruptions; it does not make GPU memory persistent.

A server restart or expired session loses remote state. The client must not silently treat old tensor handles as new allocations. Restart the workload from a saved checkpoint when its session is lost.

Reconnect details differ between the Python and CUDA protocols. Do not assume identical retry counts or error types. Keep regular application checkpoints for training that must survive server or machine loss.

Shut down deliberately

Stop clients before shutting down their GPU server. A running cloud GPU may continue billing after the client exits; stopping a rented instance may still incur storage costs. Manage that instance through its provider independently of rgpu-run, which owns only its local tunnel and child process.