Troubleshooting and limits

Connection refused or handshake failure

Check that the server is running, the SSH tunnel is open, and the client points to the correct local port. The PyTorch device uses RGPU_OPSERVER and port 9720. The CUDA shim uses RGPU_SERVER and port 9713. The two protocols cannot connect to one another.

PyTorch version mismatch

Print torch.__version__ in both environments after installation. Align major.minor versions and verify a CUDA-enabled server wheel works with its driver. Avoid making RGPU_ALLOW_VERSION_MISMATCH=1 the permanent fix.

Deployment takes several minutes

The first deploy_opserver.sh run creates a remote environment and downloads a multi-gigabyte CUDA-enabled PyTorch wheel. The installer prints its current CUDA index and pip progress. Wait for LISTENING; do not start a second deployment while the first pip process is active. If it ends with DEAD, read ~/opserver.log on the GPU host.

CUDA is unavailable on the server

Run python -c 'import torch; print(torch.cuda.is_available())' on the GPU host. Check the driver and installed Torch build. rgpu-opserver --device cpu is an explicit test mode, not a GPU setup repair.

“Couldn't swap” a model parameter

Move the model to rgpu before its first grad-tracking forward pass. An existing autograd graph can hold parameter references that prevent tensor swapping. Reconstruct the model on the intended device and load saved weights if you need a fresh training setup.

torch.compile selects the wrong backend

Pass backend="rgpu" explicitly, with dynamic=False for fixed-shape graph shipping. For diagnosis, use backend=rgpu.compile_backend(compiler="eager") to keep graph shipping while avoiding server-side Inductor compilation.

A custom operation fails

The Python opserver resolves ATen operations. A non-ATen custom operator has no supported server implementation, even if the graph falls back to eager execution. Replace it with supported operations or treat it as a missing feature.

Data-dependent output with out=

The client cannot preallocate an output whose size depends on GPU data. Use the operation without out=; expect a host wait to obtain the result metadata.

backward() on a local GPU fails after importing rgpu

A process that imports rgpu cannot also train on a local cuda or mps device. The local backward() fails with:

RuntimeError: opt_ready_stream && opt_parent_stream INTERNAL ASSERT FAILED at ".../torch/csrc/autograd/engine.cpp"

This is a PyTorch limitation, not a failure of the remote session. PyTorch assumes one accelerator per process, and naming a custom device makes that device the accelerator, so gradients on any other one lose their stream. It is tracked as pytorch/pytorch#161129 and affects PyTorch 2.8 and later.

Forward passes on the local device still work, and so does everything on rgpu. To compare local and remote training, run them in separate processes and import rgpu only in the remote one.

The session is gone

The GPU process may have restarted, or the disconnected session may have expired. Its old tensor handles are no longer valid. Restart the workload and restore a saved checkpoint. Extending a grace period cannot recover memory after a server restart.

The CUDA shim starts but math fails

Check that all required runtime and math-library shims are loaded. PyTorch may preload stock libraries by absolute path. Use the provided test scripts and inspect loaded library paths. An unimplemented driver call should report CUDA_ERROR_NOT_SUPPORTED.

Remote execution is slower

Start with rgpu.stats() with the PyTorch device. Look for repeated host reads, small batches, large uploads and compilation cost over a short run. Follow the measurement checklist before selecting an optimization.

Report a problem

Open a GitHub issue with the backend, client/server versions, host GPU and driver, command, minimal reproducer and full error. Include statistics or timings when relevant. Remove credentials and private input data from logs.