rGPU documentation

This path intercepts CUDA driver, runtime and supported math-library calls. It is independent of the Python rgpu device. The client needs Linux; a Mac uses a Linux container.

Build and test without a GPU

With Docker running and Python 3 with pip installed, run this from the repository root:

./scripts/build_client.sh

This fetches missing CUDA headers, regenerates code when needed, builds the container image if absent, then compiles the shim and runs the test suite. You do not need to run fetch_headers.sh or codegen/run.sh separately; those are available when you want to run only an individual preparation step.

The driver shim is build/libcuda.so.1. GPU-free tests exercise client serialization and server dispatch against a fake CUDA driver. They cannot establish support for every real GPU workload.

Run the server

On a GPU host with the NVIDIA driver and CUDA toolkit:

cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j
./build/rgpu-server

Restrict inbound port 9713 before starting the server. The current C++ server binds all IPv4 interfaces and has no authentication. An SSH tunnel does not close the externally listening port. Use host/cloud firewall rules to deny untrusted access. See deployment.

Run a smoke test through a tunnel

ssh -N -L 9713:127.0.0.1:9713 user@gpu-host

In another client terminal:

LD_LIBRARY_PATH=build RGPU_SERVER=127.0.0.1:9713 ./build/rpc_smoke

Run any CUDA program

The shim does not depend on Python or PyTorch. It stands in for the CUDA libraries themselves, so a Linux program that loads them as shared libraries can use the remote GPU unchanged. This includes C and C++ programs, other frameworks and inference engines.

The repository's driver-API example shows the whole loop. scripts/deploy_server.sh builds its kernel on the GPU host as build/vecadd.fatbin and copies it back. With the tunnel above running:

LD_LIBRARY_PATH=build RGPU_SERVER=127.0.0.1:9713 ./build/vecadd build/vecadd.fatbin

For your own program, preload the stand-ins for the libraries it uses:

LD_PRELOAD=$PWD/build/libcudart.so.12:$PWD/build/libcuda.so.1 \
RGPU_SERVER=127.0.0.1:9713 ./your-program

Add build/libcublas.so.12, build/libcublasLt.so.12 or build/libcudnn.so.9 to LD_PRELOAD if it uses those libraries. The server then needs the real ones; see RGPU_CUBLAS, RGPU_CUBLASLT and RGPU_CUDNN in the configuration reference.

The program has to meet four conditions:

It must load the CUDA runtime as a shared library. nvcc links the runtime statically by default, and the static runtime cannot run on a remote driver. Build with nvcc -cudart shared.
It must be built against CUDA 12. The stand-ins are libcudart.so.12, libcublas.so.12 and libcudnn.so.9.
It must use only shimmed libraries. cuFFT, cuSPARSE, cuRAND and NCCL have no stand-ins and have not been tried.
It must use only the calls PyTorch and llama.cpp make. The cuBLAS stand-ins forward the GEMM calls (cublasSgemm, cublasGemmEx, cublasLtMatmul and their setup), and the cuDNN stand-in forwards the backend graph API and batch normalisation. Any other call returns NOT_SUPPORTED and logs its name.

What has been checked:

Program	Real GPU	Fake driver (CI)
Driver-API programs (`tests/cuda/vecadd.cpp`, `tests/hw/`)	Yes	Yes
Runtime-API calls, including `<<<>>>` kernel launches	Yes (`tests/hw/cublas_check.cu`)	Yes (`tests/cudart_smoke.cpp`)
cuBLAS and cuBLASLt GEMMs	Yes (`tests/hw/cublas_check.cu`)	Yes
cuDNN backend-graph convolution and batch normalisation	Yes (`tests/hw/cudnn_check.cu`)	Yes
PyTorch, up to ResNet-18 inference	Yes	Not applicable
llama.cpp and Ollama, one small model	Yes, output identical to native	Not applicable

tests/hw/math_check.sh runs the two standalone checks on a GPU host, once against the real libraries and once through the shims, and compares every result with a CPU reference. TensorFlow and CuPy have not been tried. JAX does not run through this path; GPU jax.jit fails in the separate experiment under jax/ in the repository.

Run Ollama and llama.cpp

Ollama and llama.cpp run through the shim, with output identical to a native run. The setup, the one setting it needs and the measured speeds are on their own page: Ollama and llama.cpp.

Run PyTorch

For a GPU-free client, scripts/run_torch.sh builds and runs the container path. It downloads PyTorch and CUDA libraries, which require several gigabytes of disk space.

For a loopback check on an existing GPU host:

./scripts/remote_torch.sh user@gpu-host

The test ladder checks CUDA availability, memory operations, math libraries and ResNet-18 inference against a CPU reference. Inspect which libraries actually load.

LD_LIBRARY_PATH alone is insufficient for PyTorch: library RPATH and absolute-path preloads can take precedence. Use the provided scripts, which arrange interception, rather than assuming the driver shim alone covers the stack. The runtime, cuBLAS, cuBLASLt and cuDNN need their corresponding shims too.

Compatibility limits

Managed memory and zero-copy host mapping are unsupported across this network boundary. Unimplemented API stubs return CUDA_ERROR_NOT_SUPPORTED; successful startup does not imply every CUDA feature is available.

See the generated codegen/report.txt after code generation for the API coverage breakdown. For diagnosis, RGPU_BATCH=0 disables deferred batching so failures surface closer to their source, at the cost of more round trips.

CUDA shim