Hosting GLM-5.2 (755B) on RTX PRO 6000 Blackwell with vLLM

We serve GLM-5.2 in FP8, about 755 GB in 141 shards, on two G4-class VMs with sixteen RTX PRO 6000 Blackwell GPUs between them, using vLLM and Ray behind LiteLLM so it is usable from pi and Claude Code. A plain pip install vllm && vllm serve does not work on this hardware for this model. This page explains why, lists every patch and flag we needed, and shows how the whole thing is brought up, validated and recovered.

Why pip install fails here

vLLM ships around fifteen attention backends and picks one by GPU compute capability. Its tables are written for what vendors test. H100 and H200 are SM90, B200 is SM100, and both have mature kernels for everything GLM-5.2 needs. The RTX PRO 6000 is SM120, where GLM-5.2-class sparse-MLA kernels barely exist: one new, shape-limited FlashInfer module.

GLM-5.2 is not a vanilla transformer. It combines multi-head latent attention with a DeepSeek-V3.2-style sparse "lightning indexer", where each query attends to its top-2048 keys, chosen per layer. Serving it needs sparse-MLA kernels, which are GPU-specific CUDA. On SM120 the only implementation JIT-compiles at startup, so it needs nvcc and ninja on PATH, and its decode kernels exist only for a hard-coded shape table of heads in {8, 16, 32, 64, 128} by top-k in {128, 512, 1024, 2048}.

The result is that the default pipeline fails in five layers, each invisible until you hit it.

The five failures, in order

Plus a model-packaging bug that is independent of the hardware: the checkpoint's config.json ships index_topk_pattern: null, while vLLM's sparse path reads only that field. The expensive indexer then runs on all 78 layers when only 21 should, attention corrupts, and you get the "France France France" looping bug.

The model-specific bits explained

The layer map. GLM-5.2 has 78 layers. 21 of them are "full" and run the indexer; 57 are "shared" and skip it. The checkpoint stores this in indexer_types, but vLLM reads index_topk_pattern, which ships null. Derive the pattern from your checkpoint's config and pass it in. Do not copy the string from a guide; pruned variants differ.

python3 -c "import json; print(''.join('F' if t=='full' else 'S' for t in json.load(open('config.json'))['indexer_types']))"
# → 78-char F/S string, passed via:
--hf-overrides '{"index_topk_pattern":"FFFSSS..."}'

The shape dispatch table. The SM120 sparse decode kernel is instantiated only for the head counts and top-k values in Figure 4. Tensor-parallel 16 gives 4 heads per GPU, which is not dispatchable. There are three ways out. Pad the heads from 4 to 8 with zero rows, which is mathematically exact and which we validated against a pure-PyTorch reference. Add a (4, 2048) instantiation, which we tried: the kernel was nondeterministic and wrong, so do not trust an untested shape. Or drop to tensor-parallel 8 for 8 heads per GPU, which is impossible here because 755 GB of weights do not fit in one VM's 768 GB once the KV cache is counted.

Dense fallback. With everything patched, long-context output is still garbage on stock FlashInfer 0.6.x. The community b12x image (see the last section) has working sparse kernels; stock does not. Dense TRITON_MLA is correct at every length we tested, from 30 to 23K tokens. The trade-off is quadratic long-context prefill and a ceiling around 256K.

The patches

Four small edits, each tagged in a comment so they can be re-applied after an upgrade.

A. Dense fallback. In vllm/model_executor/models/deepseek_v2.py, at both occurrences:

self.is_v32 = False  # DENSE_FALLBACK: sparse SM120 kernels corrupt long context

plus export VLLM_ATTENTION_BACKEND=TRITON_MLA.

B. Abort race. In vllm/v1/core/sched/scheduler.py:

# req_index = model_runner_output.req_id_to_index[req_id]
req_index = model_runner_output.req_id_to_index.get(req_id)
if req_index is None: continue  # ABORT_RACE_FIX

C. Cross-node allreduce deadlock. In vllm/distributed/device_communicators/custom_all_reduce.py, an early return at the top of _init_mnnvl_buffer, tagged PATCHED-NO-CROSSNODE-SYMM. vLLM falls back to pynccl.

D. Python 3.11 syntax. In flashinfer/comm/fd_exchange.py, add from __future__ import annotations as line 1.

Two more edits sit in our venv but are inert on the dense path: HEAD_PAD_TP16 and MASKED_MHA_NA in flashinfer_mla_sparse_sm120.py. They are only needed if sparse is re-enabled.

All changes at a glance

Launch and two-node recovery

The exact command that works after the patches. The pattern string is for the full zai-org GLM-5.2-FP8 checkpoint; derive your own for other variants.

vllm serve <model-dir> \
  --served-model-name glm-5.2 \
  --tensor-parallel-size 16 --distributed-executor-backend ray \
  --max-model-len 262144 \
  --enable-chunked-prefill --max-num-batched-tokens 8192 \
  --max-num-seqs 8 --block-size 64 \
  --gpu-memory-utilization 0.92 \
  --load-format fastsafetensors \
  --reasoning-parser glm47 \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --hf-overrides '{"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'

Two-node bring-up:

# both VMs: ray stop --force, verify nvidia-smi shows 0 compute procs
# VM A: ray start --head --port=6380   (env block: PATH incl venv bin + cuda, NCCL flags, ulimit)
# VM B: ray start --address=<HEAD_IP>:6380
# verify: ray.init() → ALIVE=2 GPUS=16
# VM A: nohup bash launch_vllm_multi.sh > vllm_serve.log 2>&1 &

Startup takes six to ten minutes: the JIT compile plus about four minutes of weight load. The success markers in the log are Using TRITON_MLA attention backend, Setting kv cache block size to 64 and Application startup complete.

Clients

LiteLLM on the head VM fronts vLLM with both wire formats; set your own API key in its config. One config line matters for speed:

extra_body: {chat_template_kwargs: {enable_thinking: false}}

That disables hidden reasoning and makes responses about 2.3× faster. Remove it for hard reasoning tasks.

• pi: add a provider pointing at the LiteLLM endpoint with the OpenAI-completions API, then pi --model <provider>/glm-5.2, or /model in-session.
• Claude Code: set ANTHROPIC_BASE_URL to the LiteLLM endpoint, ANTHROPIC_AUTH_TOKEN to your key, and ANTHROPIC_MODEL=glm-5.2.

Validate before serving to humans

Performance

The first configuration that served correctly, the one above, decoded at about 27 tok/s for a single request. The two-node allreduce over Ethernet is the ceiling there. Further tuning on the same two VMs has since taken single-stream decode to 85 to 90 tok/s, with about 640 tok/s aggregate across parallel requests. Those changes are not written up yet; they get their own page once they are reproduced from a clean VM.

The alternative: the b12x route

Manuel Parra's guide at th3nolo.com, Serving GLM-5.2-504B on 8× RTX PRO 6000, keeps sparse attention fully working with a community image. The checkpoint is 0xSero/GLM-5.2-504B, REAP-pruned by Cerebras to NVFP4 at 288 GB, which fits one eight-GPU VM. The image is a vLLM built for sm_120a with working B12X sparse kernels, published by voipmonitor.

--attention-backend B12X_MLA_SPARSE --moe-backend b12x --kv-cache-dtype fp8 --quantization modelopt_fp4
--enable-chunked-prefill --max-num-batched-tokens 8192 --decode-context-parallel-size 8
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0   # plus the same index_topk_pattern fix, and clear VLLM_B12X_MLA_EXTEND_MAX_CHUNKS

The long-context breakthrough on that route: the b12x indexer locks a workspace of about 708 MB, and the fix is bounding per-step work with chunked prefill at 8192, decode context parallel of 8 and the profiler estimate switched off. Cold 48K, 128K and 240K prompts then come back correct at 43 to 45 tok/s. Add fastsafetensors for four-minute loads and a scale-to-zero watchdog.

Both routes need the same index_topk_pattern fix, the same NCCL discipline, and the same three validation tests.

References and credits

• th3nolo.com, Serving GLM-5.2-504B on 8× RTX PRO 6000 GPUs (Manuel Parra): the pattern fix, the NCCL flags, dense-fallback isolation, the b12x route and the DCP long-context config.
• 0xSero for the REAP-pruned checkpoint and recipe, voipmonitor for the b12x image, Cerebras for REAP, and the FlashInfer and vLLM SM120 contributors.