4x A100 CUDA control

GLM-5.2 abliterated Q3 runs at 128K on four A100 80GB GPUs.

This result establishes a reproducible CUDA control for the large GLM lane: Huihui GLM-5.2 abliterated, Q3_K_M-class GGUF, q8 K, turbo2 V, 128K context, and six MoE expert layers spilled to CPU.

Model: GLM-5.2 abliterated Quant: UD_Q3_K_M GPU: 4x A100-SXM4-80GB Status: profiling-proven, not quality-promoted

Verdict

The useful A100 profile is glm52-ablit-q3km-a100-128k-sixspill: TurboQuant llama-server, 128K context, q8 K, turbo2 V, -b 1024 -ub 128, layer split across four GPUs, and exactly six MoE expert layers on CPU.

This clears the practical wall-speed floor for single-request agent use, including long context. It is not a Hermes daily-driver promotion yet: the run proves fit, endpoint behavior, and raw speed, not task quality or tool reliability under real workflows.

Frozen Profile

Model repo	huihui-ai/Huihui-GLM-5.2-abliterated-GGUF
Revision	`5cf1c361c20164063c0012177c510021f4147042`
GGUF	`GLM-5.2-UD-Q3_K_M-00001-of-00009.gguf`
Hardware	Four A100-SXM4-80GB GPUs with all-to-all NVLink visible
Runtime	llama.cpp TurboQuant fork, tested branch `codex/glm52-a100-runtime-experiments` at `0cf69160`
Context	`--ctx-size 131072`
KV cache	`--cache-type-k q8_0 --cache-type-v turbo2`
Placement	`-ngl 999 --split-mode layer --tensor-split 1,1,1,1` with six MoE expert layers on CPU
Batch geometry	`-b 1024 -ub 128`
Server slots	`--parallel 1`; parallel requests queue behind one decode lane
Observed provider price	RunPod 4x A100 at `$5.96/hr`, plus storage while volumes are retained

The Trick

GLM-5.2 Q3 does not fit this 128K q8/turbo2 profile as a simple "everything on GPU" launch. The useful compromise is precise MoE placement: keep the model distributed across all four GPUs, then spill a small fixed set of expert tensors to CPU:

-ot 'blk.(20|21|40|41|60|61).ffn_.*_exps.weight=CPU'

That recovers enough memory headroom without changing the model artifact, quant, context target, K cache, V cache, chat template, or reasoning mode. It is a fit/speed control for future work on MI300X, H200/B200-class GPUs, DS4, KTransformers-style offload, or custom MoE scheduling.

Official Raw Endpoint Profile

The benchmark below is BENCORP's AA-lite raw OpenAI-compatible endpoint profile. It measures endpoint speed and latency only. It does not store raw prompt or response text, and it is not a quality benchmark.

Workload	Wall output tok/s	Server output tok/s	Elapsed p50	TTFT	Runs
1k input / 1k output	14.063	14.116	71.111s	11.979s	3/3
10k input / 1.5k output	13.920	13.964	107.761s	68.446s	3/3
100k target input / 2k output	10.804	10.869	173.541s	799.583s	3/3
10 concurrent 1k requests	2.617	14.303	385.336s	n/a	10/10

The parallel result is intentionally poor because the frozen server profile uses one slot. The useful signal is that individual decode speed stays near 14 tok/s for short and mid prompts, then falls to about 10.8 wall tok/s at the long-context target.

Reproduce The Shape

The provider path and model staging directory can change. The runtime contract should not change unless the experiment is explicitly testing a new profile:

llama-server \
  --model GLM-5.2-UD-Q3_K_M-00001-of-00009.gguf \
  --ctx-size 131072 \
  -ngl 999 \
  --split-mode layer \
  --tensor-split 1,1,1,1 \
  -ot 'blk.(20|21|40|41|60|61).ffn_.*_exps.weight=CPU' \
  --cache-type-k q8_0 \
  --cache-type-v turbo2 \
  -b 1024 \
  -ub 128 \
  --flash-attn on \
  --parallel 1 \
  --jinja \
  --no-webui \
  --no-mmproj \
  --fit off \
  --reasoning off \
  --reasoning-budget 0 \
  --no-mmap \
  -t 64 \
  -tb 64

Treat topology as part of the profile. A 4x A100 PCIe host or a weak interconnect should be re-profiled before using these numbers for cost comparisons.

Open Gates

Run Hermes daily-driver quality checks before promotion.
Test real tool-call behavior, not only one simple API smoke.
Compare against high-memory NVIDIA and MI300X/MI325X pricing.
Explore whether DS4, KTransformers-style offload, or custom MoE scheduling can improve long-context TTFT and parallel throughput.
Do not treat this as an optimal serving engine; treat it as the current CUDA control.