RTX 3090 rung

Fitting Ornith 35B Q5 at 128K on one RTX 3090.

The useful result is not that Ornith can be squeezed onto a 24GB GPU. It is that a single 3090 can keep Q5 weights, q8 K, turbo2 V, and 128K context while still decoding fast enough for agent work.

Model: Ornith 1.0 35B Quant: Q5_K_S GPU: RTX 3090 24GB Status: runtime-proven, quality-gated

Verdict

Best current single-3090 Ornith runtime: Q5_K_S weights, q8 K, turbo2 V, TURBO_LAYER_ADAPTIVE=7, 128K context, --n-gpu-layers 41, -b 512, -ub 64, and two layer-0 MoE expert tensors on CPU.

This is a runtime win, not a final quality promotion. The profile improved high-context server decode by about 20 tokens per second over the older Q5 baseline, but agent-task results are conflicting: historical hard-suite runs reached 6/6, while the latest rerun was 3/6. Treat it as the best proven fit/speed profile and keep quality evaluation open.

Frozen Profile

Model repo	bartowski/deepreinforce-ai_Ornith-1.0-35B-GGUF
GGUF	`deepreinforce-ai_Ornith-1.0-35B-Q5_K_S.gguf`
Hardware	One RTX 3090, 24GB VRAM
Context	`--ctx-size 131072`
KV cache	`--cache-type-k q8_0 --cache-type-v turbo2`
TurboQuant	`TURBO_LAYER_ADAPTIVE=7`
Placement	`--n-gpu-layers 41`, with layer-0 `ffn_down_exps` and `ffn_gate_exps` on CPU
Batch geometry	`-b 512 -ub 64`
Practical stability	`GGML_CUDA_DISABLE_GRAPHS=1` for the eval launcher

The Trick

The first Q5 fit used --n-gpu-layers 39. It preserved q8 K, turbo2 V, and 128K context, but high-context decode sat near 50 to 54 server tokens per second. A blunt move to more GPU layers was tight. The useful move was tensor-level placement: keep one more layer on GPU, but force only two layer-0 MoE expert tensors to CPU:

-ot '^blk\\.0\\.ffn_(down|gate)_exps\\.weight$=CPU'

That recovered just enough allocator headroom without weakening the weight quant, K cache, V cache, or context target. Layer count was too blunt; tensor placement was precise enough.

Runtime Proof

Case	Q5 ngl39 server tok/s	Promoted server tok/s	Delta	Promoted wall tok/s
`ctx98304/out8192`	53.617	77.304	+23.687	43.813
`ctx114688/out8192`	51.139	72.290	+21.150	62.840
`ctx122880/out8192`	50.004	69.723	+19.719	64.206

Observed VRAM was tight but usable: about 24,076 MiB idle and 24,130 MiB during active decode. That is why the exact host, template, graph behavior, and ubatch matter.

Quality Evidence

The runtime proof and quality proof are separate. Runtime says the profile fits and moves. Quality says whether it should become a daily agent profile.

Check	Result	Meaning
Terminal hard suite	Historical 6/6, latest rerun 3/6	Useful but conflicting. Do not quality-promote yet.
IFEval full	Strict prompt accuracy 0.834, loose prompt accuracy 0.872	Instruction following is credible for a 3090 profile.
SimpleBench 10-case ladder	5/10 at max 8192 tokens	Reasoning is not a clean win.
MMLU-Pro lite 100	59/100	Small subset only. Use as a smoke, not a leaderboard score.
LiveBench lite 50	Mean score 0.553	Small subset with local harness fixes. Directional only.

Rejected Paths

ub128 OOMed during CUDA graph launch.
ub96 and ub112 were not reliable enough for the practical launcher.
Lowering K to q5_1, q5_0, or q4_0 changed a core quality variable and collapsed on long-context prefill.
--n-gpu-layers 41 -ot output.*=CPU fit but decoded too slowly.
GGML_CUDA_FORCE_CUBLAS=ON was worse for this profile.
Graph-limit experiments improved stability evidence, not speed. Graph-disabled ub64 remains the practical launcher.

Reproduce The Shape

This page publishes the stable server shape, not provider secrets or private tunnel details. The essential runtime contract is:

GGML_CUDA_DISABLE_GRAPHS=1
TURBO_LAYER_ADAPTIVE=7

llama-server \
  --model deepreinforce-ai_Ornith-1.0-35B-Q5_K_S.gguf \
  --ctx-size 131072 \
  --cache-type-k q8_0 \
  --cache-type-v turbo2 \
  --n-gpu-layers 41 \
  -b 512 \
  -ub 64 \
  -ot '^blk\\.0\\.ffn_(down|gate)_exps\\.weight$=CPU' \
  -fa on \
  --parallel 1 \
  -fit off \
  --reasoning off \
  --jinja \
  --no-webui \
  --no-mmproj

The command assumes a TurboQuant-capable llama.cpp build and the matching GGUF file. For a less aggressive fallback, keep the same quality variables and use --n-gpu-layers 40 without the layer-0 tensor offload.