c111b3a9b0
Phase 01 (LLM Tuning): - Gemma4 26B: 74.65 t/s (fast) - Qwen 35B: 61.62 t/s (balanced) - Gemma4 31B: 16.0 t/s (deep-coder) - Qwen 27B: 16.7 t/s (deep-logic) - Qwen 122B: 8.95 t/s (ultra, GPU 1 only) Phase 02 (API Engine): - FastAPI reverse proxy on port 8000 - /engine/switch hot-swap with 503 protection - config/engine_models.json as single source of truth - Replaced 4 individual .bat files with unified engine File cleanup: - scripts/ 85 files -> 9 + _archive/ - Root .bat files -> _archive/
2.0 KiB
2.0 KiB
phase, task, total_tasks, status, last_updated
| phase | task | total_tasks | status | last_updated |
|---|---|---|---|---|
| 01-llm-tuning | 3 | 5 | in_progress | 2026-04-06T21:18:00+09:00 |
<current_state>
We are currently assessing the max context bounds and generation speed for dense/mid-sized models (Qwen 27B and Gemma 4 31B) in Q4_K_M formats. Qwen 27B booted successfully with -c 262144. We need to run its benchmark and then move on to testing the Gemma 4 31B context bounding limit to see if it also fits 256K.
</current_state>
<completed_work>
- Task 1: Evaluate 122B Dual GPU vs Single GPU dynamics - Done
- Task 2: Prove physical memory bandwidth limits of DDR4 on MoE architecture - Done
- Task 3: Test Qwen 27B Dense max logic - In progress, booted successfully at -c 262144 inside 24GB VRAM </completed_work>
<remaining_work>
- Task 3: Finish speed benchmark of Qwen 27B at 256K context.
- Task 4: Find maximum stable context for Gemma-4 31B Q4_K_M (17.0GB) and speed test. </remaining_work>
<decisions_made>
- Concluded that hitting 20t/s on 122B Q4_K_M is physically impossible via system DDR4 RAM. The limit is ~10-12 t/s.
- Addressed
cudaMalloc failedfor dual GPU memory splitting.n-cpu-moeleaves a massive asymmetry that intrinsically fails to full-load dual 12GB VRAM cards efficiently. - Pivoted entirely away from 122B and 35B optimization, redirecting efforts to dense models (27B and 31B) to guarantee speed and 256K context. </decisions_made>
<next_action>
Start with: Re-run node scripts/find_max_dense.mjs but make sure CUDA_VISIBLE_DEVICES correctly spans all GPUs or is explicitly blank ($env:CUDA_VISIBLE_DEVICES=""), to get the speed test output for Qwen 27B and Gemma 31B.
</next_action>