c111b3a9b0
Phase 01 (LLM Tuning): - Gemma4 26B: 74.65 t/s (fast) - Qwen 35B: 61.62 t/s (balanced) - Gemma4 31B: 16.0 t/s (deep-coder) - Qwen 27B: 16.7 t/s (deep-logic) - Qwen 122B: 8.95 t/s (ultra, GPU 1 only) Phase 02 (API Engine): - FastAPI reverse proxy on port 8000 - /engine/switch hot-swap with 503 protection - config/engine_models.json as single source of truth - Replaced 4 individual .bat files with unified engine File cleanup: - scripts/ 85 files -> 9 + _archive/ - Root .bat files -> _archive/
44 lines
2.0 KiB
Markdown
44 lines
2.0 KiB
Markdown
---
|
|
phase: 01-llm-tuning
|
|
task: 3
|
|
total_tasks: 5
|
|
status: in_progress
|
|
last_updated: 2026-04-06T21:18:00+09:00
|
|
---
|
|
|
|
<current_state>
|
|
We are currently assessing the max context bounds and generation speed for dense/mid-sized models (Qwen 27B and Gemma 4 31B) in Q4_K_M formats. Qwen 27B booted successfully with `-c 262144`. We need to run its benchmark and then move on to testing the Gemma 4 31B context bounding limit to see if it also fits 256K.
|
|
</current_state>
|
|
|
|
<completed_work>
|
|
|
|
- Task 1: Evaluate 122B Dual GPU vs Single GPU dynamics - Done
|
|
- Task 2: Prove physical memory bandwidth limits of DDR4 on MoE architecture - Done
|
|
- Task 3: Test Qwen 27B Dense max logic - In progress, booted successfully at -c 262144 inside 24GB VRAM
|
|
</completed_work>
|
|
|
|
<remaining_work>
|
|
|
|
- Task 3: Finish speed benchmark of Qwen 27B at 256K context.
|
|
- Task 4: Find maximum stable context for Gemma-4 31B Q4_K_M (17.0GB) and speed test.
|
|
</remaining_work>
|
|
|
|
<decisions_made>
|
|
|
|
- Concluded that hitting 20t/s on 122B Q4_K_M is physically impossible via system DDR4 RAM. The limit is ~10-12 t/s.
|
|
- Addressed `cudaMalloc failed` for dual GPU memory splitting. `n-cpu-moe` leaves a massive asymmetry that intrinsically fails to full-load dual 12GB VRAM cards efficiently.
|
|
- Pivoted entirely away from 122B and 35B optimization, redirecting efforts to dense models (27B and 31B) to guarantee speed and 256K context.
|
|
</decisions_made>
|
|
|
|
<blockers>
|
|
- None. Hardware limitations acknowledged and bounded.
|
|
</blockers>
|
|
|
|
<context>
|
|
The user demanded explicit proof and answers regarding hardware utilization and VRAM filling geometry. With those physically justified, they requested a shift to new assets (Qwen 27B, Gemma 4 31B). We found that 27B at Q4_K_M (15.5GB) fits 256K into the dual RTX 3060 perfectly.
|
|
</context>
|
|
|
|
<next_action>
|
|
Start with: Re-run `node scripts/find_max_dense.mjs` but make sure `CUDA_VISIBLE_DEVICES` correctly spans all GPUs or is explicitly blank (`$env:CUDA_VISIBLE_DEVICES=""`), to get the speed test output for Qwen 27B and Gemma 31B.
|
|
</next_action>
|