feat: Variet Engine v1.0 + 5-model tuning complete
Phase 01 (LLM Tuning): - Gemma4 26B: 74.65 t/s (fast) - Qwen 35B: 61.62 t/s (balanced) - Gemma4 31B: 16.0 t/s (deep-coder) - Qwen 27B: 16.7 t/s (deep-logic) - Qwen 122B: 8.95 t/s (ultra, GPU 1 only) Phase 02 (API Engine): - FastAPI reverse proxy on port 8000 - /engine/switch hot-swap with 503 protection - config/engine_models.json as single source of truth - Replaced 4 individual .bat files with unified engine File cleanup: - scripts/ 85 files -> 9 + _archive/ - Root .bat files -> _archive/
This commit is contained in:
43
.planning/phases/01-llm-tuning/.continue-here.md
Normal file
43
.planning/phases/01-llm-tuning/.continue-here.md
Normal file
@@ -0,0 +1,43 @@
|
||||
---
|
||||
phase: 01-llm-tuning
|
||||
task: 3
|
||||
total_tasks: 5
|
||||
status: in_progress
|
||||
last_updated: 2026-04-06T21:18:00+09:00
|
||||
---
|
||||
|
||||
<current_state>
|
||||
We are currently assessing the max context bounds and generation speed for dense/mid-sized models (Qwen 27B and Gemma 4 31B) in Q4_K_M formats. Qwen 27B booted successfully with `-c 262144`. We need to run its benchmark and then move on to testing the Gemma 4 31B context bounding limit to see if it also fits 256K.
|
||||
</current_state>
|
||||
|
||||
<completed_work>
|
||||
|
||||
- Task 1: Evaluate 122B Dual GPU vs Single GPU dynamics - Done
|
||||
- Task 2: Prove physical memory bandwidth limits of DDR4 on MoE architecture - Done
|
||||
- Task 3: Test Qwen 27B Dense max logic - In progress, booted successfully at -c 262144 inside 24GB VRAM
|
||||
</completed_work>
|
||||
|
||||
<remaining_work>
|
||||
|
||||
- Task 3: Finish speed benchmark of Qwen 27B at 256K context.
|
||||
- Task 4: Find maximum stable context for Gemma-4 31B Q4_K_M (17.0GB) and speed test.
|
||||
</remaining_work>
|
||||
|
||||
<decisions_made>
|
||||
|
||||
- Concluded that hitting 20t/s on 122B Q4_K_M is physically impossible via system DDR4 RAM. The limit is ~10-12 t/s.
|
||||
- Addressed `cudaMalloc failed` for dual GPU memory splitting. `n-cpu-moe` leaves a massive asymmetry that intrinsically fails to full-load dual 12GB VRAM cards efficiently.
|
||||
- Pivoted entirely away from 122B and 35B optimization, redirecting efforts to dense models (27B and 31B) to guarantee speed and 256K context.
|
||||
</decisions_made>
|
||||
|
||||
<blockers>
|
||||
- None. Hardware limitations acknowledged and bounded.
|
||||
</blockers>
|
||||
|
||||
<context>
|
||||
The user demanded explicit proof and answers regarding hardware utilization and VRAM filling geometry. With those physically justified, they requested a shift to new assets (Qwen 27B, Gemma 4 31B). We found that 27B at Q4_K_M (15.5GB) fits 256K into the dual RTX 3060 perfectly.
|
||||
</context>
|
||||
|
||||
<next_action>
|
||||
Start with: Re-run `node scripts/find_max_dense.mjs` but make sure `CUDA_VISIBLE_DEVICES` correctly spans all GPUs or is explicitly blank (`$env:CUDA_VISIBLE_DEVICES=""`), to get the speed test output for Qwen 27B and Gemma 31B.
|
||||
</next_action>
|
||||
Reference in New Issue
Block a user