Variet/variet_llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

wip: [06-install-and-evaluate-hermes-agent] paused at task 5/5

Browse Source
This commit is contained in:
Variet-Worker
2026-04-09 21:55:23 +09:00
parent bef6c410cc
commit 219985b9ce
4 changed files with 19 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
.planning/phases/01-llm-tuning/.continue-here.md
View File

@@ -1,43 +0,0 @@
---
phase: 01-llm-tuning
task: 3
total_tasks: 5
status: in_progress
last_updated: 2026-04-06T21:18:00+09:00
---
<current_state>
We are currently assessing the max context bounds and generation speed for dense/mid-sized models (Qwen 27B and Gemma 4 31B) in Q4_K_M formats. Qwen 27B booted successfully with `-c 262144`. We need to run its benchmark and then move on to testing the Gemma 4 31B context bounding limit to see if it also fits 256K.
</current_state>
<completed_work>
- Task 1: Evaluate 122B Dual GPU vs Single GPU dynamics - Done
- Task 2: Prove physical memory bandwidth limits of DDR4 on MoE architecture - Done
- Task 3: Test Qwen 27B Dense max logic - In progress, booted successfully at -c 262144 inside 24GB VRAM
</completed_work>
<remaining_work>
- Task 3: Finish speed benchmark of Qwen 27B at 256K context.
- Task 4: Find maximum stable context for Gemma-4 31B Q4_K_M (17.0GB) and speed test.
</remaining_work>
<decisions_made>
- Concluded that hitting 20t/s on 122B Q4_K_M is physically impossible via system DDR4 RAM. The limit is ~10-12 t/s.
- Addressed `cudaMalloc failed` for dual GPU memory splitting. `n-cpu-moe` leaves a massive asymmetry that intrinsically fails to full-load dual 12GB VRAM cards efficiently.
- Pivoted entirely away from 122B and 35B optimization, redirecting efforts to dense models (27B and 31B) to guarantee speed and 256K context.
</decisions_made>
<blockers>
- None. Hardware limitations acknowledged and bounded.
</blockers>
<context>
The user demanded explicit proof and answers regarding hardware utilization and VRAM filling geometry. With those physically justified, they requested a shift to new assets (Qwen 27B, Gemma 4 31B). We found that 27B at Q4_K_M (15.5GB) fits 256K into the dual RTX 3060 perfectly.
</context>
<next_action>
Start with: Re-run `node scripts/find_max_dense.mjs` but make sure `CUDA_VISIBLE_DEVICES` correctly spans all GPUs or is explicitly blank (`$env:CUDA_VISIBLE_DEVICES=""`), to get the speed test output for Qwen 27B and Gemma 31B.
</next_action>