c111b3a9b0
Phase 01 (LLM Tuning): - Gemma4 26B: 74.65 t/s (fast) - Qwen 35B: 61.62 t/s (balanced) - Gemma4 31B: 16.0 t/s (deep-coder) - Qwen 27B: 16.7 t/s (deep-logic) - Qwen 122B: 8.95 t/s (ultra, GPU 1 only) Phase 02 (API Engine): - FastAPI reverse proxy on port 8000 - /engine/switch hot-swap with 503 protection - config/engine_models.json as single source of truth - Replaced 4 individual .bat files with unified engine File cleanup: - scripts/ 85 files -> 9 + _archive/ - Root .bat files -> _archive/
11 lines
570 B
Batchfile
11 lines
570 B
Batchfile
@echo off
|
|
echo =========================================================
|
|
echo Qwen3.5 35B-A3B API Server (MoE Optimized)
|
|
echo [INFO] --cpu-moe: Expert weights forced to CPU RAM
|
|
echo [INFO] -ngl 999: All attention layers on GPU
|
|
echo [INFO] -np 1: Single slot for max speed
|
|
echo =========================================================
|
|
echo.
|
|
llama_bin_run\llama-server.exe --model models\Qwen3.5-35B-A3B-Q4_K_M.gguf -ngl 999 --cpu-moe -c 4096 -np 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 -ub 512 -b 2048 -t 6 --mlock --prio 2 --port 8000 --host 0.0.0.0
|
|
pause
|