Skip to content

Journey: Native Rust LLM Training

When you have domain-specific application data housed in a Rust or typical structured backend and want to use it to fine-tune a model, you hit a massive tooling disconnect.

You have to pull the data directly from production, dump it into JSONL files, transfer them, spin up complex Virtual Environments (venv/Conda), manage nested CUDA PyTorch dependencies, and fight Python multi-threading environments in Jupyter notebooks. Your application logic effectively divorces the ML operations layer.

The Vox Paradigm: Zero-Python Native Fine-tuning

Section titled “The Vox Paradigm: Zero-Python Native Fine-tuning”

The Vox toolchain resolves this tension by providing native hardware-accelerated QLoRA fine-tuning via MENS: vox mens train dispatches Candle + qlora-rs in vox-populi (HF weights through Rust hf-hub). vox-tensor supplies VoxTokenizer, JSONL loading, and the Burn scratch path — a different lane from HF QLoRA.

You can extract corpus pairs, assemble train.jsonl, and run training without a Python training loop. The operator surface is the CLI and corpus commands today; in-language orchestration remains a product direction.

Authoritative pipeline map (sources → compiler → goldens → corpus → Mens): Vox source → Mens pipeline SSOT. Dataset contract: Mens training data contract.

Illustrative snippet (not the shipped CLI)

Section titled “Illustrative snippet (not the shipped CLI)”

The following Vox-shaped pseudocode sketches how training might be expressed in source; the supported path today is vox mens train (see mens-training.md).

// We store high-quality agent queries and responses in a Codex table.
table AgentTelemetry {
prompt_text: str
optimal_response: str
}
type QloraConfig {
r: int
lora_alpha: int
batch_size: int
epochs: int
}
query telemetry_sample_count() to int {
return len(db.AgentTelemetry.all())
}
mutation record_telemetry_pair(prompt_text: str, optimal_response: str) to Result[str] {
db.AgentTelemetry.insert({
prompt_text: prompt_text,
optimal_response: optimal_response
})?
return Ok("recorded")
}
fn build_qlora_Run(base_model: str, cfg: QloraConfig) to Result[str] {
if cfg.r <= 0 {
return Error("LoRA rank must be positive")
}
return Ok("Configured QLoRA run for " + base_model)
}

On NVIDIA hardware, build vox-cli with mens-candle-cuda (see mens-training.md and workspace build notes in AGENTS.md). Then:

Terminal window
vox mens corpus pairs … # produce target/dogfood/train.jsonl (see expl-ml-pipeline)
vox mens train --device cuda --data-dir target/dogfood --output-dir mens/runs/latest

--backend qlora and --tokenizer hf are defaults: weights are fetched natively; no PyTorch training stack.

  • Maturity: stable for the vox mens train CLI path on supported presets; GPU kernels require the documented CUDA build alias (see AGENTS.md).
  • Limitation ids: L-005 (default vox-cli build may omit GPU train/serve features until rebuilt with the Mens CUDA feature set).