Skip to content

vox-tensor API Reference (generated)

Crate: vox-tensor · Version: 0.6.0 · Target: Library (lib.rs)

Pure-CPU JSONL data loaders / training-pair types (Burn extracted 2026-05-08).

Pure-CPU data loaders and replay buffers for Vox training pipelines.

Used by:

  • vox-corpus for corpus extraction
  • vox-ml-cli for training data preparation
  • vox-plugin-mens-candle-cuda for QLoRA training data ingestion
DependencyKind
vox-actor-runtimeNormal
ModuleDescription
dataPure-Rust tokenizer and JSONL DataLoader — always compiled, no GPU required.
replayExperience replay buffer for catastrophic forgetting mitigation — always compiled, no GPU required.
Dependency Topology, diagram 17 nodes, 6 edges: vox-tensor → vox-actor-runtime; vox-ml-cli → vox-tensor; vox-plugin-mens-candle-… → vox-tensor; vox-plugin-mens-candle-… → vox-tensor; vox-plugin-mens-candle-… → vox-tensor; vox-populi → vox-tensorvox-tensorvox-actor-runtimevox-ml-clivox-plugin-mens-candle-corevox-plugin-mens-candle-cudavox-plugin-mens-candle-metalvox-populi
Diagram source
graph {
direction: right
node vox_tensor: "vox-tensor"
node vox_actor_runtime: "vox-actor-runtime"
vox_tensor -> vox_actor_runtime
node vox_ml_cli: "vox-ml-cli"
vox_ml_cli -> vox_tensor
node vox_plugin_mens_candle_core: "vox-plugin-mens-candle-core"
vox_plugin_mens_candle_core -> vox_tensor
node vox_plugin_mens_candle_cuda: "vox-plugin-mens-candle-cuda"
vox_plugin_mens_candle_cuda -> vox_tensor
node vox_plugin_mens_candle_metal: "vox-plugin-mens-candle-metal"
vox_plugin_mens_candle_metal -> vox_tensor
node vox_populi: "vox-populi"
vox_populi -> vox_tensor
}
Consumer CrateRole
vox-ml-cliDirect workspace dependent
vox-plugin-mens-candle-coreDirect workspace dependent
vox-plugin-mens-candle-cudaDirect workspace dependent
vox-plugin-mens-candle-metalDirect workspace dependent
vox-populiDirect workspace dependent
KindSymbolSignatureSourceDescription
structdata::SyntaxSpanpub struct SyntaxSpancrates/vox-tensor/src/data.rs#L9-L14A byte-level span aligned with a syntax element kind and a loss weight.
structdata::ChatmlTurnpub struct ChatmlTurncrates/vox-tensor/src/data.rs#L18-L23One turn in a ChatML conversation.
structdata::TrainingPairpub struct TrainingPaircrates/vox-tensor/src/data.rs#L30-L55A single prompt→response training pair or multi-turn sequence (matches dogfood JSONL schema). Deserializes via TrainingPairRaw (private struct) so that instruction → prompt and output → response normalization happens on the way in, even when both canonical and alias keys are present.
fndata::effective_promptpub fn effective_prompt(&self) -> Option<&String>crates/vox-tensor/src/data.rs#L113-L115—
fndata::effective_responsepub fn effective_response(&self) -> Option<&String>crates/vox-tensor/src/data.rs#L117-L119—
structdata::VoxTokenizerpub struct VoxTokenizercrates/vox-tensor/src/data.rs#L142Legacy Burn / dogfood tokenizer: ASCII + ChatML fence compounds only.
fndata::encodepub fn encode(text: &str) -> Vec<u32>crates/vox-tensor/src/data.rs#L149-L178Encode a string into a sequence of token IDs. Greedy longest-match on compound tokens first, then single-char ASCII, then UNK for anything else.
fndata::decodepub fn decode(ids: &[u32]) -> Stringcrates/vox-tensor/src/data.rs#L181-L201Decode token IDs back into a String (best-effort, for debugging).
fndata::encode_chatmlpub fn encode_chatml(system: &str, user: &str, assistant: &str) -> Vec<u32>crates/vox-tensor/src/data.rs#L204-L211Format in ChatML and encode to token IDs for training.
fndata::encode_chatml_turnspub fn encode_chatml_turns(turns: &[ChatmlTurn]) -> Vec<u32>crates/vox-tensor/src/data.rs#L214-L224Format multi-turn sequence in ChatML and encode.
fndata::encode_chatml_inference_prefixpub fn encode_chatml_inference_prefix(system: &str, user: &str) -> Vec<u32>crates/vox-tensor/src/data.rs#L227-L234ChatML prefix through user turn (open assistant slot) — for native inference with VoxTokenizer.
fndata::tokenize_for_trainingpub fn tokenize_for_training(crates/vox-tensor/src/data.rs#L239Tokenize and pad/truncate to exactly max_len tokens. Labels mask the prompt with -100 so the model only learns to reproduce the assistant response.
fndata::tokenize_turns_for_trainingpub fn tokenize_turns_for_training(crates/vox-tensor/src/data.rs#L259Tokenize multi-turn turns for training.
structdata::JsonlDataLoaderpub struct JsonlDataLoadercrates/vox-tensor/src/data.rs#L323-L328A sequential iterator over a JSONL training file. Yields (input_tensor, label_tensor) pairs encoded with VoxTokenizer. Tokens are padded/truncated to max_len. The GPU feature flag is not required here — tensors are emitted as 1-D burn tensors only when the consumer is compiled with gpu.
fndata::openpub fn open<P: AsRef<Path>>(crates/vox-tensor/src/data.rs#L336Open a JSONL training file. * max_len — maximum sequence length in tokens (default 512 for CPU, 2048 for GPU) * system_prompt — injected as the system role in ChatML format * min_rating — skip rows with rating < min_rating (0 = include all)
fndata::newpub fn new<P: AsRef<Path>>(path: P) -> std::io::Result<Self>crates/vox-tensor/src/data.rs#L352-L354Convenience: open with sensible CPU defaults (max_len=512, no rating filter).
fndata::count_jsonl_recordspub fn count_jsonl_records<P: AsRef<Path>>(path: P) -> std::io::Result<usize>crates/vox-tensor/src/data.rs#L403-L413Count the number of non-empty lines in a JSONL file (O(1) RAM).
enumdata::MalformedJsonlPolicypub enum MalformedJsonlPolicycrates/vox-tensor/src/data.rs#L417-L423How to handle lines that are non-empty but not valid [TrainingPair] JSON.
fndata::load_all_with_policypub fn load_all_with_policy<P: AsRef<Path>>(crates/vox-tensor/src/data.rs#L428Load all records from a JSONL file into memory. With [MalformedJsonlPolicy::Skip], malformed lines are skipped (historical default).
fndata::load_allpub fn load_all<P: AsRef<Path>>(path: P, min_rating: u8) -> std::io::Result<Vec<TrainingPair>>crates/vox-tensor/src/data.rs#L469-L471Load all records from a JSONL file into memory. Skips malformed lines.
structreplay::ReplayConfigpub struct ReplayConfigcrates/vox-tensor/src/replay.rs#L26-L35Replay buffer configuration.
structreplay::ReplaySamplepub struct ReplaySamplecrates/vox-tensor/src/replay.rs#L52-L60A training pair with optional loss tracking for mix-CD prioritisation.
fnreplay::loss_deltapub fn loss_delta(&self) -> f64crates/vox-tensor/src/replay.rs#L64-L69Loss delta (positive means the model is forgetting this sample).
fnreplay::is_at_riskpub fn is_at_risk(&self, threshold: f64) -> boolcrates/vox-tensor/src/replay.rs#L72-L74Whether this sample is “at risk” of being forgotten.
structreplay::ReplayBufferpub struct ReplayBuffercrates/vox-tensor/src/replay.rs#L80-L86Experience replay buffer that mixes base pre-training data into fine-tuning batches.
fnreplay::newpub fn new(config: ReplayConfig) -> Selfcrates/vox-tensor/src/replay.rs#L89-L95—
fnreplay::load_from_jsonlpub fn load_from_jsonl(&mut self, path: &Path) -> anyhow::Result<usize>crates/vox-tensor/src/replay.rs#L98-L124Load base pre-training data from a JSONL file into the buffer.
fnreplay::add_samplepub fn add_sample(&mut self, pair: TrainingPair)crates/vox-tensor/src/replay.rs#L127-L150Add a single training pair to the buffer.
fnreplay::lenpub fn len(&self) -> usizecrates/vox-tensor/src/replay.rs#L153-L155Number of samples in the buffer.
fnreplay::is_emptypub fn is_empty(&self) -> boolcrates/vox-tensor/src/replay.rs#L158-L160Whether the buffer is empty.
fnreplay::update_losspub fn update_loss(&mut self, prompt_hash: u64, new_loss: f64)crates/vox-tensor/src/replay.rs#L163-L172Update loss observations for a sample (identified by prompt hash).
fnreplay::select_replay_indicespub fn select_replay_indices(&mut self, count: usize) -> Vec<usize>crates/vox-tensor/src/replay.rs#L181-L223Select count replay sample indices for mixing into a fine-tuning batch. When mix-CD is enabled, at-risk samples (loss increased) are preferentially selected. Otherwise, round-robin selection is used. Returns indices into the internal sample buffer. Use [Self::get_pair] to retrieve the actual TrainingPair for each index.
fnreplay::get_pairpub fn get_pair(&self, idx: usize) -> Option<&TrainingPair>crates/vox-tensor/src/replay.rs#L226-L228Get the training pair at a given buffer index.
fnreplay::select_replay_batchpub fn select_replay_batch(&mut self, count: usize) -> Vec<TrainingPair>crates/vox-tensor/src/replay.rs#L231-L237Convenience: select and collect training pairs in one call.
fnreplay::replay_count_for_batchpub fn replay_count_for_batch(&self, batch_size: usize) -> usizecrates/vox-tensor/src/replay.rs#L240-L242Compute how many replay samples to mix into a batch of batch_size.
fnreplay::statspub fn stats(&self) -> ReplayBufferStatscrates/vox-tensor/src/replay.rs#L245-L257Summary statistics for telemetry.
structreplay::ReplayBufferStatspub struct ReplayBufferStatscrates/vox-tensor/src/replay.rs#L262-L266Telemetry summary for the replay buffer.