Skip to content

vox-quantize API Reference (generated)

Crate: vox-quantize · Version: 0.6.0 · Target: Library (lib.rs)

Data-free k-quant PTQ engine: SafeTensors -> Candle GGML quantized SafeTensors (GPU-first, CPU fallback).

Data-free k-quant post-training quantization engine.

SafeTensors model in -> quantized SafeTensors-canonical artifact out. Device-selectable: GPU when available (cuda/metal feature), CPU fallback. The engine sits behind the engine feature (off by default), so CORE builds that depend on this crate pull no Candle.

FeatureDefault / Enables
cudaengine, candle-core/cuda
default—
enginedep:candle-core
metalengine, candle-core/metal
ModuleDescription
device—
engine—
error—
policy—
read—
recombine—
verify—
write—
Dependency Topology, diagram 13 nodes, 2 edges: vox-ml-cli → vox-quantize; vox-populi → vox-quantizevox-quantizevox-ml-clivox-populi
Diagram source
graph {
direction: right
node vox_quantize: "vox-quantize"
node vox_ml_cli: "vox-ml-cli"
vox_ml_cli -> vox_quantize
node vox_populi: "vox-populi"
vox_populi -> vox_quantize
}
Consumer CrateRole
vox-ml-cliDirect workspace dependent
vox-populiDirect workspace dependent
KindSymbolSignatureSourceDescription
enumdevice::DevicePrefpub enum DevicePrefcrates/vox-quantize/src/device.rs#L9-L15User device preference. Auto picks the best available accelerator, falling back to CPU. Mirrors vox-plugin-mens-candle-cuda::device_select.
fndevice::selectpub fn select(pref: DevicePref) -> Result<Device, QuantizeError>crates/vox-quantize/src/device.rs#L17-L35—
structengine::QuantizeRequestpub struct QuantizeRequestcrates/vox-quantize/src/engine.rs#L17-L23—
fnengine::quantizepub fn quantize(req: &QuantizeRequest) -> Result<QuantReport, QuantizeError>crates/vox-quantize/src/engine.rs#L25—
enumerror::QuantizeErrorpub enum QuantizeErrorcrates/vox-quantize/src/error.rs#L6-L21—
enumpolicy::TensorRolepub enum TensorRolecrates/vox-quantize/src/policy.rs#L10-L17—
fnpolicy::from_keypub fn from_key(key: &str) -> Selfcrates/vox-quantize/src/policy.rs#L20-L51—
enumpolicy::QuantMixturepub enum QuantMixturecrates/vox-quantize/src/policy.rs#L55-L61—
fnpolicy::target_forpub fn target_for(&self, role: TensorRole) -> Option<GgmlDType>crates/vox-quantize/src/policy.rs#L64-L87—
fnpolicy::bits_per_weightpub fn bits_per_weight(dtype: GgmlDType) -> f64crates/vox-quantize/src/policy.rs#L93-L95Bits-per-weight for a GGML quantized dtype, derived from candle_core’s real block layout (type_size bytes / block_size elements) — never a separately hardcoded constant.
structpolicy::QuantizePlanpub struct QuantizePlancrates/vox-quantize/src/policy.rs#L100-L110Exact sizing for a plan_quantize run: the output artifact’s size, the peak RAM the engine needs to produce it, and the derived bpw/params.
fnpolicy::plan_quantizepub fn plan_quantize(crates/vox-quantize/src/policy.rs#L129Size a quantization run exactly from the safetensors headers in input_dir, with no fitted parameter anywhere. For every tensor in every *.safetensors file, classifies its role (TensorRole::from_key), resolves the mixture’s target dtype for that role (QuantMixture::target_for), applies GGML block-size alignment (resolve_dtype), and sizes it with candle’s real GgmlDType block layout — elems / block_size * type_size (this also covers KeepF32 / fallback GgmlDType::F32, whose block_size() == 1 and type_size() == 4 reduce the same formula to elems * 4). peak_bytes = 2*S_max + output_bytes + 4*4*p_max: - 2*S_max — the largest shard’s on-disk size, doubled for the one-slot shard cache holding one shard while the next is read (see read.rs). - output_bytes — the exact total computed above. - 4*4*p_max — round_trip_mse (verify.rs) holds 4 live F32 buffers (src, deq, diff, sq) the size of the largest tensor.
fnpolicy::resolve_dtypepub fn resolve_dtype(target: GgmlDType, last_dim: usize) -> GgmlDTypecrates/vox-quantize/src/policy.rs#L185-L209Enforce GGML block-size alignment against the tensor’s last dimension.
structread::SafeTensorsSourcepub struct SafeTensorsSourcecrates/vox-quantize/src/read.rs#L12-L21A SafeTensors model source: single model.safetensors or sharded via model.safetensors.index.json (HF weight_map).
fnread::openpub fn open(dir: &Path) -> Result<Self, QuantizeError>crates/vox-quantize/src/read.rs#L107-L177—
fnread::tensor_namespub fn tensor_names(&self) -> &[String]crates/vox-quantize/src/read.rs#L222-L224—
fnread::load_f32pub fn load_f32(&self, name: &str) -> Result<Tensor, QuantizeError>crates/vox-quantize/src/read.rs#L227-L247Load a tensor and cast to f32 on CPU.
fnrecombine::recombinepub fn recombine(crates/vox-quantize/src/recombine.rs#L14Build a complete model in out_dir by taking every base tensor and overwriting the keys present in the merged subset. Errors if the subset contains a key absent from the base (a sign of an adapter/base mismatch).
fnrecombine::recombine_with_shard_budgetpub fn recombine_with_shard_budget(crates/vox-quantize/src/recombine.rs#L31Same as [recombine], but writes shards of at most shard_bytes each instead of one unsharded file, and takes the merged overrides already loaded rather than a path to load them from. Never holds more than one shard’s worth of tensors resident: sizes for grouping come from safetensors headers (no tensor data read), and each shard’s tensors are loaded, written, and dropped before the next group is read.
structverify::TensorQuantStatpub struct TensorQuantStatcrates/vox-quantize/src/verify.rs#L9-L17—
structverify::QuantReportpub struct QuantReportcrates/vox-quantize/src/verify.rs#L20-L26—
fnverify::round_trip_msepub fn round_trip_mse(src: &Tensor, q: &QTensor) -> Result<f64, QuantizeError>crates/vox-quantize/src/verify.rs#L29-L35Dequantize q and compute mean-squared error against the f32 source src.
fnverify::round_trip_max_abspub fn round_trip_max_abs(src: &Tensor, q: &QTensor) -> Result<f64, QuantizeError>crates/vox-quantize/src/verify.rs#L38-L42Max absolute error against the f32 source.
structwrite::TensorMetapub struct TensorMetacrates/vox-quantize/src/write.rs#L15-L20—
structwrite::ArtifactWriterpub struct ArtifactWritercrates/vox-quantize/src/write.rs#L34-L59—
fnwrite::newpub fn new() -> Selfcrates/vox-quantize/src/write.rs#L95-L107Streaming writer with no output directory yet — flushes shards to a scratch directory as tensors are added and moves them into the real directory at [Self::finish]. Keeps new()’s no-argument, non-fallible signature so engine.rs (which doesn’t know the output directory at construction time) is unchanged. Creating the scratch directory can fail (disk full, no writable temp root, …), but new() can’t return Result, so that creation is deferred to the first call to Self::stage (from add_quantized or add_f32), which already returns Result and can propagate it via QuantizeError::Io instead of panicking.
fnwrite::with_shard_budgetpub fn with_shard_budget(crates/vox-quantize/src/write.rs#L145Streaming writer that flushes shards directly into out_dir, once shard_bytes of pending tensor data has accumulated, with no scratch dir or move at finish. mixture is accepted for symmetry with finish’s signature; finish’s own mixture argument is what’s written to quant-metadata.json.
fnwrite::add_quantizedpub fn add_quantized(crates/vox-quantize/src/write.rs#L164—
fnwrite::add_f32pub fn add_f32(&mut self, name: &str, t: &Tensor) -> Result<(), QuantizeError>crates/vox-quantize/src/write.rs#L184-L201—
fnwrite::finishpub fn finish(mut self, out_dir: &Path, mixture: &str) -> Result<(), QuantizeError>crates/vox-quantize/src/write.rs#L270-L333—