Skip to content

vox-llm-egress API Reference (generated)

Crate: vox-llm-egress · Version: 0.6.0 · Target: Library (lib.rs)

Sanctioned low-layer LLM provider egress: chat_once/stream_once/embed_once + per-provider AIMD throttle. Pure wire — no config/secret resolution (callers pass a resolved EgressRequest). The single path all OpenAI-compatible inference egress goes through.

Sanctioned low-layer LLM provider egress. Pure wire: callers pass a fully-resolved [EgressRequest] (resolution lives in vox_config::resolve_egress); this crate does throttle + HTTP + 429 handling + response parsing. It owns NO config/secret resolution and NO telemetry-to-db (both pull higher layers) — chat_once returns the tokens/cost callers need to record telemetry themselves.

DependencyKind
vox-http-clientNormal
vox-openaiNormal
ModuleDescription
repetition_guard—
throttle—
StructDescription
EgressRequestA fully-resolved provider request. No resolution happens in this crate. Debug is hand-written to redact api_key so the bearer token can never leak into logs/traces/error messages. headers here carry only non-secret attribution (HTTP-Referer / X-Title); the bearer is applied separately from api_key.
ChatMessageOne chat message on the wire (OpenAI-compatible). The three trailing fields are additive plumbing for a full tool-calling turn (Task 1.3b): an assistant message MAY carry tool_calls (the calls it requested), and a role: "tool" result message carries tool_call_id (and optionally name) to correlate the result back to the specific call. All three are skip_serializing_if = "Option::is_none" so every existing plain {role, content} caller (ghost_text, inline_edit, plan, eval, judge, route, …) continues to serialize an unchanged wire payload — no tool_calls/tool_call_id/name keys at all, not even null, when left None.
LlmImageUrl—
ToolDefTool definition passed through to the provider.
EgressToolCallA single tool invocation the model requested, parsed from the OpenAI-compatible message.tool_calls wire shape: {"id": "...", "type": "function", "function": {"name": "...", "arguments": "<json-encoded string>"}}. This crate defines its own minimal type here rather than reusing vox_openai::chat_completion::ChatCompletionToolCall (which this crate already depends on for other reasons) for two reasons: (1) that type has no id field, which a future tool-dispatch loop needs to correlate a call with its result message, and (2) chat_once parses the whole response body as a raw serde_json::Value rather than through vox_openai’s typed ChatCompletionResponse — pulling in a typed sub-struct for just this one field would be inconsistent with the rest of this file. A future consolidation task may want to unify these; not done here to keep this change to pure additive plumbing.
ChatParamsPer-call generation parameters.
EgressChatResponseParsed chat result. Carries usage/cost/latency so callers record telemetry.
EnumDescription
LlmContentPart—
EgressErrorStructured egress failure so callers map to their own error types.
FunctionDescription
is_context_exceededTrue when this error’s underlying HTTP response body looks like a provider-reported “prompt exceeded the model’s context window” failure. See [is_context_exceeded_error] for the matching rules and their caveats. Only [EgressError::Status] carries a response body, so every other variant is false (a rate-limit, transport failure, or JSON-decode failure is never a context-overflow condition).
is_context_exceeded_errorBest-effort classifier for “the provider rejected this request because the prompt (plus requested completion) exceeded the model’s context window.” Vox routes chat through 5 provider lanes (see ChatProviderRouteKind in vox_actor_runtime::model_resolution) — OpenRouter, a generic OpenAI-compatible endpoint, HuggingFace Router, HuggingFace Dedicated, and Ollama — and each reports context overflow with different HTTP-body phrasing. This mirrors the pattern LiteLLM uses to normalize the same class of provider-specific error strings into a single ContextWindowExceededError (see docs/src/architecture/multi-provider-local-cloud-routing-research-2026-07-30.md). # Matching approach This does a case-insensitive substring match of body against a curated phrase list covering common OpenAI-compatible, OpenRouter, HuggingFace, and Ollama overflow phrasings. It is deliberately not status-code-gated: providers use a mix of 400, 413, and even 422 for this condition (and proxies like OpenRouter may relay a wrapped upstream body under its own status), so gating on a fixed status code would silently miss real cases. The tradeoff is the inverse risk — a body that happens to contain one of these phrases for an unrelated reason would be misclassified — but that is judged unlikely for a technical error string. # Completeness caveat This is a best-effort, non-exhaustive phrase list, not a guaranteed-complete provider taxonomy. Provider error text changes over time and new providers may phrase this differently again. Extend the phrase list as new provider phrasings are observed in the wild; do not treat a false result as proof the request wasn’t a context-overflow failure.
estimate_costThe ONE cost estimate: (prompt+completion tokens)/1000 * cost_per_1k. Used only when the provider reports no cost. Callers MUST NOT re-implement this math (single source).
TypeDescription
ChatStreamStreaming item type for [stream_once].
ChatDeltaEventStreamStreaming item type for [stream_events_once] (carries both delta.content and delta.tool_calls fragments).
Dependency Topology, diagram 17 nodes, 6 edges: vox-llm-egress → vox-http-client; vox-llm-egress → vox-openai; vox-actor-runtime → vox-llm-egress; vox-cli → vox-llm-egress; vox-config → vox-llm-egress; vox-gamify → vox-llm-egressvox-llm-egressvox-http-clientvox-openaivox-actor-runtimevox-clivox-configvox-gamify
Diagram source
graph {
direction: right
node vox_llm_egress: "vox-llm-egress"
node vox_http_client: "vox-http-client"
vox_llm_egress -> vox_http_client
node vox_openai: "vox-openai"
vox_llm_egress -> vox_openai
node vox_actor_runtime: "vox-actor-runtime"
vox_actor_runtime -> vox_llm_egress
node vox_cli: "vox-cli"
vox_cli -> vox_llm_egress
node vox_config: "vox-config"
vox_config -> vox_llm_egress
node vox_gamify: "vox-gamify"
vox_gamify -> vox_llm_egress
}
Consumer CrateRole
vox-actor-runtimeDirect workspace dependent
vox-cliDirect workspace dependent
vox-configDirect workspace dependent
vox-gamifyDirect workspace dependent
KindSymbolSignatureSourceDescription
structEgressRequestpub struct EgressRequestcrates/vox-llm-egress/src/lib.rs#L33-L46A fully-resolved provider request. No resolution happens in this crate. Debug is hand-written to redact api_key so the bearer token can never leak into logs/traces/error messages. headers here carry only non-secret attribution (HTTP-Referer / X-Title); the bearer is applied separately from api_key.
structChatMessagepub struct ChatMessagecrates/vox-llm-egress/src/lib.rs#L75-L96One chat message on the wire (OpenAI-compatible). The three trailing fields are additive plumbing for a full tool-calling turn (Task 1.3b): an assistant message MAY carry tool_calls (the calls it requested), and a role: "tool" result message carries tool_call_id (and optionally name) to correlate the result back to the specific call. All three are skip_serializing_if = "Option::is_none" so every existing plain {role, content} caller (ghost_text, inline_edit, plan, eval, judge, route, …) continues to serialize an unchanged wire payload — no tool_calls/tool_call_id/name keys at all, not even null, when left None.
enumLlmContentPartpub enum LlmContentPartcrates/vox-llm-egress/src/lib.rs#L100-L105—
structLlmImageUrlpub struct LlmImageUrlcrates/vox-llm-egress/src/lib.rs#L108-L110—
structToolDefpub struct ToolDefcrates/vox-llm-egress/src/lib.rs#L114-L118Tool definition passed through to the provider.
structEgressToolCallpub struct EgressToolCallcrates/vox-llm-egress/src/lib.rs#L134-L147A single tool invocation the model requested, parsed from the OpenAI-compatible message.tool_calls wire shape: {"id": "...", "type": "function", "function": {"name": "...", "arguments": "<json-encoded string>"}}. This crate defines its own minimal type here rather than reusing vox_openai::chat_completion::ChatCompletionToolCall (which this crate already depends on for other reasons) for two reasons: (1) that type has no id field, which a future tool-dispatch loop needs to correlate a call with its result message, and (2) chat_once parses the whole response body as a raw serde_json::Value rather than through vox_openai’s typed ChatCompletionResponse — pulling in a typed sub-struct for just this one field would be inconsistent with the rest of this file. A future consolidation task may want to unify these; not done here to keep this change to pure additive plumbing.
structChatParamspub struct ChatParams<'a>crates/vox-llm-egress/src/lib.rs#L151-L160Per-call generation parameters.
structEgressChatResponsepub struct EgressChatResponsecrates/vox-llm-egress/src/lib.rs#L164-L190Parsed chat result. Carries usage/cost/latency so callers record telemetry.
enumEgressErrorpub enum EgressErrorcrates/vox-llm-egress/src/lib.rs#L194-L199Structured egress failure so callers map to their own error types.
fnis_context_exceededpub fn is_context_exceeded(&self) -> boolcrates/vox-llm-egress/src/lib.rs#L257-L264True when this error’s underlying HTTP response body looks like a provider-reported “prompt exceeded the model’s context window” failure. See [is_context_exceeded_error] for the matching rules and their caveats. Only [EgressError::Status] carries a response body, so every other variant is false (a rate-limit, transport failure, or JSON-decode failure is never a context-overflow condition).
fnis_context_exceeded_errorpub fn is_context_exceeded_error(_status_code: u16, body: &str) -> boolcrates/vox-llm-egress/src/lib.rs#L297-L331Best-effort classifier for “the provider rejected this request because the prompt (plus requested completion) exceeded the model’s context window.” Vox routes chat through 5 provider lanes (see ChatProviderRouteKind in vox_actor_runtime::model_resolution) — OpenRouter, a generic OpenAI-compatible endpoint, HuggingFace Router, HuggingFace Dedicated, and Ollama — and each reports context overflow with different HTTP-body phrasing. This mirrors the pattern LiteLLM uses to normalize the same class of provider-specific error strings into a single ContextWindowExceededError (see docs/src/architecture/multi-provider-local-cloud-routing-research-2026-07-30.md). # Matching approach This does a case-insensitive substring match of body against a curated phrase list covering common OpenAI-compatible, OpenRouter, HuggingFace, and Ollama overflow phrasings. It is deliberately not status-code-gated: providers use a mix of 400, 413, and even 422 for this condition (and proxies like OpenRouter may relay a wrapped upstream body under its own status), so gating on a fixed status code would silently miss real cases. The tradeoff is the inverse risk — a body that happens to contain one of these phrases for an unrelated reason would be misclassified — but that is judged unlikely for a technical error string. # Completeness caveat This is a best-effort, non-exhaustive phrase list, not a guaranteed-complete provider taxonomy. Provider error text changes over time and new providers may phrase this differently again. Extend the phrase list as new provider phrasings are observed in the wild; do not treat a false result as proof the request wasn’t a context-overflow failure.
typeChatStreampub type ChatStream = Pin<Box<dyn Stream<Item = Result<String, EgressError>> + Send>>crates/vox-llm-egress/src/lib.rs#L334Streaming item type for [stream_once].
typeChatDeltaEventStreampub type ChatDeltaEventStream =crates/vox-llm-egress/src/lib.rs#L337Streaming item type for [stream_events_once] (carries both delta.content and delta.tool_calls fragments).
fnestimate_costpub fn estimate_cost(prompt_tokens: u32, completion_tokens: u32, cost_per_1k: f64) -> f64crates/vox-llm-egress/src/lib.rs#L343-L345The ONE cost estimate: (prompt+completion tokens)/1000 * cost_per_1k. Used only when the provider reports no cost. Callers MUST NOT re-implement this math (single source).
enumrepetition_guard::RepetitionTrippedpub enum RepetitionTrippedcrates/vox-llm-egress/src/repetition_guard.rs#L34-L39Reason a streaming generation was aborted by RepetitionGuard.
structrepetition_guard::RepetitionGuardpub struct RepetitionGuardcrates/vox-llm-egress/src/repetition_guard.rs#L43-L49O(1) per-token streaming loop breaker (NotT3Chat RepetitionGuard pattern).
fnrepetition_guard::newpub const fn new() -> Selfcrates/vox-llm-egress/src/repetition_guard.rs#L60-L68Construct a fresh RepetitionGuard with a zeroed 1024-slot ring buffer.
fnrepetition_guard::observe_hashpub fn observe_hash(&mut self, token_hash: u64) -> Option<RepetitionTripped>crates/vox-llm-egress/src/repetition_guard.rs#L72-L102Observe a pre-hashed token chunk (token_hash, e.g. from vox_crypto::fast_hash(chunk.as_bytes())) after whitespace filtering.
fnrepetition_guard::observe_chunk_with_hasherpub fn observe_chunk_with_hasher<F>(crates/vox-llm-egress/src/repetition_guard.rs#L106Observe a raw text chunk. Pure-whitespace chunks (" ", "\n", indentation) are ignored so markdown tables and indented code blocks never false-trip.
fnrepetition_guard::observe_chunkpub fn observe_chunk(&mut self, chunk: &str) -> Option<RepetitionTripped>crates/vox-llm-egress/src/repetition_guard.rs#L123-L129Observe a raw text chunk using the default 64-bit hasher.
structthrottle::ProviderThrottlepub struct ProviderThrottlecrates/vox-llm-egress/src/throttle.rs#L18-L25—
structthrottle::Permitpub struct Permit<'a>crates/vox-llm-egress/src/throttle.rs#L28-L30RAII permit: releases the slot on drop.
fnthrottle::newpub fn new(max_limit: usize) -> Selfcrates/vox-llm-egress/src/throttle.rs#L40-L50—
fnthrottle::current_limitpub fn current_limit(&self) -> usizecrates/vox-llm-egress/src/throttle.rs#L52-L54—
fnthrottle::acquirepub async fn acquire(&self) -> Permit<'_>crates/vox-llm-egress/src/throttle.rs#L57-L86Wait for a free slot (and for any active cooldown to elapse).
fnthrottle::on_rate_limitedpub fn on_rate_limited(&self, retry_after: Option<Duration>)crates/vox-llm-egress/src/throttle.rs#L89-L100Multiplicative decrease + optional header-driven cooldown.
fnthrottle::on_successpub fn on_success(&self)crates/vox-llm-egress/src/throttle.rs#L103-L112Additive increase: +1 permit per 8 consecutive successes, up to max.
fnthrottle::for_providerpub fn for_provider(provider: &str, max_limit: usize) -> &'static ProviderThrottlecrates/vox-llm-egress/src/throttle.rs#L121-L130Throttle for provider, created on first use with max_limit (the caller resolves the limit from VoxConfig — this crate takes no config dependency). The first call’s limit wins; later calls reuse the existing per-provider throttle. Leaked intentionally: providers are a small fixed set per process.
fnthrottle::acquire_permitpub async fn acquire_permit(provider: &str, max_limit: usize) -> Permit<'static>crates/vox-llm-egress/src/throttle.rs#L133-L135Acquire a permit for provider, creating its throttle with max_limit on first use.
fnthrottle::on_rate_limitedpub fn on_rate_limited(provider: &str, retry_after: Option<Duration>)crates/vox-llm-egress/src/throttle.rs#L138-L140Feed a 429 back to provider’s throttle (halve window + optional cooldown).
fnthrottle::on_successpub fn on_success(provider: &str)crates/vox-llm-egress/src/throttle.rs#L143-L145Record a success for provider’s throttle (additive recovery).
fnthrottle::retry_after_from_headerspub fn retry_after_from_headers(headers: &reqwest::header::HeaderMap) -> Option<Duration>crates/vox-llm-egress/src/throttle.rs#L148-L172Parse Retry-After (seconds) or X-RateLimit-Reset (epoch ms) into a wait.