hf2q — local model conversion and inference
Open, Uncensored, & Local — Sovereign AI
Hugging Face to Quant (hf2q) converts and serves models, optimized for you.
Rust + Apple Silicon, channeling intelligence from your machine.
Install
Choose the standalone installer, Cargo, or a source build. Every option shown here is ready to use.
Standalone
curl -fsSL https://hf2q.us/install.sh | shSource
git clone https://github.com/robertelee78/hf2q.git
cd hf2q
git checkout 64a08cb3d7ca46e1a780ce22bcaa3cdff3def750
GIT_COMMIT_SHA="$(git rev-parse HEAD)" cargo build --release --locked
export PATH="$PWD/target/release:$PATH"Cargo
cargo install hf2q --version 0.1.20 --lockedThe artifact rail
One continuous line from exact pinned source weights to a local streamed response. Conversion and runtime stay inside hf2q's owned path.
Source weights
Source bytes from a supported Hugging Face model family enter the boundary.
hf2q conversion
Verified mapping and quantization run inside hf2q's owned Rust path.
GGUF artifact
A compact, reusable GGUF file is written to disk.
Owned Metal runtime
The artifact docks into mlx-native — pure-Rust Metal kernels on Apple Silicon.
Local response
A streamed, OpenAI-compatible response leaves the local endpoint.
Guarantees
What hf2q owns, and what it refuses to outsource — built for agent loops, not just chat.
Owned conversion
Weights are mapped and quantized inside hf2q's Rust path. No third-party conversion service handles your model.
Owned runtime
Inference runs in hf2q's Metal runtime on Apple Silicon. No external server sits between you and the response.
Fail-closed support
Unsupported model families are refused up front rather than silently mis-served.
Enforced tool calls
Tool calls and JSON output are grammar-constrained at decode time and streamed as OpenAI-style deltas. A malformed call fails closed instead of leaking into the text.
Parallel agents, full context
Inflight batching serves independent agent slots, and every slot gets the model's full context — never a share divided by the slot count. KV, template, and tool-call state stay isolated per conversation.
Prefixes that survive tool turns
Agent turns reuse the live KV prefix instead of re-prefilling the whole transcript, concurrent agents share a cached system prompt, and every response reports its cached-token count.
Deterministic replays
Sampling is a pure function of seed and decode step, so scheduling and thread assignment cannot perturb a run. Agent traces reproduce exactly.
No meter running
Localhost by default, no telemetry, no quota or billing layer. A 429 only ever means the box is busy right now.
Run models locally
Choose one of two model-preparation paths: download a compatible GGUF from Hugging Face, or create your own from source with hf2q. Both paths end at the same local OpenAI-compatible endpoint. The getting-started guide uses Qwen3.8 as one complete example.
Option A · Download a GGUF
Use a compatible pre-quantized GGUF from Hugging Face. No conversion is needed.
curl -fL <hf-model-url> -o model.ggufOption B · Quantize from source
Give hf2q a supported Hugging Face model ID. It downloads and verifies the source, then produces a GGUF using your setup default.
hf2q convert <owner>/<model> --output model.ggufThen · Serve and chat
Whichever path you choose, serve the resulting GGUF on localhost, then chat with it from the terminal.
hf2q chat