Use with Ollama¶
Ollama loads one merged GGUF, not an adapter. Since v0.34.1 it rejects LoRA adapters with LoRA adapters are no longer supported (server/create.go), so the Modelfile ADAPTER line no longer works. astro-weights export --splice writes the merged file: a copy of a base GGUF with the adapter folded in.
You need:
ADAPTER: an adapter directory saved by astro-weights, for example byastro-weights build.BASE.gguf: the GGUF of the base model, the file you would run without the edit. It may be quantized.HF_DIR: the same base model as safetensors, a directory withmodel.safetensors, or withmodel.safetensors.index.jsonplus shards.- The
ggufextra.
Write the merged GGUF¶
pip install 'astro-weights[gguf]'
astro-weights export ADAPTER --splice BASE.gguf --base-model HF_DIR --merged OUT.gguf
Output:
- Each edited tensor is the base weight read from
HF_DIRplus the adapter's delta, stored as--out-type:q8_0(default),f16orbf16. Every other tensor is copied as raw bytes fromBASE.ggufand keeps its type, so a quantized base stays quantized. Metadata is copied too. q8_0needs a tensor whose row length is a multiple of 32. Any other length is an error (exit 2) and writes no file; use--out-type f16there.- Only
mlp.down_projandself_attn.o_projmodules map to GGUF tensors; see How-to: export GGUF and a model card. - When the adapter records base weight hashes, a weight in
HF_DIRthat does not match is an error (wrong base model?), and nothing is written. - An existing
OUT.ggufis refused unless--forceis given.OUT.ggufcannot beBASE.gguf, cannot sit insideADAPTER, and its directory must exist. - The merged file is verified against
BASE.ggufbefore it is moved into place. A failed check leaves no file. --jsonprintswritten,replaced(the GGUF tensor names) andout_type.
Load it in Ollama¶
Put a Modelfile next to the merged file:
Then create and run the model:
- Ollama does not quantize a GGUF on import (Ollama: importing a model). The merged file is used as written.
- The chat template must match the base model's. The merged file carries the metadata of
BASE.gguf, so a template stored there comes along; a wrong template in the Modelfile gives bad output. - LM Studio loads GGUF files too, so the same
OUT.ggufshould work there.
Not tested yet: the splice is tested on small synthetic GGUFs, including a base with Q8_0 and Q4_K tensors. A merged file from a Q4_K_M base has not been run end to end in Ollama or LM Studio.
Licence¶
The merged file contains the base model's weights, so the base model's licence applies to it. Check that licence before you share the file.
Keep the adapter small: llama.cpp¶
llama.cpp still loads the adapter on its own, with no merged copy of the model:
Write adapter.gguf with astro-weights export ADAPTER --gguf adapter.gguf --arch ARCH; see How-to: export GGUF and a model card.