Hugging Peers

llama.cpp-clover-moe

☆ Save
by IsValorum llama.cpp mit
543 downloads 1 likes on Hugging Face updated

Model card

llama.cpp with Clover support (clover_moe)

Prebuilt Clover runtime by IsValorum

Support on Ko-fi

[!IMPORTANT] Independently computed using dedicated cloud compute clusters, funded 100% out-of-pocket.

If you consider these releases to be of high quality and they help you in your day-to-day workflow, consider supporting on Ko-fi. Every contribution directly sustains future cloud compute hours and helps support my university studies.

Prebuilt Linux x86_64 binaries for Diffuspace/Clover-35B-A3B-Preview, built from llama.cpp PR #31306 at commit 4838ffb0c8ade92f34d9157deea38d92758ab824.

Mainline llama.cpp does not load clover_moe. This build adds the converter, the runtime graph and the draft-self speculative mode. The PR is not merged upstream; pin this repository for reproducibility.

Compatibility

  • Model: Diffuspace/Clover-35B-A3B-Preview (clover_moe). Works with the BF16 and Q8_0 GGUFs and with third-party clover_moe GGUFs such as the IQ4_XS. It does not load the original safetensors.
  • Linux x86_64: the CPU package runs on any Linux; the CUDA package requires an NVIDIA GPU and CUDA 12.x runtime, with archs 80/86/89/90 (A100, H100/H200, L40S, RTX 4090, A10G). Blackwell 100/120 is not in this build.
  • Windows: no native binary here. Use WSL2 with the Linux CPU or CUDA package; a native Windows build must be compiled from the PR with CMake and MSVC. The GGUF files themselves are OS-agnostic.
  • macOS: no binary here; use a Metal build of the PR or the third-party IQ4_XS Mac build.
  • Frontends: not LM Studio or Ollama. Clover needs this clover_moe runtime until the PR merges upstream.
File Description
llama-clover-b4838ffb-linux-x64-cpu.tar.gz Portable CPU build
llama-clover-b4838ffb-linux-x64-cuda12.tar.gz CUDA build, archs 80;86;89;90
llama-clover-b4838ffb-converter.tar.gz HF-to-GGUF converter, conversion code and gguf-py
BUILD_INFO.json Compiler, flags and SHA256
SHA256SUMS Checksums

Usage

tar -xzf llama-clover-b4838ffb-linux-x64-cuda12.tar.gz
cd llama-clover-b4838ffb-linux-x64-cuda12
export LD_LIBRARY_PATH=$PWD/bin:$LD_LIBRARY_PATH
./bin/llama-cli -m clover-bf16.gguf -ngl 99 -fa on -c 4096 -p "Hello"
./bin/llama-speculative-simple -m clover-bf16.gguf --spec-type draft-self --spec-draft-n-max 15 -ctk bf16 -ctv bf16 -n 128 -c 2048 -b 256 -ub 256 -ngl 99

Conversion:

tar -xzf llama-clover-b4838ffb-converter.tar.gz
python llama-clover-converter/llama.cpp/convert_hf_to_gguf.py Diffuspace/Clover-35B-A3B-Preview --outtype bf16 --outfile clover-bf16.gguf

Quantization and importance matrix:

./bin/llama-quantize --allow-requantize clover-bf16.gguf clover-q8_0.gguf Q8_0
./bin/llama-imatrix -m clover-q8_0.gguf -f calibration.raw -o clover.imatrix.gguf --output-format gguf --process-output --no-ppl --no-escape -c 2048 -b 2048 -ub 2048 --chunks 611 -ngl 99

The CUDA package requires a compatible NVIDIA driver and CUDA 12.x runtime. The CPU package is built with GGML_NATIVE=OFF. llama.cpp is MIT licensed; the Clover support is PR #31306 by Maphist0; the Clover weights are Apache-2.0, Copyright DiffuSpace and the Qwen team.