FedQuant Phi-3-mini LoRA Adapter
A LoRA adapter for Phi-3-mini-4k-instruct, trained with FedQuant, a federated fine-tuning method for devices with different memory sizes. Domain: Indian legal NLP.
What is FedQuant?
- Each client gets a LoRA rank based on its RAM tier: r = 8, 16 or 32.
- The server projects all adapters to the lowest rank (r* = 8), then takes a weighted average.
- The averaged adapter is zero-padded back to each client's own rank.
Training setup
| Item | Value |
|---|---|
| Base model | Phi-3-mini-4k-instruct (3.8B) |
| Federated rounds | 12 |
| Training clients | 4 (client ranks 8, 8, 16, 32) |
| LoRA target layers | merged qkv_proj and output projection |
| LoRA alpha / dropout | 2r / 0.05 |
| Optimizer | AdamW, lr 2e-4 |
| Batch size | 1 (grad. accumulation 4) |
| Max sequence length | 256 tokens |
| Precision | BF16 |
| Hardware | 1x NVIDIA H100 |
| Training time | about 12.3 hours |
| Training data | 64,999 examples (10 legal tasks) |
Results (Round 12)
- Final training loss, highest-rank client (r = 32): 0.094
- Average perplexity over 10 tasks: 6.058
- Task metrics use 100 validation samples per task.
On-device test
This adapter was merged into the base model, converted to GGUF and quantized to INT4 (Q4_K_M, 2.3 GB). It ran on one Android phone (Snapdragon 720G, 5.5 GB RAM) at about 3.9 tokens per second with llama.cpp. Only inference was run on the phone. The GGUF file is not included in this repo.
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "microsoft/Phi-3-mini-4k-instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "nadeem172/fedquant-phi3-mini-lora")
Limitations
- Training data was generated using rule-based and template methods from Indian statutory text.
- Covers Indian statutory law only.
- Clients were simulated on one GPU, not on physical phones.
- Outputs are for information only. They are not legal advice and need human review.