19 downloads
0 likes
on Hugging Face
updated
Model card
124M GPT with Symbolic Reasoning Distillation
A 124M-parameter GPT-2 trained from scratch on
FineWeb-Edu
with knowledge distillation from
SmolLM-135M-Instruct.
| Component |
Value |
| Parameters |
~124M |
| Layers |
12 |
| Heads |
12 |
| Embedding dim |
768 |
| Context |
512 |
| Loss |
0.5 CE + 0.5 KL |
| Hardware |
1x A100 |
| Time |
~75 min |
| Tokens |
327,680,000 |
| Best loss |
326.0111 |