MS-LongWriter-GLM4-9B-Chat

🤖 [LongWriter Dataset] • 💻 [Github Repo] • 📃 [LongWriter Paper]

MS-LongWriter-GLM4-9B-Chat is trained based on glm4-9b-chat, and is capable of generating 10,000+ words at once.

Compared to the work in the original LongWriter paper, MS-LongWriter-GLM4-9B-Chat begins training directly from the GLM4-9B-Chat model, while performing significant distillation on the LongWriter-6k to obtain 666 high-quality samples.

Datasets

1. LongWriter-6k-filtered, which is based on the LongWriter-6k
2. Magpie-Qwen2-Pro-200K-Chinese , random sampling 6k examples.
3. Magpie-Qwen2-Pro-200K-English , random sampling 6k examples.

Model

We use ms-swift to fine-tune the GLM4-9B-Chat model.

1. Installation

pip install ms-swift[llm]

1. Fine-tuning

Envs:

Nvidia A100(80G) x 4

Run:

swift sft \
    --model_type glm4-9b-chat \
    --dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \
    --max_length 28672 \
    --num_train_epochs 2 \
    --eval_steps 200 \
    --batch_size 1 \
    --gradient_accumulation_steps 64 \
    --gradient_checkpointing true \
    --warmup_ratio 0.1 \
    --learning_rate 1e-5 \
    --sft_type full \
    --loss_name long-ce \
    --check_dataset_strategy warning \
    --save_only_model false \
    --save_total_limit -1 \
    --lazy_tokenize true \
    --dataloader_num_workers 1 \
    --resume_only_model true \
    --neftune_noise_alpha

Evaluation

Refer to LongWriter Evaluation from the EvalScope.