Hugging Peers

phase1_qbase_internvid

☆ Save
by SamChen567 text-generation transformers
243 downloads 0 likes on Hugging Face updated

Model card

phase1_qbase_internvid (native-res, video-only, 2 epochs)

Phase 1 of the qbase video-token compressor curriculum: a learned-query compressor ("qbase") inserted between the frozen SigLIP-NaViT vision encoder and the frozen Qwen2-7B LLM of VideoLLaMA3-7B, trained with caption cross-entropy.

This repo was re-used. The repo name previously referred to an older Phase-1 run (16384-token video budget, 40 % still images, 3 epochs / 3,441 steps); no weights from that run were ever uploaded here. Everything in this repo is the run described below.

This checkpoint needs the custom code in the training repository (videollama3/model, Videollama3Qwen2ForCausalLM with token_compressor); it does not load with stock transformers alone.

What is trained

module status
vision encoder (SigLIP-NaViT) frozen
token_compressor (qbase, 170M) trained
mm_projector (mlp2x_gelu, 17M) trained
<|compression_start|> / <|compression_end|> embedding rows trained (all other rows frozen)
LLM (Qwen2-7B) + lm_head frozen

Warm start: SamChen567/compressor_pretrain_video_norm (qbase + projector + the two embedding rows).

Compressor

  • transformer_decoder_flat: 8 layers, 8 heads, hidden 1152, K = 64 learned queries (+ sin/cos position encoding); cross-attention onto the segment's encoder tokens with (t, h, w) RoPE.
  • Adaptive segmentation (non-learned): N = ⌊T/4⌋ + 1 segments per video, a forced cut every 8 frames, remaining cuts at the largest consecutive-frame cosine distance of the per-frame mean encoder feature. Each segment is 1–8 frames. Deterministic top-k (segment_sample_tau = 0).
  • Option A (match_encoder_scale): output rescaled to the encoder tokens' per-dim mean/std, then a learnable per-channel γ/β.
  • Option B (distr_loss_weight = 0.05): CORAL-style distribution-match auxiliary loss (see the note on its effective weight below).

Input layout seen by the LLM

Each segment carries its own time range (segment_time_marks = True); the per-frame Time X.0s: text of the uncompressed template is removed:

<|im_start|>user
Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ... 
{prompt}<|im_end|>
<|im_start|>assistant
{caption}<|im_end|>

Seconds are integers (first / last frame of the segment). Example: a 120 s clip at 1 fps → 31 segments → 1,984 compressed tokens (vs. 35,880 uncompressed at 640×360). Checkpoints trained before 2026-10-12 used one whole-video range instead; build the layout from this checkpoint's config.json.

Data

  • InternVid clips < 180 s, 264,186 samples, video only (no still images).
  • Captions re-generated by Qwen3-VL-8B-Instruct at 1 fps, native resolution, with several prompt styles (baseline_default, scene_shift, motion_scene, ...).
  • One turn per sample: <video>\n{prompt} → caption. Loss on the assistant caption only.

Vision geometry

  • fps = 1, max_frames = 180.
  • Native resolution: per-video budget 53,820 merged tokens (180 × 299), per-frame cap 2048, per-frame floor 64. 640×360 → 299 tokens/frame (patch 14, merge 2).

Training
epochs / steps 2 / 1,376
global batch 384 (2 per device × 8 GPUs × 24 grad-acc, flattened B = 1 sequences)
LR qbase 2e-5, projector 5e-6; 10 % warmup, cosine; weight decay 0
precision / parallel bf16, DeepSpeed ZeRO-1, gradient checkpointing
wall time 11.5 h

The exact launcher is included as launcher.sh (env overrides used for this run: EPOCHS=2 COMPRESSOR_LR=2e-5 PER_DEVICE_BATCH=2, data anno_data/internvid_qwen3vl_lt180.json). The full loss curve is in trainer_state.json.

Loss

steps logged loss ≈ per-token CE (×8, see note)
1–25 0.2312 1.85
664–688 (end of epoch 1) 0.1790 1.43
689–713 (start of epoch 2) 0.1779 1.42
1352–1376 (end) 0.1733 1.39

No NaN, no skipped samples; grad norm stayed in 0.15–0.33. These are training losses only — no held-out evaluation is reported here.

Notes on loss scaling (known quirk of the training code)

  • The custom compute_loss divides the CE sum by the token count gathered across all 8 GPUs but does not multiply back by the number of processes, so the logged loss is ≈ per-token CE / 8. AdamW is nearly scale-invariant, so this mostly affects the readout; with these small gradients max_grad_norm = 1.0 never clips.
  • The Option-B term is divided only by grad-accum steps, not by 8, so relative to CE its effective weight is ≈ 0.05 × 8 = 0.4, not the configured 0.05. Earlier Phase-1/2 runs from the same code share this behaviour.