DialFactSum: RoSE Benchmark Evaluation on Salesforce/rose Dataset

This repository presents the evaluation results of our DialFactSum models on the Salesforce/rose dataset under the RoSE (Revisiting, Organizing, and Structuring Evaluation for Dialogue Summarization) Benchmark.

We report the standard ACU and Normalized ACU metrics, demonstrating that our DialFactSum-ACU-8B (trained with GRPO) achieves state-of-the-art performance compared to existing strong baselines.
All baseline results have been re‑evaluated using GPT‑4o + G‑eval for a fair and unified comparison.

Dataset & Benchmark

• Dataset: Salesforce/rose
• Benchmark: RoSE Benchmark (Dialogue Summarization)
• Evaluation Metrics:
  • ACU (Atomic Content Unit): Measures the precision of atomic factual units in the summary.
  • Normalized ACU: A normalized version of ACU to account for length variations.
• Unified Evaluation Protocol: GPT‑4o with G‑eval (consistent across all models)

Model Performance & Comparison

We evaluate our models against a comprehensive set of dialogue summarization baselines. Our approaches leverage Qwen3-8B fine‑tuned with SFT (Supervised Fine‑Tuning) and GRPO (Group Relative Policy Optimization) .

Quantitative Results

Below is the full leaderboard comparison on the RoSE Benchmark test split, with all baseline scores obtained under the same GPT‑4o + G‑eval evaluator.

Visual Comparison

The figures below provide an intuitive comparison of model performance across the two metrics. Our DialFactSum-ACU-8B variants are highlighted in red, clearly demonstrating a substantial margin over all prior baselines.

1. ACU Score Comparison

2. Normalized ACU Score Comparison

Key Findings & Analysis

1. State‑of‑the‑Art Results: Our DialFactSum-ACU-8B models significantly outperform all previous baselines. The Base (T=0) configuration achieves the highest scores on both ACU (0.6947) and Normalized ACU (0.4649) .
2. GRPO vs. SFT: The models trained with GRPO exhibit a substantial performance leap over the standard SFT approach (+17.4 absolute points in ACU). This validates the effectiveness of reinforcement learning from preferences for improving factual consistency and content selection in dialogue summarization.
3. Comparison to Baselines (Unified GPT‑4o Evaluation): Our DialFactSum-ACU-8B model outperforms the previous best baseline, Ctrl‑DiaSumm, by 13.8 percentage points in ACU (0.6947 vs. 0.5568), demonstrating a clear and consistent advantage under the same evaluation protocol.