MediBerta: a medieval Latin encoder for editorial and analytical work on legal and religious texts

MediBerta is XLM-RoBERTa large with continued pre-training (masked language modelling) on about 143 million tokens of medieval Latin, predominantly legal and ecclesiastical prose. The model is intended as an encoder backbone for tasks in the digital editing and analysis of such texts. These include the restoration of sentence and paragraph boundaries in unpunctuated transcriptions, the reconstruction of text flow and initials, the tagging of citations and named entities, and retrieval. The etp-latin-legal-references-ner tagger is built on MediBerta. The companion model CanonBerta-v1 was trained on the same documents with the canon-law material upsampled.

The model was developed as part of the LOEWE project "Embedding the Past" (LOEWE-Exploration, TU Darmstadt, AdWL|Mainz), funded by HMWK Hessen, PI Prof. Dr. Michael Schonhardt. It was first presented at the International Medieval Congress, Leeds, on 8 July 2026 ("CanonBERT(a): Large Language Models for Editing and Analysing Medieval Canon Law Collections").

Training data

The training text is drawn from openly licensed corpora and public-domain editions. It was lower-cased and stripped of punctuation. This corresponds to the form in which transcriptions of manuscripts and early prints are usually available and in which the model is intended to be applied.

The model weights are released under the Apache License 2.0. The source texts retain their own licences and are not distributed with the model. For this reason the prepared training set on the Hub (mschonhardt/MedBerta) remains private.

MediBerta is the baseline run, in which each sequence occurs once and no corpus is upsampled. The prepared set comprises 291,630 sequences of 512 tokens (149.3 million tokens) from 14,730 documents. Of these, 236,446 sequences from 14,670 documents are general medieval Latin, 53,240 sequences come from the 47 large works of the School of Salamanca, and 1,944 sequences from the core canon-law collections. One per cent of the documents, stratified by category, was held out as whole documents. This leaves 286,062 sequences (142.8 million tokens) for training and 5,568 sequences (2.8 million tokens) for validation. CanonBerta-v1 was trained on the same documents, with the canon-law, papal and charter texts repeated two to four times.

Training

Continued pre-training (domain-adaptive pre-training) of FacebookAI/xlm-roberta-large with masked language modelling, using the script train.py in this repository (Hugging Face Trainer).

The training script (train.py) and the logs are included in this repository: train.log, training_log.jsonl, trainer_state.json, run_metadata.json (full configuration), dataset_stats.json, tokenizer_diagnostics.json and the TensorBoard event files under runs/.

Evaluation

Intrinsic evaluation on the held-out documents after epoch 10 yields a validation loss of 1.048, a perplexity of 2.85 and a masked-token accuracy of 0.770.

The downstream evaluation concerns segmentation, the task for which the model was first developed. Each encoder was fine-tuned with the same segmentation head (SegmentBerta), which restores sentence and paragraph boundaries in unpunctuated, lower-cased medieval Latin. Evaluation used sliding windows on a held-out set.

The results suggest that the task rewards adaptation to editorial conventions, that is, to the places where editors set boundaries, more than adaptation to the language of the sources. The training data consist of editions. As a backbone for citation and entity tagging the model performs well: the reference tagger built on it reaches a span F1 of 0.83.

Use

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("mschonhardt/MediBerta")
model = AutoModelForMaskedLM.from_pretrained("mschonhardt/MediBerta")

Input should be lower-cased medieval Latin without punctuation, in line with the pre-training data. The tokenizer is the unchanged XLM-R SentencePiece model. For token or sequence classification, the model is loaded with the corresponding AutoModelFor… head and fine-tuned.

Limitations

The model was trained on editions and a small number of prints, not on diplomatic transcriptions. Abbreviations, scribal spellings and OCR noise are therefore underrepresented. Since the pre-training text was lower-cased and unpunctuated, the model has not encountered capitals or punctuation in this domain. Legal and ecclesiastical prose predominates.

Environmental impact

The figure is an estimate: 9.5 GPU hours on a 600 W card at an assumed average draw of 400 W, multiplied by a factor of 1.4 for server and cooling, amount to about 5.3 kWh. At the German grid average of about 0.38 kg CO2e per kWh, this corresponds to roughly 2.9 kg CO2e.

Citation

Schonhardt, Michael: MediBerta: a medieval Latin encoder for editorial and analytical work on legal and religious texts. Embedding the Past (LOEWE-Exploration, TU Darmstadt, AdWL|Mainz), 2026. https://huggingface.co/mschonhardt/MediBerta

@misc{schonhardt2026mediberta,
  author       = {Schonhardt, Michael},
  title        = {MediBerta: a medieval Latin encoder for editorial and analytical work on legal and religious texts},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/mschonhardt/MediBerta}},
  note         = {Developed in the project Embedding the Past (LOEWE-Exploration, TU Darmstadt, AdWL|Mainz), funded by HMWK Hessen}
}

Related: CanonBerta-v1 (companion model weighted towards canon law), etp-latin-legal-references-ner, FacebookAI/xlm-roberta-large.