Roberta Legal Portuguese

This repository provides the related resources to the paper RoBERTaLexPT: A Legal RoBERTa Model pretrained with deduplication for Portuguese.

[!TIP] Check out Roberta Legal Portuguese in 🤗 Collection!

Corpora

We compile two main corpora for pre-training: - LegalPT, a Portuguese legal corpus - CrawlPT, a Portuguese general corpus used for comparison.

| Corpus | Domain | Tokens (B) | Size (GiB) | |-----------------|:-------:|:----------:|:----------:| | LegalPT | Legal | 22.5 | 125.1 | | CrawlPT | | | | |      brWaC | General | 2.7 | 16.3 | |      CC100 (PT) | General | 8.4 | 49.1 | |     OSCAR-2301 (PT) | General | 18.1 | 97.8 |

Deduplication was done by using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022). We used 5-grams and a signature of size 256, considering two documents to be identical if their Jaccard Similarity exceeded 0.7.

• LegalPT (deduplicated)
• CrawlPT (deduplicated)

Datasets

PortuLex benchmark is a four-task benchmark designed to evaluate the quality and performance of language models in the Portuguese legal context.

| Dataset | Task | Train | Dev | Test | |---------------|------|-------|-------|-------| | RRI | CLS | 8.26k | 1.05k | 1.47k | | LeNER-Br | NER | 7.83k | 1.18k | 1,39k | | UlyssesNER-Br | NER | 3.28k | 489 | 524 | | FGV-STF | NER | 415 | 60 | 119 |

Models

Our model was pretrained in four different configurations: - Solely on BrWaC (RoBERTaTimbau_base). - Solely on the LegalPT corpus (RoBERTaLegalPT-base) - Solely on the CrawlPT corpus (RoBERTaCrawlPT-base) - Combining both corpora (RoBERTaLexPT-base)

Macro F1-Score (\%) for multiple models evaluated on PortuLex benchmark test splits:

| Model | LeNER | UlyNER-PL | FGV-STF | RRIP | Average (%) | |----------------------------------------------------------------------------|-----------|-----------------|-------------|:---------:|-----------------| | | | Coarse/Fine | Coarse | | | | BERTimbau-based | 88.34 | 86.39/83.83 | 79.34 | 82.34 | 83.78 | | BERTimbau-large | 88.64 | 87.77/84.74 | 79.71 | 83.79 | 84.60 | | Albertina-PT-BR-base | 89.26 | 86.35/84.63 | 79.30 | 81.16 | 83.80 | | Albertina-PT-BR-xlarge | 90.09 | 88.36/86.62 | 79.94 | 82.79 | 85.08 | | BERTikal-base | 83.68 | 79.21/75.70 | 77.73 | 81.11 | 79.99 | | JurisBERT-base | 81.74 | 81.67/77.97 | 76.04 | 80.85 | 79.61 | | BERTimbauLAW-base | 84.90 | 87.11/84.42 | 79.78 | 82.35 | 83.20 | | Legal-XLM-R-base | 87.48 | 83.49/83.16 | 79.79 | 82.35 | 83.24 | | Legal-XLM-R-large | 88.39 | 84.65/84.55 | 79.36 | 81.66 | 83.50 | | Legal-RoBERTa-PT-large | 87.96 | 88.32/84.83 | 79.57 | 81.98 | 84.02 | | Ours | | | | | | | RoBERTaTimbau-base (Reproduction of BERTimbau) | 89.68 | 87.53/85.74 | 78.82 | 82.03 | 84.29 | | RoBERTaLegalPT-base (Trained on LegalPT) | 90.59 | 85.45/84.40 | 79.92 | 82.84 | 84.57 | | RoBERTaCrawlPT-base (Trained on CrawlPT) | 89.24 | 88.22/86.58 | 79.88 | 82.80 | 84.83 | | RoBERTaLexPT-base (Trained on CrawlPT + LegalPT) | 90.73 | 88.56/86.03 | 80.40 | 83.22 | 85.41 |

In summary, RoBERTaLexPT consistently achieves top legal NLP effectiveness despite its base size. With sufficient pre-training data, it can surpass larger models. The results highlight the importance of domain-diverse training data over sheer model scale.

Citation

bibtex @inproceedings{garcia-etal-2024-robertalexpt, title = "{R}o{BERT}a{L}ex{PT}: A Legal {R}o{BERT}a Model pretrained with deduplication for {P}ortuguese", author = "Garcia, Eduardo A. S. and Silva, Nadia F. F. and Siqueira, Felipe and Albuquerque, Hidelberg O. and Gomes, Juliana R. S. and Souza, Ellen and Lima, Eliomar A.", editor = "Gamallo, Pablo and Claro, Daniela and Teixeira, Ant{\'o}nio and Real, Livy and Garcia, Marcos and Oliveira, Hugo Gon{\c{c}}alo and Amaro, Raquel", booktitle = "Proceedings of the 16th International Conference on Computational Processing of Portuguese", month = mar, year = "2024", address = "Santiago de Compostela, Galicia/Spain", publisher = "Association for Computational Lingustics", url = "https://aclanthology.org/2024.propor-1.38", pages = "374--383", }

Acknowledgment

This work has been supported by the AI Center of Excellence (Centro de Excelência em Inteligência Artificial – CEIA) of the Institute of Informatics at the Federal University of Goiás (INF-UFG).