Natural Portuguese Language Benchmark (Napolab)

The Napolab is your go-to collection of Portuguese datasets for the evaluation of Large Language Models.

Medium Article: "The Hidden Truth About LLM Performance: Why Your Benchmark Results Might Be Misleading"

Napolab Leaderboard

Browse the Napolab Leaderboard and stay up to date with the latest advancements in Portuguese language models.

Napolab for Large Language Models (LLMs)

A format of Napolab specifically designed for researchers experimenting with Large Language Models (LLMs) is now available. This format includes two main fields:

• Prompt: The input prompt to be fed into the LLM.
• Answer: The expected classification output label from the LLM, which is always a number between 0 and 5.

The dataset in this format can be accessed at https://huggingface.co/datasets/ruanchaves/napolab. If youve used Napolab for LLM evaluations, please share your findings with us!

Models

Nicholas Kluge, et al. have fine-tuned TeenyTinyLlama models on the FaQUAD-NLI and HateBR datasets from Napolab. For more information, please refer to the article "TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese".

We've made several models, fine-tuned on this benchmark, available on Hugging Face Hub:

| Datasets | mDeBERTa v3 | BERT Large | BERT Base | |:----------------------------:|:--------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------:| | ASSIN 2 - STS | Link | Link | Link | | ASSIN 2 - RTE | Link | Link | Link | | ASSIN - STS | Link | Link | Link | | ASSIN - RTE | Link | Link | Link | | HateBR | Link | Link | Link | | FaQUaD-NLI | Link | Link | Link | | PorSimplesSent | Link | Link | Link |

For model fine-tuning details and benchmark results, visit EVALUATION.md.

Guidelines

Napolab adopts the following guidelines for the inclusion of datasets:

• Natural: As much as possible, datasets consist of natural Portuguese text or professionally translated text.
• Reliable: Metrics correlate reliably with human judgments (accuracy, F1 score, Pearson correlation, etc.).
• Public: Every dataset is available through a public link.
• Human: Expert human annotations only. No automatic or unreliable annotations.
• General: No domain-specific knowledge or advanced preparation is needed to solve dataset tasks.

Napolab currently includes the following datasets:

| | | | | :---: | :---: | :---: | |assin | assin2 | rerelem| |hatebr| reli-sa | faquad-nli | |porsimplessent | | |

** Contribute**: We're open to expanding Napolab! Suggest additions in the issues. For more information, read our CONTRIBUTING.md.

For broader accessibility, all datasets have translations in Catalan, English, Galician and Spanish using the facebook/nllb-200-1.3B model via Easy-Translate.

Usage

To reproduce the Napolab benchmark available on the Hugging Face Hub locally, follow these steps:

1. Clone the repository and install the library:

bash git clone https://github.com/ruanchaves/napolab.git cd napolab pip install -e .

1. Generate the benchmark file:

python from napolab import export_napolab_benchmark, convert_to_completions_format input_df = export_napolab_benchmark() output_df = convert_to_completions_format(input_df) output_df.reset_index().to_csv("test.csv", index=False)

Citation

If you would like to cite our work or models, please reference the Master's thesis Lessons Learned from the Evaluation of Portuguese Language Models.

@mastersthesis{chaves2023lessons, title={Lessons learned from the evaluation of Portuguese language models}, author={Chaves Rodrigues, Ruan}, year={2023}, school={University of Malta}, url={https://www.um.edu.mt/library/oar/handle/123456789/120557} }

Disclaimer

The HateBR dataset, including all its components, is provided strictly for academic and research purposes. The use of the HateBR dataset for any commercial or non-academic purpose is expressly prohibited without the prior written consent of SINCH.