Tailoring RAG Strategies for Industrial Protocols: A Comparative Study on PROFIBUS Document Retrieval using Gemma and GPT Models

✍🏾 Authors: Thaís Medeiros, Morsinaldo Medeiros, Matheus Andrade, Marianne Diniz, Ivanovitch Silva, Massimiliano Gaffurini, Dennis Brandão, Paolo Ferrari

📚 Overview

This repository contains the experimental study conducted for the paper "Tailoring RAG Strategies for Industrial Protocols: A Comparative Study on PROFIBUS Document Retrieval using Gemma and GPT Models."

The study investigates the effectiveness of different Retrieval-Augmented Generation (RAG) strategies — Base, Decomposition, and HyDE — combined with two Large Language Models (LLMs): - Gemma 3 (12B parameters) — open-source - GPT-4o-mini — proprietary (OpenAI)

The application domain is industrial automation, specifically focusing on technical documents related to the PROFIBUS communication protocol. The documents were organized into two categories: - Developer Documents - Engineering Documents

A set of 20 questions of varying complexity was formulated to assess the retrieval and generation capabilities of different RAG strategies and models. The outputs were compared against reference answers generated by NotebookLM and evaluated using automatic metrics such as ROUGE, METEOR, BERTScore, and MATTR.

🗂️ Repository Structure

text ETFA2025-Industrial-RAG/ ├── RAG_Dev/ # Developer documents (Gemma model) │ ├── 2_RAG_Base.ipynb │ ├── 3_RAG_Decomposition.ipynb │ └── 4_RAG_HyDE.ipynb ├── RAG_Dev_gpt/ # Developer documents (GPT model) │ ├── 2_RAG_Base.ipynb │ ├── 3_RAG_Decomposition.ipynb │ └── 4_RAG_HyDE.ipynb ├── RAG_Eng/ # Engineer documents (Gemma model) │ ├── 2_RAG_Base.ipynb │ ├── 3_RAG_Decomposition.ipynb │ └── 4_RAG_HyDE.ipynb ├── RAG_Eng_gpt/ # Engineer documents (GPT model) │ ├── 2_RAG_Base.ipynb │ ├── 3_RAG_Decomposition.ipynb │ └── 4_RAG_HyDE.ipynb ├── Validation_Results/ # Resulting DataFrames from `5_Validation_notebookLM_Sample.ipynb` ├── 1_Text_Extraction.ipynb # Text extraction ├── 5_Validation_notebookLM_Sample.ipynb # Validation against NotebookLM references ├── 6_Analysis_Sample.ipynb # Analysis of results ├── requirements.txt # Python dependencies └── README.md # Project overview and usage

📝 Questions

The evaluation involved 20 manually formulated questions covering different complexity levels within PROFIBUS technical documentation, distributed across the Developer and Engineer categories. The full set of questions is organized below:

Developer Category

1. What is a GSD file, where can you get it, and what is its function?
2. What is a token message and how and which devices use it?
3. Which protocol parameter describes the slave's action time after receiving a message?
4. How to calculate DP cycle time?
5. Briefly explain the function of a watchdog timer in a PROFIBUS DP slave.
6. Briefly explain the meaning of “sync” and “freeze” modes. Which device(s) must support these modes?
7. Discuss the factors that have a significant effect on the overall cycle time of a DP network.
8. What are the four operating modes of a DP class 1 master? Briefly explain the interaction between the master and its assigned slaves in each mode.
9. The data unit of a configuration telegram contains the following 3 bytes represented in hexadecimal notation. Decode the meaning of the bytes. \n- Byte 0: D1 \n- Byte 1: 23 \n- Byte 2: 70
10. Briefly explain how a slave device, which is in cyclic data exchange, communicates the presence of a diagnostic fault to its controlling master. What does the master do about this?

Engineering Category

1. List the main causes of reflections in Profibus cables.
2. What are the different interference coupling mechanisms in Profibus cables? What measures can be taken to reduce each type of interference?
3. In which situations should spur lines not be used in Profibus DP network topology?
4. Determine the minimum separation distance between a Profibus cable and the following cables: a. AS-Interface type Profibus cable b. 24V power cable for a digital output module c. Single-phase 240V AC power cable for supplying a DC power supply d. Grounding equipotential bonding cable between cabinets e. Three-phase 380V power cable for a Profibus frequency inverter
5. Two control rooms located at opposite ends of an area must share the same Profibus network. Each room has its own grounding system, and there is a significant potential difference between them. List three methods to install the network in a way that avoids issues caused by separate ground potentials.
6. Discuss the factors that have a significant effect on the overall cycle time of a DP network.
7. A PA segment is used to wire 24 slave devices in a non-hazardous area. Determine the maximum spur length when: a. Each spur carries only one slave, b. Each spur carries four slaves. What is the maximum length of the main trunk cable in each case?
8. Does PA wiring have to use screened twisted pair cable?
9. A non-hazardous plant is currently fitted with 4 to 20 mA instrumentation, wired using a mixture of single and multi-core cable. What are the main considerations when considering moving over to a PROFIBUS PA system?
10. Determine the maximum trunk and spur cable lengths for a non-intrinsically safe MBP segment with 20 slaves each connected via the maximum possible spur length. Estimate the required coupler current rating if each slave device takes a maximum of 14 mA.

📝 Sample Questions

Two representative examples are presented as follows, including generated answers and reference responses.

Q1. What is a GSD file, where can you get it, and what is its function? (Developer)

• Generated Answer (Base + Gemma - Developer):
  ```text A GSD file (German: Geräte-Stücklisten Datei, meaning "device description file") is a text file that describes a PROFIBUS slave device to a master controller.

Where to get it: GSD files are provided by the manufacturer of the PROFIBUS slave device. They are typically available for download from the manufacturer's website or included on the device's documentation.

Function: The GSD file contains information such as:

• Device name and manufacturer
• Module type and hardware revision
• Input/Output data block definitions (data types, addresses, etc.)
• Parameterization options
• Communication parameters

The PROFIBUS master uses this information to correctly configure and communicate with the slave device. Without a GSD file, the master would not know how to interpret the data from the slave or how to configure its parameters. ```

• Generated Answer (Base + GPT - Developer):
  ```text A GSD file, or General Station Description file, is a crucial component in the PROFIBUS protocol that provides detailed information about a device's capabilities and parameters. It serves as a standardized electronic data sheet that allows for the configuration and integration of devices from different manufacturers within a PROFIBUS network.

Function of a GSD File:

1. Device Description: It contains essential information such as vendor and device names, hardware and software versions, supported baud rates, and signal assignments.
2. Configuration Support: The GSD file enables the configuration of PROFIBUS networks by providing the necessary parameters for both master and slave devices. This includes details on the maximum number of slaves that can be connected to a master and the specific I/O channels available on slave devices.
3. Diagnostic Information: It includes diagnostic texts and information about available modules, which is particularly useful for troubleshooting and maintenance.
4. Flexibility: The format allows for both mandatory and optional parameters, making it adaptable to various device types and functionalities.

Where to Obtain a GSD File:

GSD files can typically be obtained from the device manufacturer’s website or through their technical support. Manufacturers usually provide GSD files for their devices as part of the product documentation, ensuring that users can easily access the necessary files for network configuration.

In summary, GSD files are essential for the effective integration and operation of devices in a PROFIBUS network, facilitating a manufacturer-independent configuration process. ```

• Reference Answer (NotebookLM - Developer):
  ```text A General Station Description (GSD) file is a specially formatted ASCII file that serves as an electronic device data sheet for PROFIBUS devices. It is also referred to as a "Communication Feature List". You can typically obtain the GSD file from the device manufacturer of a DP-Slave or a DP-Master. In the case of Profile GSD files, which are defined by the PA profile, they are made available by PNO (PROFIBUS and PROFINET International) and can be found on the PI web server. The function of a GSD file is multifaceted:

  • It provides configuration tools with information about a PROFIBUS device. This includes:
    • Information needed to identify the connected device.
    • A description of device data that can be accessed via the network, such as configurable parameters.
    • A description of the communication capabilities supported by the device, such as transmission rate.
    • Additional vendor-specific information.
  • It allows a configuration tool to automate the device configuration process.
  • It provides an open, consistent, and compatible approach for performing device configuration. All devices with a communication interface according to IEC 61784-1 CP3/1 and CP3/2 shall have a GSD file.
  • Its main intention is to provide device information on a PROFIBUS communication network.
  • It characterizes the features and performance capabilities of PROFIBUS devices.
  • It enables the user to check all data in the configuration phase of a PROFIBUS system, allowing errors to be avoided as early as possible.
  • Based on its defined file format, it makes it possible to realize vendor-independent configuration tools for PROFIBUS systems.
  • The configuration tool uses the GSD files for testing the data entered regarding the observance of limits and validity related to the performance of the individual device.
  • The distinction of the GSD files is achieved by the vendor- and device-identifiers.
  • It aims to achieve a simple Plug and Play configuration for PROFIBUS devices by listing the communication features.
  • It comprises definitions for the physical layer, the protocol layer, the modeling of the slave / IO device, startup parameterization of the communication and application layers.
  • A Profile-GSD allows for device replacement without changing the network configuration of the automation system, as the cyclic data exchange is not impacted.
  • Via the GSD file, cyclic communication related device functionality is described. In essence, the GSD file acts as a device-specific data sheet in a machine-readable format, enabling seamless integration and configuration of PROFIBUS devices from various manufacturers within a PROFIBUS network. ``` ---

Q1. List the main causes of reflections in Profibus cables. (Eng)

• Generated Answer (HyDE + Gemma - Eng):
  ```text The main causes of reflections in PROFIBUS cables, according to the provided documentation, are:

1. PROFIBUS cable too long: Excessively long cables act like condensers, distorting the signal (rounding rising edges) and potentially preventing the receiver from correctly recognizing it.
2. Non-Connected PROFIBUS Plugs: Unused PROFIBUS plugs with integrated inductors generate signal interference if not connected to a station.
3. Defective Bus Terminator: This includes having too many terminators active (resulting in a 180° reflected signal) or a missing terminator (resulting in a non-inverted reflection). A break in the data lines also produces a similar effect. ```

• Generated Answer (HyDE + GPT - Eng):
  ```text The main causes of reflections in Profibus cables are:

1. Cable too long: Excessive length can alter the signal shape, leading to recognition issues by the receiver.
2. Non-connected Profibus plugs: Open connectors can create signal interference due to integrated inductors.
3. Defective bus terminator: Errors such as too many or missing terminators can cause signal reflections, either inverting or maintaining the signal's amplitude.

These factors can disrupt communication and signal integrity in Profibus networks. ```

• Reference Answer (NotebookLM - Eng):
  ```text The main causes of reflections in PROFIBUS cables, according to the sources, are:

• Cable faults such as cable breaks, bad contacts, and damaged PROFIBUS cables.

• Faulty terminating resistors.

• Stubs. Stubs are not permitted in RS485 segments.

• Defective bus terminators, which can occur if too many terminators are switched on or if a bus terminator is missing. In both cases, signal reflections are generated.

• The assumed capacitive load of the cable, which is why series inductances are integrated into the connector plate for cable termination according to IEC 61158-2.

• Improper or absent bus terminations at the ends of each PROFIBUS segment. Bus terminations are necessary for good signal transmission and to avoid reflections.

Reflections can corrupt PROFIBUS telegrams and are a sign of problems within the PROFIBUS system. Checking for reflections can be done with a handheld device during commissioning. ```

📊 Evaluation Data

The following tables present the results according to the average performance, considering all 10 questions applied to each model and technique, and the statistical test results comparing the techniques pairwise. The mean values provide a general overview of the models’ behavior across multiple evaluation metrics, while the statistical analysis — including p-values and Cohen’s d effect sizes — identifies whether the observed differences between techniques are statistically significant and quantifies the magnitude of these differences.

Developer - Average

| model | tech | rouge1-P | rouge1-R | rouge1-F1 | rouge2-P | rouge2-R | rouge2-F1 | rougeL-P | rougeL-R | rougeL-F1 | METEOR | BERTScore Precision | BERTScore Recall | BERTScore F1 | MATTR | |:--------|:--------------|-----------:|-----------:|------------:|-----------:|-----------:|------------:|-----------:|-----------:|------------:|---------:|----------------------:|-------------------:|---------------:|--------:| | GPT | Base | 0.3501 | 0.6883 | 0.4518 | 0.133 | 0.2584 | 0.1705 | 0.1784 | 0.3562 | 0.2308 | 0.358 | 0.8632 | 0.8925 | 0.8775 | 0.7392 | | GPT | Decomposition | 0.2742 | 0.6756 | 0.388 | 0.0952 | 0.2359 | 0.135 | 0.1425 | 0.3526 | 0.2019 | 0.3282 | 0.8527 | 0.8887 | 0.8703 | 0.7392 | | GPT | HyDE | 0.7062 | 0.3677 | 0.4746 | 0.2778 | 0.1464 | 0.1881 | 0.3566 | 0.1851 | 0.239 | 0.222 | 0.8948 | 0.8662 | 0.8802 | 0.7457 | | Gemma | Base | 0.6209 | 0.1888 | 0.2719 | 0.2105 | 0.0587 | 0.0867 | 0.3651 | 0.1024 | 0.1506 | 0.1165 | 0.881 | 0.8333 | 0.8562 | 0.7582 | | Gemma | Decomposition | 0.5569 | 0.3712 | 0.4336 | 0.1499 | 0.1019 | 0.118 | 0.2634 | 0.1724 | 0.2028 | 0.2099 | 0.8727 | 0.86 | 0.8662 | 0.7579 | | Gemma | HyDE | 0.5706 | 0.2417 | 0.3174 | 0.1667 | 0.0713 | 0.0941 | 0.3134 | 0.1256 | 0.1677 | 0.1469 | 0.878 | 0.8421 | 0.8595 | 0.7669 |

Developer - Statistical Test

Gemma

| metric | techA | techB | pvalue | cohend | |:--------------------|:--------------|:--------------|----------:|----------:| | rouge1-P | Base | Decomposition | 0.1244 | 0.54 | | rouge1-R | Base | Decomposition | 0.0031 | -1.26 | | rouge1-F1 | Base | Decomposition | 0.001 | -1.5 | | rouge2-P | Base | Decomposition | 0.0294 | 0.82 | | rouge2-R | Base | Decomposition | 0.0169 | -0.93 | | rouge2-F1 | Base | Decomposition | 0.0402 | -0.76 | | rougeL-P | Base | Decomposition | 0.02 | 0.89 | | rougeL-R | Base | Decomposition | 0.0048 | -1.18 | | rougeL-F1 | Base | Decomposition | 0.0056 | -1.14 | | METEOR | Base | Decomposition | 0.0059 | -1.13 | | BERTScore Precision | Base | Decomposition | 0.138 | 0.51 | | BERTScore Recall | Base | Decomposition | 0.0112 | -1 | | BERTScore F1 | Base | Decomposition | 0.0463 | -0.73 | | MATTR | Base | Decomposition | 0.9791 | 0.01 | | rouge1-P | Base | HyDE | 0.0546 | 0.7 | | rouge1-R | Base | HyDE | 0.2087 | -0.43 | | rouge1-F1 | Base | HyDE | 0.2715 | -0.37 | | rouge2-P | Base | HyDE | 0.0457 | 0.73 | | rouge2-R | Base | HyDE | 0.3941 | -0.28 | | rouge2-F1 | Base | HyDE | 0.6326 | -0.16 | | rougeL-P | Base | HyDE | 0.0731 | 0.64 | | rougeL-R | Base | HyDE | 0.2061 | -0.43 | | rougeL-F1 | Base | HyDE | 0.2941 | -0.35 | | METEOR | Base | HyDE | 0.2034 | -0.43 | | BERTScore Precision | Base | HyDE | 0.3627 | 0.3 | | BERTScore Recall | Base | HyDE | 0.1951 | -0.44 | | BERTScore F1 | Base | HyDE | 0.3522 | -0.31 | | MATTR | Base | HyDE | 0.6487 | -0.15 | | rouge1-P | Decomposition | HyDE | 0.6216 | -0.16 | | rouge1-R | Decomposition | HyDE | 0.0304 | 0.81 | | rouge1-F1 | Decomposition | HyDE | 0.0333 | 0.79 | | rouge2-P | Decomposition | HyDE | 0.4899 | -0.23 | | rouge2-R | Decomposition | HyDE | 0.0933 | 0.59 | | rouge2-F1 | Decomposition | HyDE | 0.188 | 0.45 | | rougeL-P | Decomposition | HyDE | 0.0976 | -0.58 | | rougeL-R | Decomposition | HyDE | 0.0357 | 0.78 | | rougeL-F1 | Decomposition | HyDE | 0.0873 | 0.61 | | METEOR | Decomposition | HyDE | 0.0666 | 0.66 | | BERTScore Precision | Decomposition | HyDE | 0.2722 | -0.37 | | BERTScore Recall | Decomposition | HyDE | 0.0663 | 0.66 | | BERTScore F1 | Decomposition | HyDE | 0.2317 | 0.41 | | MATTR | Decomposition | HyDE | 0.6756 | -0.14 |

GPT

| metric | techA | techB | pvalue | cohend | |:--------------------|:--------------|:--------------|----------:|----------:| | rouge1-P | Base | Decomposition | 0.0394 | 0.76 | | rouge1-R | Base | Decomposition | 0.485 | 0.23 | | rouge1-F1 | Base | Decomposition | 0.0325 | 0.8 | | rouge2-P | Base | Decomposition | 0.0621 | 0.67 | | rouge2-R | Base | Decomposition | 0.1416 | 0.51 | | rouge2-F1 | Base | Decomposition | 0.0541 | 0.7 | | rougeL-P | Base | Decomposition | 0.0597 | 0.68 | | rougeL-R | Base | Decomposition | 0.7699 | 0.1 | | rougeL-F1 | Base | Decomposition | 0.057 | 0.69 | | METEOR | Base | Decomposition | 0.0149 | 0.95 | | BERTScore Precision | Base | Decomposition | 0.0371 | 0.77 | | BERTScore Recall | Base | Decomposition | 0.2032 | 0.43 | | BERTScore F1 | Base | Decomposition | 0.0408 | 0.75 | | MATTR | Base | Decomposition | nan | nan | | rouge1-P | Base | HyDE | 0 | -2.53 | | rouge1-R | Base | HyDE | 0 | 2.52 | | rouge1-F1 | Base | HyDE | 0.1633 | -0.48 | | rouge2-P | Base | HyDE | 0.0001 | -2 | | rouge2-R | Base | HyDE | 0.0001 | 2.1 | | rouge2-F1 | Base | HyDE | 0.103 | -0.57 | | rougeL-P | Base | HyDE | 0 | -2.36 | | rougeL-R | Base | HyDE | 0.0001 | 2.05 | | rougeL-F1 | Base | HyDE | 0.3206 | -0.33 | | METEOR | Base | HyDE | 0 | 2.97 | | BERTScore Precision | Base | HyDE | 0.0001 | -2.14 | | BERTScore Recall | Base | HyDE | 0.0016 | 1.41 | | BERTScore F1 | Base | HyDE | 0.1182 | -0.55 | | MATTR | Base | HyDE | 0.5407 | -0.2 | | rouge1-P | Decomposition | HyDE | 0 | -4.93 | | rouge1-R | Decomposition | HyDE | 0 | 3.06 | | rouge1-F1 | Decomposition | HyDE | 0.0096 | -1.04 | | rouge2-P | Decomposition | HyDE | 0.0001 | -2.22 | | rouge2-R | Decomposition | HyDE | 0.0008 | 1.55 | | rouge2-F1 | Decomposition | HyDE | 0.0292 | -0.82 | | rougeL-P | Decomposition | HyDE | 0 | -3.59 | | rougeL-R | Decomposition | HyDE | 0 | 2.65 | | rougeL-F1 | Decomposition | HyDE | 0.0497 | -0.72 | | METEOR | Decomposition | HyDE | 0.0001 | 2.18 | | BERTScore Precision | Decomposition | HyDE | 0 | -3.47 | | BERTScore Recall | Decomposition | HyDE | 0.0058 | 1.14 | | BERTScore F1 | Decomposition | HyDE | 0.0141 | -0.96 | | MATTR | Decomposition | HyDE | 0.5407 | -0.2 |

Engineering - Average

| model | tech | rouge1-P | rouge1-R | rouge1-F1 | rouge2-P | rouge2-R | rouge2-F1 | rougeL-P | rougeL-R | rougeL-F1 | METEOR | BERTScore Precision | BERTScore Recall | BERTScore F1 | MATTR | |:--------|:--------------|-----------:|-----------:|------------:|-----------:|-----------:|------------:|-----------:|-----------:|------------:|---------:|----------------------:|-------------------:|---------------:|--------:| | GPT | Base | 0.6834 | 0.3678 | 0.4514 | 0.3068 | 0.1595 | 0.1982 | 0.3747 | 0.2031 | 0.2483 | 0.2357 | 0.9011 | 0.8784 | 0.8895 | 0.7763 | | GPT | Decomposition | 0.6282 | 0.34 | 0.4066 | 0.2675 | 0.1316 | 0.1656 | 0.3323 | 0.1805 | 0.2155 | 0.2169 | 0.8875 | 0.8704 | 0.8787 | 0.7997 | | GPT | HyDE | 0.8065 | 0.1156 | 0.1913 | 0.3945 | 0.0504 | 0.0855 | 0.5916 | 0.0816 | 0.1363 | 0.0816 | 0.9017 | 0.8201 | 0.8588 | 0.8087 | | Gemma | Base | 0.3215 | 0.6393 | 0.4158 | 0.1292 | 0.2623 | 0.1684 | 0.1769 | 0.3506 | 0.2289 | 0.3411 | 0.872 | 0.8907 | 0.8812 | 0.7643 | | Gemma | Decomposition | 0.5267 | 0.4626 | 0.4671 | 0.1789 | 0.1563 | 0.1592 | 0.2336 | 0.2125 | 0.2109 | 0.2709 | 0.8769 | 0.8781 | 0.8774 | 0.8092 | | Gemma | HyDE | 0.5976 | 0.3188 | 0.4032 | 0.2018 | 0.1038 | 0.1326 | 0.3002 | 0.1622 | 0.2042 | 0.1943 | 0.8857 | 0.8689 | 0.8772 | 0.8015 |

Engineering - Statistical Test

Gemma

| metric | techA | techB | pvalue | cohend | |:--------------------|:--------------|:--------------|----------:|----------:| | rouge1-P | Base | Decomposition | 0.0074 | -1.09 | | rouge1-R | Base | Decomposition | 0.032 | 0.8 | | rouge1-F1 | Base | Decomposition | 0.0371 | -0.77 | | rouge2-P | Base | Decomposition | 0.0441 | -0.74 | | rouge2-R | Base | Decomposition | 0.0174 | 0.92 | | rouge2-F1 | Base | Decomposition | 0.5635 | 0.19 | | rougeL-P | Base | Decomposition | 0.0512 | -0.71 | | rougeL-R | Base | Decomposition | 0.0028 | 1.28 | | rougeL-F1 | Base | Decomposition | 0.0209 | 0.88 | | METEOR | Base | Decomposition | 0.0237 | 0.86 | | BERTScore Precision | Base | Decomposition | 0.3803 | -0.29 | | BERTScore Recall | Base | Decomposition | 0.0656 | 0.66 | | BERTScore F1 | Base | Decomposition | 0.3525 | 0.31 | | MATTR | Base | Decomposition | 0.0108 | -1.01 | | rouge1-P | Base | HyDE | 0.0009 | -1.53 | | rouge1-R | Base | HyDE | 0.0002 | 1.91 | | rouge1-F1 | Base | HyDE | 0.5644 | 0.19 | | rouge2-P | Base | HyDE | 0.0395 | -0.76 | | rouge2-R | Base | HyDE | 0.002 | 1.35 | | rouge2-F1 | Base | HyDE | 0.1213 | 0.54 | | rougeL-P | Base | HyDE | 0.0024 | -1.32 | | rougeL-R | Base | HyDE | 0.0002 | 1.9 | | rougeL-F1 | Base | HyDE | 0.0636 | 0.67 | | METEOR | Base | HyDE | 0.0001 | 2.09 | | BERTScore Precision | Base | HyDE | 0.0451 | -0.74 | | BERTScore Recall | Base | HyDE | 0.0043 | 1.2 | | BERTScore F1 | Base | HyDE | 0.3083 | 0.34 | | MATTR | Base | HyDE | 0.0014 | -1.44 | | rouge1-P | Decomposition | HyDE | 0.0514 | -0.71 | | rouge1-R | Decomposition | HyDE | 0.0006 | 1.65 | | rouge1-F1 | Decomposition | HyDE | 0.0013 | 1.46 | | rouge2-P | Decomposition | HyDE | 0.3334 | -0.32 | | rouge2-R | Decomposition | HyDE | 0.0044 | 1.19 | | rouge2-F1 | Decomposition | HyDE | 0.0862 | 0.61 | | rougeL-P | Decomposition | HyDE | 0.0216 | -0.88 | | rougeL-R | Decomposition | HyDE | 0.0058 | 1.14 | | rougeL-F1 | Decomposition | HyDE | 0.5761 | 0.18 | | METEOR | Decomposition | HyDE | 0.0001 | 2.12 | | BERTScore Precision | Decomposition | HyDE | 0.1748 | -0.47 | | BERTScore Recall | Decomposition | HyDE | 0.0697 | 0.65 | | BERTScore F1 | Decomposition | HyDE | 0.9567 | 0.02 | | MATTR | Decomposition | HyDE | 0.4283 | 0.26 |

GPT

| metric | techA | techB | pvalue | cohend | |:--------------------|:--------------|:--------------|----------:|----------:| | rouge1-P | Base | Decomposition | 0.1126 | 0.56 | | rouge1-R | Base | Decomposition | 0.3054 | 0.34 | | rouge1-F1 | Base | Decomposition | 0.0544 | 0.7 | | rouge2-P | Base | Decomposition | 0.1953 | 0.44 | | rouge2-R | Base | Decomposition | 0.0298 | 0.81 | | rouge2-F1 | Base | Decomposition | 0.012 | 0.99 | | rougeL-P | Base | Decomposition | 0.1717 | 0.47 | | rougeL-R | Base | Decomposition | 0.0212 | 0.88 | | rougeL-F1 | Base | Decomposition | 0.0114 | 1 | | METEOR | Base | Decomposition | 0.1097 | 0.56 | | BERTScore Precision | Base | Decomposition | 0.0249 | 0.85 | | BERTScore Recall | Base | Decomposition | 0.0112 | 1.01 | | BERTScore F1 | Base | Decomposition | 0.0029 | 1.28 | | MATTR | Base | Decomposition | 0.2569 | -0.38 | | rouge1-P | Base | HyDE | 0.0001 | -2.25 | | rouge1-R | Base | HyDE | 0 | 2.61 | | rouge1-F1 | Base | HyDE | 0.0001 | 2.21 | | rouge2-P | Base | HyDE | 0.0075 | -1.08 | | rouge2-R | Base | HyDE | 0.0001 | 2.25 | | rouge2-F1 | Base | HyDE | 0.0001 | 2.1 | | rougeL-P | Base | HyDE | 0.0001 | -2.03 | | rougeL-R | Base | HyDE | 0 | 2.43 | | rougeL-F1 | Base | HyDE | 0.0003 | 1.83 | | METEOR | Base | HyDE | 0 | 2.57 | | BERTScore Precision | Base | HyDE | 0.8662 | -0.05 | | BERTScore Recall | Base | HyDE | 0 | 2.37 | | BERTScore F1 | Base | HyDE | 0.0001 | 2.2 | | MATTR | Base | HyDE | 0.3248 | -0.33 | | rouge1-P | Decomposition | HyDE | 0.0011 | -1.5 | | rouge1-R | Decomposition | HyDE | 0 | 2.97 | | rouge1-F1 | Decomposition | HyDE | 0.0003 | 1.84 | | rouge2-P | Decomposition | HyDE | 0.0042 | -1.2 | | rouge2-R | Decomposition | HyDE | 0 | 2.41 | | rouge2-F1 | Decomposition | HyDE | 0.0026 | 1.3 | | rougeL-P | Decomposition | HyDE | 0.0001 | -2 | | rougeL-R | Decomposition | HyDE | 0 | 2.86 | | rougeL-F1 | Decomposition | HyDE | 0.0039 | 1.22 | | METEOR | Decomposition | HyDE | 0 | 2.84 | | BERTScore Precision | Decomposition | HyDE | 0.0323 | -0.8 | | BERTScore Recall | Decomposition | HyDE | 0.0001 | 2.13 | | BERTScore F1 | Decomposition | HyDE | 0.0057 | 1.14 | | MATTR | Decomposition | HyDE | 0.7925 | -0.09 |

🌎 About Conect2AI

The research group Conect2AI is composed of undergraduate and graduate students from the Federal University of Rio Grande do Norte (UFRN). Our mission is to apply Artificial Intelligence (AI) and Machine Learning (ML) to emerging areas.

• Embedded Intelligence & IoT: Optimizing resource management and energy efficiency in connected environments.
  
• Energy Transition & Mobility: Leveraging AI to optimize the energy consumption of connected vehicles and foster more efficient, sustainable mobility.