RAGoon : High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing

RAGoon is a set of NLP utilities for multi-model embedding production, high-dimensional vector visualization, and aims to improve language model performance by providing contextually relevant information through search-based querying, web scraping and data augmentation techniques.

Quick install

The reference page for RAGoon is available on the official page of PyPI: RAGoon.

python pip install ragoon

Usage

This section provides an overview of different code blocks that can be executed with RAGoon to enhance your NLP and language model projects.

Embeddings production

This class handles loading a dataset from Hugging Face, processing it to add embeddings using specified models, and provides methods to save and upload the processed dataset.

```python from ragoon import EmbeddingsDataLoader from datasets import load_dataset

Initialize the dataset loader with multiple models

loader = EmbeddingsDataLoader( token="hftoken", dataset=loaddataset("louisbrulenaudet/dac6-instruct", split="train"), # If dataset is already loaded. # datasetname="louisbrulenaudet/dac6-instruct", # If you want to load the dataset from the class. modelconfigs=[ {"model": "bert-base-uncased", "queryprefix": "Query:"}, {"model": "distilbert-base-uncased", "queryprefix": "Query:"} # Add more model configurations as needed ] )

Uncomment this line if passing dataset_name instead of dataset.

loader.load_dataset()

Process the splits with all models loaded

loader.process( column="output", preload_models=True )

To access the processed dataset

processeddataset = loader.getdataset() print(processed_dataset[0]) ```

You can also embed a single text using multiple models:

```python from ragoon import EmbeddingsDataLoader

Initialize the dataset loader with multiple models

loader = EmbeddingsDataLoader( token="hftoken", modelconfigs=[ {"model": "bert-base-uncased"}, {"model": "distilbert-base-uncased"} ] )

Load models

loader.load_models()

Embed a single text with all loaded models

text = "This is a single text for embedding." embeddingresult = loader.batchencode(text)

Output the embeddings

print(embedding_result) ```

Similarity search and index creation

The SimilaritySearch class is instantiated with specific parameters to configure the embedding model and search infrastructure. The chosen model, louisbrulenaudet/tsdae-lemone-mbert-base, is likely a multilingual BERT model fine-tuned with TSDAE (Transfomer-based Denoising Auto-Encoder) on a custom dataset. This model choice suggests a focus on multilingual capabilities and improved semantic representations.

The cuda device specification leverages GPU acceleration, crucial for efficient processing of large datasets. The embedding dimension of 768 is typical for BERT-based models, representing a balance between expressiveness and computational efficiency. The ip (inner product) metric is selected for similarity comparisons, which is computationally faster than cosine similarity when vectors are normalized. The i8 dtype indicates 8-bit integer quantization, a technique that significantly reduces memory usage and speeds up similarity search at the cost of a small accuracy rade-off.

```python import polars as pl from ragoon import ( dataset_loader, SimilaritySearch, EmbeddingsVisualizer )

dataset = dataset_loader( name="louisbrulenaudet/dac6-instruct", streaming=False, split="train" )

dataset.savetodisk("dataset.hf")

instance = SimilaritySearch( model_name="louisbrulenaudet/tsdae-lemone-mbert-base", device="cuda", ndim=768, metric="ip", dtype="i8" )

embeddings = instance.encode(corpus=dataset["output"])

ubinaryembeddings = instance.quantizeembeddings( embeddings=embeddings, quantization_type="ubinary" )

int8embeddings = instance.quantizeembeddings( embeddings=embeddings, quantization_type="int8" )

instance.createusearchindex( int8embeddings=int8embeddings, indexpath="./usearchint8.index", save=True )

instance.createfaissindex( ubinaryembeddings=ubinaryembeddings, indexpath="./faissubinary.index", save=True )

topkscores, topkindices = instance.search( query="Dfinir le rle d'un intermdiaire concepteur conformment l'article 1649 AE du Code gnral des Impts.", topk=10, rescoremultiplier=4 )

try: dataframe = pl.fromarrow(dataset.data.table).withrow_index()

except: dataframe = pl.fromarrow(dataset.data.table).withrow_count( name="index" )

scoresdf = pl.DataFrame( { "index": topkindices, "score": topkscores } ).withcolumns( pl.col("index").cast(pl.UInt32) )

searchresults = dataframe.filter( pl.col("index").isin(topkindices) ).join( scores_df, how="inner", on="index" )

print("search_results") ```

Embeddings visualization

This class provides functionality to load embeddings from a FAISS index, reduce their dimensionality using PCA and/or t-SNE, and visualize them in an interactive 3D plot.

```python from ragoon import EmbeddingsVisualizer

visualizer = EmbeddingsVisualizer( indexpath="path/to/index", datasetpath="path/to/dataset" )

visualizer.visualize( method="pca", savehtml=True, htmlfilename="embeddingvisualization.html" ) ```

Dynamic web search

RAGoon is a Python library that aims to improve the performance of language models by providing contextually relevant information through retrieval-based querying, web scraping, and data augmentation techniques. It integrates various APIs, enabling users to retrieve information from the web, enrich it with domain-specific knowledge, and feed it to language models for more informed responses.

RAGoon's core functionality revolves around the concept of few-shot learning, where language models are provided with a small set of high-quality examples to enhance their understanding and generate more accurate outputs. By curating and retrieving relevant data from the web, RAGoon equips language models with the necessary context and knowledge to tackle complex queries and generate insightful responses.

```python from groq import Groq

from openai import OpenAI

from ragoon import WebRAG

Initialize RAGoon instance

ragoon = WebRAG( googleapikey="yourgoogleapikey", googlecx="yourgooglecx", completionclient=Groq(apikey="yourgroqapi_key") )

Search and get results

query = "I want to do a left join in Python Polars" results = ragoon.search( query=query, completionmodel="Llama3-70b-8192", maxtokens=512, temperature=1, )

Print results

print(results) ```

Badge

Building something cool with RAGoon? Consider adding a badge to your project card.

markdown [<img src="https://raw.githubusercontent.com/louisbrulenaudet/ragoon/main/assets/badge.svg" alt="Built with RAGoon" width="200" height="32"/>](https://github.com/louisbrulenaudet/ragoon)

Citing this project

If you use this code in your research, please use the following BibTeX entry.

BibTeX @misc{louisbrulenaudet2024, author = {Louis Brul Naudet}, title = {RAGoon : High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing}, howpublished = {\url{https://github.com/louisbrulenaudet/ragoon}}, year = {2024} }

Feedback

If you have any feedback, please reach out at louisbrulenaudet@icloud.com.