Research Question

Is it possible to infer the temperature parameter value used by an LLM from only the generated text?

Probably yes.

Similarity between generated texts with same temperature level (various colors) from the prompt: \ "What will technology look like in 2050?"

Approach

LLM Thermometer uses semantic similarity between multiple responses to estimate temperature:

1. Generation: Produce multiple responses from an LLM using the same prompt
2. Similarity Analysis: Measure semantic similarity between responses
3. Temperature Estimation: Infer temperature based on response diversity
   • Higher temperature → More diverse responses (lower similarity)
   • Lower temperature → More consistent responses (higher similarity)

The reports, hosted on GitHub Pages, contains experiments metadata, charts, and tables.

Usage

```bash

Set required environment variables

export LLMAPIKEY="yourapikey" export LLMBASEURL="https://api.provider.com/v1" export EMBAPIKEY="yourembeddingapikey" export EMBBASE_URL="https://api.provider.com/v1" ```

```bash

Generate samples

llm-thermometer generate \ --language-model "model-name" \ --prompt "What will technology look like in 2050?" \ --samples 32 \ --data-dir ./data \ --temperature 0.7 \

Measure semantic similarity

llm-thermometer measure \ --embedding-model "embedding-model-name" \ --data-dir ./data

Generate report

llm-thermometer report \ --data-dir ./data \ --docs-dir ./docs

Or using Makefile...

make generate make measure make report make docs ```

Installation

The preferred way to install llm-thermometer is using uv (although you can also use pip).

```bash

Clone the repository

git clone https://github.com/S1M0N38/llm-thermometer.git cd llm-thermometer

Create a virtual environment

uv init

Install the package

uv sync ```

Models Local Deployment with Docker

If you have a GPU available, you can run both the Language Model and embedding model locally using docker-compose:

```bash

Set HF_HOME environment variable for model caching

export HF_HOME="/path/to/huggingface/cache"

Start the models

docker-compose up -d

Language model will be available at http://localhost:41408

Embedding model will be available at http://localhost:41409

```

Requirements

• Python 3.12+
• OpenAI-compatible API endpoints (/chat/completions and /embeddings)
• NVIDIA GPU (for local deployment with docker-compose)

Contributing

This research project is still in its early stages, and I welcome any feedback, suggestions, and contributions! If you're interested in discussing ideas or have questions about the approach, please start a conversation in GitHub Discussions.

For detailed information on setting up your development environment, understanding the project structure, and the contribution workflow, please refer to CONTRIBUTING.md.