https://github.com/aidotse/deploy-llm
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (6.3%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: aidotse
- Language: Shell
- Default Branch: main
- Size: 25.4 KB
Statistics
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Deploy LLM
This is a repository for a production-ready LLM deployment (in time)
Components
- LLM serving with vLLM
- Extracting and serving statistics with Prometheus
- Exposing GPU metrics with dcgm-exporter
- Displaying statistics with Grafana dashboard
Installation
To install all the requirements you need the following dependencies on your machine
- Cuda
- Nvidia drivers
- Docker
- Nvidia container toolkit
You can install all of these dependencies on a fresh ubuntu machine using ./install-script.
Be careful running this command if you do not have a clean machine.
Reboot after installing for the changes to take effect.
Configuration of secrets
You will want to change some settings before using this configuration in production,
like changing the VLLM secret key and the grafana password. Configuration of secrets is
handled in an .env file in the repository root. This file is ignored by git to ensure
secrets are not accidentally committed. You can initalize a .env from our provided
example by running
sh
cp .env.example .env
Running
Start the deployment with docker compose up -d
Owner
- Name: AI Sweden
- Login: aidotse
- Kind: organization
- Email: info@ai.se
- Location: Sweden
- Website: www.ai.se
- Repositories: 14
- Profile: https://github.com/aidotse
AI Sweden is a national center for applied AI research and innovation, with the aim to strengthen the competitiveness of the Swedish industry
GitHub Events
Total
- Member event: 2
- Public event: 1
- Push event: 15
- Pull request event: 1
- Fork event: 3
- Create event: 1
Last Year
- Member event: 2
- Public event: 1
- Push event: 15
- Pull request event: 1
- Fork event: 3
- Create event: 1
Issues and Pull Requests
Last synced: 10 months ago
All Time
- Total issues: 0
- Total pull requests: 3
- Average time to close issues: N/A
- Average time to close pull requests: about 13 hours
- Total issue authors: 0
- Total pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 3
- Average time to close issues: N/A
- Average time to close pull requests: about 13 hours
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- joenaess (3)