https://github.com/bentoml/bentosglang
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.7%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: bentoml
- Language: Python
- Default Branch: main
- Size: 48.8 KB
Statistics
- Stars: 3
- Watchers: 5
- Forks: 1
- Open Issues: 1
- Releases: 0
Metadata Files
README.md
Self-host LLMs with SGLang and BentoML
This repository contains a group of BentoML example projects, showing you how to serve and deploy open-source LLMs using SGLang, a fast serving framework for LLMs and VLMs.
💡 You can use these examples as bases for advanced code customization. For simple LLM hosting with OpenAI-compatible endpoints without writing any code, see OpenLLM.
See here for a full list of BentoML example projects.
The following is an example of serving one of the LLMs in this repository: Llama 3.1 8B.
Prerequisites
- You have gained access to Llama 3.1 8B on Hugging Face.
- If you want to test the Service locally, we recommend you use an Nvidia GPU with at least 16G VRAM.
Install dependencies
```bash git clone https://github.com/bentoml/BentoSGLang.git cd BentoSGLang/llama3.1-8b-instruct
Recommend Python 3.11
pip install -r requirements.txt
export HF_TOKEN=
Run the BentoML Service
We have defined a BentoML Service in service.py. Run bentoml serve in your project directory to start the Service.
```bash $ bentoml serve
2024-11-12T02:47:06+0000 [INFO] [cli] Starting production HTTP BentoServer from "service:SGL" listening on http://localhost:3000 (Press CTRL+C to quit) 2024-11-12T02:49:31+0000 [INFO] [entry_service:bentosglang-llama3.1-8b-instruct-service:1] Service bentosglang-llama3.1-8b-instruct-service initialized ```
The server is now active at http://localhost:3000. You can interact with it using the Swagger UI or in other different ways.
CURL
```bash curl -X 'POST' \ 'http://localhost:3000/generate' \ -H 'accept: text/event-stream' \ -H 'Content-Type: application/json' \ -d '{ "prompt": "Explain superconductors in plain English", "system_prompt": "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don'\''t know the answer to a question, please don'\''t share false information.", "max_tokens": 1024, "sampling_params": null }' ```Python client
```python import bentoml with bentoml.SyncHTTPClient("http://localhost:3000") as client: response_generator = client.generate( prompt="Explain superconductors in plain English", max_tokens=1024 ) for response in response_generator: print(response, end='') ```OpenAI-compatible endpoints
```python from openai import OpenAI client = OpenAI(base_url='http://localhost:3000/v1', api_key='na') # Use the following func to get the available models client.models.list() chat_completion = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[ { "role": "user", "content": "Explain superconductors in plain English" } ], stream=True, ) for chunk in chat_completion: # Extract and print the content of the model's reply print(chunk.choices[0].delta.content or "", end="") ```Deploy to BentoCloud
After the Service is ready, you can deploy the application to BentoCloud for better management and scalability. Sign up if you haven't got a BentoCloud account.
Make sure you have logged in to BentoCloud.
bash
bentoml cloud login
Create a BentoCloud secret to store the required environment variable and reference it for deployment.
```bash bentoml secret create huggingface HFTOKEN=$HFTOKEN
bentoml deploy . --secret huggingface ```
Note: For custom deployment in your own infrastructure, use BentoML to generate an OCI-compliant image.
Owner
- Name: BentoML
- Login: bentoml
- Kind: organization
- Location: San Francisco
- Website: https://bentoml.com
- Twitter: bentomlai
- Repositories: 76
- Profile: https://github.com/bentoml
The most flexible way to serve AI models in production
GitHub Events
Total
- Watch event: 2
- Delete event: 1
- Issue comment event: 1
- Push event: 16
- Public event: 1
- Pull request event: 12
- Fork event: 1
- Create event: 3
Last Year
- Watch event: 2
- Delete event: 1
- Issue comment event: 1
- Push event: 16
- Public event: 1
- Pull request event: 12
- Fork event: 1
- Create event: 3
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 2
- Average time to close issues: N/A
- Average time to close pull requests: 11 minutes
- Total issue authors: 0
- Total pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 2
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 2
- Average time to close issues: N/A
- Average time to close pull requests: 11 minutes
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 2
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- Sherlock113 (4)
- larme (3)
- dependabot[bot] (2)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- bentoml >=1.3.10
- flashinfer ==0.1.6
- orjson ==3.10.11
- sglang ==0.3.5
- bentoml ==1.4.12
- sglang ==0.4.6.post2
- bentoml ==1.4.12
- sglang ==0.4.6.post2