Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/bentoml/bentotwilioconversationrelay

https://github.com/bentoml/bentotwilioconversationrelay

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.5%) to scientific vocabulary
Last synced: 11 months ago · JSON representation

Repository

Basic Info
  • Host: GitHub
  • Owner: bentoml
  • Language: Python
  • Default Branch: main
  • Size: 470 KB
Statistics
  • Stars: 9
  • Watchers: 4
  • Forks: 2
  • Open Issues: 2
  • Releases: 0
Created over 1 year ago · Last pushed about 1 year ago
Metadata Files
Readme

README.md

Building a Voice Chatbot with an Open-Source LLM

This repository demonstrates how to build a voice chatbot using an open-source Large Language Model (LLM). It leverages Twilio for real-time voice interactions and BentoML for deploying and serving the model.

This example uses the Llama-3.1-70B-Instruct model. It is designed to be customizable and you can extend it by adding more features such as RAG.

Read the Blog Post

See here for a full list of BentoML example projects.

Architecture

architecture

Twilio handles:

  • Voice transport (incoming and outgoing)
  • STT: Converts user voice inputs into text and sends them to the BentoML Service.
  • TTS: Converts text responses streamed from the BentoML Service into voice and sends them back to the user.
  • Detects user interruptions during LLM responses and sends interruption signals to BentoML.

BentoML handles:

  • Processes text inputs from Twilio and streams LLM responses back to Twilio in a producer-consumer pattern.
  • Uses a raw vLLM async engine for better latency and the ability to cancel LLM streams (instead of relying on the OpenAI API).
  • Handles user interruptions by canceling the current LLM stream immediately.
  • Maintains consistent message history for seamless interactions.

This project exposes two key endpoints:

  • /chat/start_call: An HTTP endpoint for initiating Twilio voice calls and generating TwiML responses.
  • /chat/ws: A WebSocket endpoint for real-time bidirectional communication, supporting voice streaming.

Prerequisites

Clone the repository and install the required dependencies.

```bash git clone https://github.com/bentoml/BentoTwilioConversationRelay.git cd BentoTwilioConversationRelay

Recommend Python 3.11

pip install -r requirements.txt ```

Run the chatbot locally

Make sure you have installed ffmpeg.

```bash

Ubuntu/Debian

sudo apt-get update && sudo apt-get install ffmpeg ```

Run it locally:

bash bentoml serve .

On the Twilio voice configuration page, set the webhook URL to include the /chat/start_call endpoint (BentoCloud deployment URL is used in the image below).

twilio-number-config

When calling the Twilio number configured, you will hear the following greeting message:

bash "Hi! I'm Jane from Bento M L. Just chat with me!!"

Deploy to BentoCloud

After the chatbot is ready, you can deploy it to BentoCloud for better management and scalability. Sign up if you haven't got a BentoCloud account.

Make sure you have logged in to BentoCloud.

bash bentoml cloud login

Deploy it to BentoCloud.

bash bentoml deploy .

Note: For custom deployment in your own infrastructure, use BentoML to generate an OCI-compliant image.

Owner

  • Name: BentoML
  • Login: bentoml
  • Kind: organization
  • Location: San Francisco

The most flexible way to serve AI models in production

GitHub Events

Total
  • Watch event: 11
  • Delete event: 1
  • Public event: 1
  • Push event: 4
  • Fork event: 2
  • Create event: 1
Last Year
  • Watch event: 11
  • Delete event: 1
  • Public event: 1
  • Push event: 4
  • Fork event: 2
  • Create event: 1

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 7
  • Total Committers: 2
  • Avg Commits per committer: 3.5
  • Development Distribution Score (DDS): 0.429
Past Year
  • Commits: 7
  • Committers: 2
  • Avg Commits per committer: 3.5
  • Development Distribution Score (DDS): 0.429
Top Committers
Name Email Commits
Sherlock Xu 6****3 4
Zhao Shenyang d****v@z****m 3
Committer Domains (Top 20 + Academic)
zsy.im: 1

Issues and Pull Requests

Last synced: about 1 year ago

All Time
  • Total issues: 1
  • Total pull requests: 2
  • Average time to close issues: N/A
  • Average time to close pull requests: about 1 hour
  • Total issue authors: 1
  • Total pull request authors: 2
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.0
  • Merged pull requests: 1
  • Bot issues: 0
  • Bot pull requests: 1
Past Year
  • Issues: 1
  • Pull requests: 2
  • Average time to close issues: N/A
  • Average time to close pull requests: about 1 hour
  • Issue authors: 1
  • Pull request authors: 2
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.0
  • Merged pull requests: 1
  • Bot issues: 0
  • Bot pull requests: 1
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • Sherlock113 (1)
  • dependabot[bot] (1)
Top Labels
Issue Labels
Pull Request Labels
dependencies (1)

Dependencies

requirements.txt pypi
  • accelerate ==1.1.1
  • autoawq ==0.2.7.post2
  • bentoml >=1.3.12
  • fastapi ==0.115.5
  • openai ==1.55.0
  • vllm ==0.6.3.post1