[](https://discord.gg/wPdN8RCYew)  [](https://badge.fury.io/py/vision-agent) 

VisionAgent

VisionAgent is a library that helps you utilize agent frameworks to generate code to solve your vision task. Check out our discord for updates and roadmaps! The fastest way to test out VisionAgent is to use our web application which you can find here.

Installation

bash pip install vision-agent

bash export ANTHROPIC_API_KEY="your-api-key" export OPENAI_API_KEY="your-api-key"

NOTE: We found using both Anthropic Claude-3.5 and OpenAI o1 to be provide the best performance for VisionAgent. If you want to use a different LLM provider or only one, see 'Using Other LLM Providers' below.

Documentation

VisionAgent Library Docs

Examples

Counting cans in an image

You can run VisionAgent in a local Jupyter Notebook Counting cans in an image

Generating code

You can use VisionAgent to generate code to count the number of people in an image: ```python from visionagent.agent import VisionAgentCoderV2 from visionagent.agent.types import AgentMessage

agent = VisionAgentCoderV2(verbose=True) codecontext = agent.generatecode( [ AgentMessage( role="user", content="Count the number of people in this image", media=["people.png"] ) ] )

with open("generatedcode.py", "w") as f: f.write(codecontext.code + "\n" + code_context.test) ```

Using the tools directly

VisionAgent produces code that utilizes our tools. You can also use the tools directly. For example if you wanted to detect people in an image and visualize the results: ```python import vision_agent.tools as T import matplotlib.pyplot as plt

image = T.loadimage("people.png") dets = T.countgdobject_detection("person", image)

visualize the countgd bounding boxes on the image

viz = T.overlayboundingboxes(image, dets)

save the visualization to a file

T.saveimage(viz, "peopledetected.png")

display the visualization

plt.imshow(viz) plt.show() ```

You can also use the tools for running on video files: ```python import vision_agent.tools as T

framesandts = T.extractframesand_timestamps("people.mp4")

extract the frames from the framesandts list

frames = [f["frame"] for f in framesandts]

run the countgd tracking on the frames

tracks = T.countgdsam2video_tracking("person", frames)

visualize the countgd tracking results on the frames and save the video

viz = T.overlaysegmentationmasks(frames, tracks) T.savevideo(viz, "peopledetected.mp4") ```

Using Other LLM Providers

You can use other LLM providers by changing config.py in the vision_agent/configs directory. For example to change to Anthropic simply just run: bash cp vision_agent/configs/anthropic_config.py vision_agent/configs/config.py

NOTE: VisionAgent moves fast and we are constantly updating and changing the library. If you have any questions or need help, please reach out to us on our discord channel.