tarsier

Vision utilities for web interaction agents 👀

https://github.com/reworkd/tarsier

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
○
Academic publication links
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (13.6%) to scientific vocabulary

Keywords

gpt4v llms ocr playwright pypi-package python selenium webscraping

Keywords from Contributors

mesh

Scientific Fields

Materials Science Physical Sciences - 40% confidence

Last synced: 6 months ago · JSON representation

Repository

Vision utilities for web interaction agents 👀

Basic Info

Host: GitHub
Owner: reworkd
License: mit
Language: Jupyter Notebook
Default Branch: main
Homepage: https://reworkd.ai
Size: 2.94 GB

Statistics

Stars: 1,723
Watchers: 13
Forks: 112
Open Issues: 17
Releases: 4

Topics

gpt4v llms ocr playwright pypi-package python selenium webscraping

Created over 2 years ago · Last pushed about 1 year ago

Metadata Files

Readme Contributing Funding License Code of conduct Citation Security Support

README.md

Tarsier Monkey

Vision utilities for web interaction agents

Main site Twitter Discord

Tarsier

If you've tried using an LLM to automate web interactions, you've probably run into questions like:

How should you feed the webpage to an LLM? (e.g. HTML, Accessibility Tree, Screenshot)
How do you map LLM responses back to web elements?
How can you inform a text-only LLM about the page's visual structure?

At Reworkd, we iterated on all these problems across tens of thousands of real web tasks to build a powerful perception system for web agents... Tarsier! In the video below, we use Tarsier to provide webpage perception for a minimalistic GPT-4 LangChain web agent.

https://github.com/reworkd/tarsier/assets/50181239/af12beda-89b5-4add-b888-d780b353304b

How does it work?

Tarsier visually tags interactable elements on a page via brackets + an ID e.g. [23]. In doing this, we provide a mapping between elements and IDs for an LLM to take actions upon (e.g. CLICK [23]). We define interactable elements as buttons, links, or input fields that are visible on the page; Tarsier can also tag all textual elements if you pass tag_text_elements=True.

Furthermore, we've developed an OCR algorithm to convert a page screenshot into a whitespace-structured string (almost like ASCII art) that an LLM even without vision can understand. Since current vision-language models still lack fine-grained representations needed for web interaction tasks, this is critical. On our internal benchmarks, unimodal GPT-4 + Tarsier-Text beats GPT-4V + Tarsier-Screenshot by 10-20%!

Installation

shell pip install tarsier

Usage

Visit our cookbook for agent examples using Tarsier:

We currently support 2 OCR engines: Google Vision and Microsoft Azure. To create service account credentials for Google, follow the instructions on this SO answer https://stackoverflow.com/a/46290808/1780891

The credentials for Microsoft Azure are stored as a simple JSON consisting of an API key and an endpoint

json { "key": "<enter_your_api_key>", "endpoint": "<enter_your_api_endpoint>" }

These values can be found in the keys and endpoint section of the computer vision resource. See the instructions at https://learn.microsoft.com/en-us/answers/questions/854952/dont-find-your-key-and-your-endpoint

Otherwise, basic Tarsier usage might look like the following:

```python import asyncio

from playwright.asyncapi import asyncplaywright from tarsier import Tarsier, GoogleVisionOCRService, MicrosoftAzureOCRService import json

def loadocrcredentials(jsonfilepath): with open(jsonfilepath) as f: credentials = json.load(f) return credentials

async def main(): # To create the service account key, follow the instructions on this SO answer https://stackoverflow.com/a/46290808/1780891

google_cloud_credentials = load_ocr_credentials('./google_service_acc_key.json')
#microsoft_azure_credentials = load_ocr_credentials('./microsoft_azure_credentials.json')

ocr_service = GoogleVisionOCRService(google_cloud_credentials)
#ocr_service = MicrosoftAzureOCRService(microsoft_azure_credentials)

tarsier = Tarsier(ocr_service)

async with async_playwright() as p:
    browser = await p.chromium.launch(headless=False)
    page = await browser.new_page()
    await page.goto("https://news.ycombinator.com")

    page_text, tag_to_xpath = await tarsier.page_to_text(page)

    print(tag_to_xpath)  # Mapping of tags to x_paths
    print(page_text)  # My Text representation of the page

if name == 'main': asyncio.run(main()) ```

Keep in mind that Tarsier tags different types of elements differently to help your LLM identify what actions are performable on each element. Specifically:

[#ID]: text-insertable fields (e.g. textarea, input with textual type)
[@ID]: hyperlinks (<a> tags)
[$ID]: other interactable elements (e.g. button, select)
[ID]: plain text (if you pass tag_text_elements=True)

Local Development

Setup

We have provided a handy setup script to get you up and running with Tarsier development.

shell ./script/setup.sh

If you modify any TypeScript files used by Tarsier, you'll need to execute the following command. This compiles the TypeScript into JavaScript, which can then be utilized in the Python package.

shell npm run build

Testing

We use pytest for testing. To run the tests, simply run:

shell poetry run pytest .

Linting

Prior to submitting a potential PR, please run the following to format your code:

shell ./script/format.sh

Supported OCR Services

[x] Google Cloud Vision
[ ] Amazon Textract (Coming Soon)
[ ] Microsoft Azure Computer Vision (Coming Soon)

Roadmap

[x] Add documentation and examples
[x] Clean up interfaces and add unit tests
[x] Launch
[x] Improve OCR text performance
[ ] Add options to customize tagging styling
[ ] Add support for other browsers drivers as necessary

Citations

bibtex @misc{reworkd2023tarsier, title = {Tarsier}, author = {Rohan Pandey and Adam Watkins and Asim Shrestha and Srijan Subedi}, year = {2023}, howpublished = {GitHub}, url = {https://github.com/reworkd/tarsier} }

Owner

Name: Reworkd
Login: reworkd
Kind: organization
Email: founders@reworkd.ai
Location: San Francisco, CA

Website: https://reworkd.ai
Twitter: reworkdai
Repositories: 8
Profile: https://github.com/reworkd

Pushing the Boundaries of OpenSource Agents

GitHub Events

Total

Issues event: 2
Watch event: 285
Delete event: 10
Issue comment event: 11
Pull request event: 21
Fork event: 27
Create event: 10

Last Year

Issues event: 2
Watch event: 285
Delete event: 10
Issue comment event: 11
Pull request event: 21
Fork event: 27
Create event: 10

Committers

Last synced: 10 months ago

All Time

Total Commits: 236
Total Committers: 12
Avg Commits per committer: 19.667
Development Distribution Score (DDS): 0.661

Past Year

Commits: 75
Committers: 8
Avg Commits per committer: 9.375
Development Distribution Score (DDS): 0.707

Top Committers

Name	Email	Commits
asim-shrestha	a**a@h**m	80
awtkns	3****s	55
dependabot[bot]	4****]	38
khoomeik	3****K	27
Sean McGuire	7****2	18
Rohan Pandey	r**n@R**l	10
ml5ah	6****h	2
Craig Mulligan	g**t@c**m	2
nik	n****k	1
mkrupskis	m**s@g**m	1
Harsh Gupta	w**k@h**n	1
Debanjum	d**m@g**m	1

Committer Domains (Top 20 + Academic)

hargup.in: 1 craigmulligan.com: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 20
Total pull requests: 155
Average time to close issues: about 2 months
Average time to close pull requests: 10 days
Total issue authors: 14
Total pull request authors: 13
Average comments per issue: 2.75
Average comments per pull request: 0.69
Merged pull requests: 90
Bot issues: 0
Bot pull requests: 96

Past Year

Issues: 6
Pull requests: 74
Average time to close issues: about 1 month
Average time to close pull requests: 7 days
Issue authors: 6
Pull request authors: 9
Average comments per issue: 2.0
Average comments per pull request: 0.73
Merged pull requests: 38
Bot issues: 0
Bot pull requests: 40

View more stats

Top Authors

Issue Authors

asim-shrestha (4)
ml5ah (2)
awtkns (2)
craigmulligan (1)
eshoyuan (1)
llermaly (1)
SaimSaudagarVenD (1)
sohamganatra (1)
Ponmo (1)
tvatter (1)
dependabot[bot] (1)
amitlevy (1)
PrathamSoni (1)
onlyoneaman (1)
rgarcia (1)

Pull Request Authors

dependabot[bot] (173)
seanmcguire12 (40)
asim-shrestha (30)
KhoomeiK (6)
awtkns (6)
ml5ah (4)
craigmulligan (3)
BilalG1 (2)
nik (2)
hargup (1)
devanshkaloti (1)
debanjum (1)
Krupskis (1)

Top Labels

Issue Labels

enhancement (3) good first issue (3) help wanted (1) dependencies (1)

Pull Request Labels

dependencies (171)

Packages

Total packages: 2
Total downloads:
- pypi 25,327 last-month

Total dependent packages: 1
(may contain duplicates)
Total dependent repositories: 1
(may contain duplicates)
Total versions: 55
Total maintainers: 1

proxy.golang.org: github.com/reworkd/tarsier

Documentation: https://pkg.go.dev/github.com/reworkd/tarsier#section-documentation
License: mit
Latest release: v0.6.0
published over 1 year ago

Versions: 4
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Dependent packages count: 6.5%

Average: 6.7%

Dependent repos count: 7.0%

Last synced: 6 months ago

pypi.org: tarsier

Vision utilities for web interaction agents

Documentation: https://tarsier.readthedocs.io/
License: mit
Latest release: 0.8.2
published over 1 year ago

Versions: 51
Dependent Packages: 1
Dependent Repositories: 1
Downloads: 25,327 Last month

Rankings

Downloads: 6.8%

Dependent packages count: 10.1%

Average: 12.8%

Dependent repos count: 21.6%

Maintainers (1)

awtkns

Last synced: 6 months ago

Dependencies

.github/workflows/python.yml actions

actions/checkout v2 composite
actions/checkout v3 composite
actions/setup-python v4 composite

poetry.lock pypi

appnope 0.1.3
asttokens 2.4.1
attrs 23.1.0
black 23.11.0
cachetools 5.3.2
certifi 2023.7.22
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
colorama 0.4.6
coverage 7.3.2
decorator 5.1.1
exceptiongroup 1.1.3
executing 2.0.1
google-api-core 2.14.0
google-auth 2.23.4
google-cloud-vision 3.4.5
googleapis-common-protos 1.61.0
greenlet 3.0.0
grpcio 1.59.2
grpcio-status 1.59.2
h11 0.14.0
idna 3.4
iniconfig 2.0.0
ipython 8.17.2
isort 5.12.0
jedi 0.19.1
matplotlib-inline 0.1.6
mypy 1.7.0
mypy-extensions 1.0.0
nest-asyncio 1.5.8
outcome 1.3.0.post0
packaging 23.2
parso 0.8.3
pathspec 0.11.2
pexpect 4.8.0
platformdirs 4.0.0
playwright 1.39.0
pluggy 1.3.0
prompt-toolkit 3.0.40
proto-plus 1.22.3
protobuf 4.25.0
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.5.0
pyasn1-modules 0.3.0
pycparser 2.21
pyee 11.0.1
pygments 2.16.1
pysocks 1.7.1
pytest 7.4.3
pytest-asyncio 0.21.1
pytest-base-url 2.0.0
pytest-cov 4.1.0
pytest-mock 3.12.0
pytest-playwright 0.4.3
python-dotenv 1.0.0
python-slugify 8.0.1
requests 2.31.0
rsa 4.9
selenium 4.15.2
six 1.16.0
sniffio 1.3.0
sortedcontainers 2.4.0
stack-data 0.6.3
text-unidecode 1.3
tokenize-rt 5.2.0
tomli 2.0.1
traitlets 5.13.0
trio 0.23.1
trio-websocket 0.11.1
typing-extensions 4.8.0
urllib3 2.0.7
wcwidth 0.2.9
webdriver-manager 4.0.1
wsproto 1.2.0

pyproject.toml pypi

google-cloud-vision ^3.4.5
playwright ^1.39.0
python ^3.10
selenium ^4.15.2

package-lock.json npm

@esbuild/android-arm 0.19.5 development
@esbuild/android-arm64 0.19.5 development
@esbuild/android-x64 0.19.5 development
@esbuild/darwin-arm64 0.19.5 development
@esbuild/darwin-x64 0.19.5 development
@esbuild/freebsd-arm64 0.19.5 development
@esbuild/freebsd-x64 0.19.5 development
@esbuild/linux-arm 0.19.5 development
@esbuild/linux-arm64 0.19.5 development
@esbuild/linux-ia32 0.19.5 development
@esbuild/linux-loong64 0.19.5 development
@esbuild/linux-mips64el 0.19.5 development
@esbuild/linux-ppc64 0.19.5 development
@esbuild/linux-riscv64 0.19.5 development
@esbuild/linux-s390x 0.19.5 development
@esbuild/linux-x64 0.19.5 development
@esbuild/netbsd-x64 0.19.5 development
@esbuild/openbsd-x64 0.19.5 development
@esbuild/sunos-x64 0.19.5 development
@esbuild/win32-arm64 0.19.5 development
@esbuild/win32-ia32 0.19.5 development
@esbuild/win32-x64 0.19.5 development
esbuild 0.19.5 development
prettier 3.1.0 development
typescript 5.2.2 development

package.json npm

esbuild ^0.19.5 development
prettier ^3.1.0 development
typescript ^5.2.2 development

tarsier-snapshots/poetry.lock pypi

annotated-types 0.6.0
anyio 4.2.0
attrs 23.2.0
bananalyzer 0.8.6
boto3 1.34.34
botocore 1.34.34
cachetools 5.3.2
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 3.3.2
colorama 0.4.6
deepdiff 6.7.1
distro 1.9.0
execnet 2.0.2
google-api-core 2.16.2
google-auth 2.27.0
google-cloud-vision 3.6.0
googleapis-common-protos 1.62.0
greenlet 3.0.3
grpcio 1.60.1
grpcio-status 1.60.1
h11 0.14.0
httpcore 1.0.2
httpx 0.26.0
idna 3.6
iniconfig 2.0.0
jinja2 3.1.3
jmespath 1.0.1
markupsafe 2.1.5
numpy 1.26.3
openai 1.11.1
ordered-set 4.1.0
outcome 1.3.0.post0
packaging 23.2
playwright 1.41.1
pluggy 1.4.0
proto-plus 1.23.0
protobuf 4.25.2
psutil 5.9.8
pyasn1 0.5.1
pyasn1-modules 0.3.0
pycparser 2.21
pydantic 2.6.0
pydantic-core 2.16.1
pyee 11.0.1
pysocks 1.7.1
pytest 7.4.4
pytest-asyncio 0.21.1
pytest-html 4.1.1
pytest-metadata 3.1.0
pytest-xdist 3.5.0
python-dateutil 2.8.2
python-dotenv 1.0.1
requests 2.31.0
rsa 4.9
s3transfer 0.10.0
selenium 4.17.2
six 1.16.0
sniffio 1.3.0
sortedcontainers 2.4.0
tabulate 0.9.0
tarsier 0.5.87
tqdm 4.66.1
trio 0.24.0
trio-websocket 0.11.1
typing-extensions 4.9.0
urllib3 2.0.7
wsproto 1.2.0

tarsier-snapshots/pyproject.toml pypi

bananalyzer ^0.8.6
numpy ^1.26.3
playwright ^1.41.1
python ^3.11
python-dotenv ^1.0.1
tarsier *