Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of @llm-tools/embedjs

@llm-tools/embedjs - v0.1.29

What's Changed

  • Fixed docs for FAQs and RAG API by @punkish in https://github.com/llm-tools/embedJs/pull/203
  • fix: remove hardcoded trailing question mark from user query by @feelbadinc in https://github.com/llm-tools/embedJs/pull/217
  • fix: remove unnecessary slice for message history by @Nunu27 in https://github.com/llm-tools/embedJs/pull/218
  • Potential fix for code scanning alert no. 2: Workflow does not contain permissions by @adhityan in https://github.com/llm-tools/embedJs/pull/219

New Contributors

  • @punkish made their first contribution in https://github.com/llm-tools/embedJs/pull/203
  • @feelbadinc made their first contribution in https://github.com/llm-tools/embedJs/pull/217
  • @Nunu27 made their first contribution in https://github.com/llm-tools/embedJs/pull/218

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.28...v0.1.29

- TypeScript
Published by adhityan 12 months ago

@llm-tools/embedjs - v0.1.28

What's Changed

  • Added image loader by @adhityan
  • Build script updates by @adhityan

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.27...v0.1.28

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.27

What's Changed

  • Fix for libsql score by @adhityan
  • Added new image loader by @adhityan

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.26...v0.1.27

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.26

What's Changed

  • Bug fix for URL parser incorrectly labelling local paths as web urls by @adhityan
  • Removed singleton for embeddings model by @adhityan

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.25...v0.1.26

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.25

What's Changed

  • Confluence loader updates by @adhityan in https://github.com/llm-tools/embedJs/pull/184

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.24...v0.1.25

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.24

What's Changed

  • Lmdb store fix by @adhityan in https://github.com/llm-tools/embedJs/pull/183

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.23...v0.1.24

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.23

What's Changed

  • Move BaseEmbedding to abstract class by @adhityan in https://github.com/llm-tools/embedJs/pull/181

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.22...v0.1.23

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.22

What's Changed

  • LibSQL support by @adhityan in https://github.com/llm-tools/embedJs/pull/178

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.21...v0.1.22

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.21

What's Changed

  • LanceDb now uses the safer existsOk by @adhityan in https://github.com/llm-tools/embedJs/pull/177

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.20...v0.1.21

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.20

What's Changed

  • fix: debug log lines issue by @adhityan in https://github.com/llm-tools/embedJs/pull/174
  • chore: revert build script by @adhityan in https://github.com/llm-tools/embedJs/pull/176

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.19...v0.1.20

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.19

What's Changed

  • chore(deps-dev): bump @eslint/plugin-kit from 0.2.2 to 0.2.3 in the npmandyarn group by @dependabot in https://github.com/llm-tools/embedJs/pull/167
  • chore(deps): bump cross-spawn from 7.0.3 to 7.0.6 in the npmandyarn group by @dependabot in https://github.com/llm-tools/embedJs/pull/168
  • docs: fix PDF data source link in overview by @jhony112 in https://github.com/llm-tools/embedJs/pull/169
  • Issues stale workflow by @adhityan in https://github.com/llm-tools/embedJs/pull/170
  • chore: quickstart doc fix by @adhityan in https://github.com/llm-tools/embedJs/pull/172

New Contributors

  • @jhony112 made their first contribution in https://github.com/llm-tools/embedJs/pull/169

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.18...v0.1.19

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.18

What's Changed

  • Allow local path loaders on markdown local by @adhityan in https://github.com/llm-tools/embedJs/pull/157
  • Add Node LLama Cpp Support by @BingKui in https://github.com/llm-tools/embedJs/pull/159
  • Doc fix by @adhityan in https://github.com/llm-tools/embedJs/pull/166

New Contributors

  • @BingKui made their first contribution in https://github.com/llm-tools/embedJs/pull/159

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.17...v0.1.18

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.17

What's Changed

  • Added Markdown loader by @adhityan in https://github.com/llm-tools/embedJs/pull/156

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.16...v0.1.17

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.16

What's Changed

  • Added xml loader by @adhityan in https://github.com/llm-tools/embedJs/pull/154

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.15...v0.1.16

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.15

What's Changed

  • Docs fixes by @adhityan in https://github.com/llm-tools/embedJs/pull/152

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.14...v0.1.15

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.14

What's Changed

  • Introduced parameter storeConversationsToDefaultThread by @adhityan in https://github.com/llm-tools/embedJs/pull/149
  • Introduce stores (replaces caches) by @adhityan in https://github.com/llm-tools/embedJs/pull/151

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.13...v0.1.14

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.13

What's Changed

  • Conversations bug fixes by @adhityan in https://github.com/llm-tools/embedJs/pull/148

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.12...v0.1.13

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.12

What's Changed

  • Confluence Loader by @adhityan
  • Renamed setQueryTemplate to setSystemMessage by @adhityan

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.11...v0.1.12

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.11

What's Changed

  • Dynamic loader improvements by @adhityan in https://github.com/llm-tools/embedJs/pull/143

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.10...v0.1.11

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.10

What's Changed

  • Update nx to v20 by @adhityan in https://github.com/llm-tools/embedJs/pull/135
  • Fixed documentation by @adhityan in https://github.com/llm-tools/embedJs/pull/138
  • Package version bumps + new confluence example by @adhityan in https://github.com/llm-tools/embedJs/pull/139

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.9...v0.1.10

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.9

What's Changed

  • Type fix in base loader by @adhityan in https://github.com/llm-tools/embedJs/pull/134

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.8...v0.1.9

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.8

What's Changed

  • Confluence documentation updated by @adhityan in https://github.com/llm-tools/embedJs/pull/131
  • Improved confluence loader by @adhityan in https://github.com/llm-tools/embedJs/pull/133

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.7...v0.1.8

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.7

What's Changed

  • OpenAI refactor by @adhityan in https://github.com/llm-tools/embedJs/pull/130

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.6...v0.1.7

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.6

0.1.6 (2024-10-13)

What's Changed

  • Added Huggingface embedding models by @adhityan in https://github.com/llm-tools/embedJs/pull/129

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.5...v0.1.6

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.5

0.1.5 (2024-10-11)

🩹 Fixes

  • minor changelog double lines removed (7e3509a)

What's Changed

  • chore: create SECURITY.md by @adhityan in https://github.com/llm-tools/embedJs/pull/125
  • docs: fixes link to main npm package in READMEs by @philnash in https://github.com/llm-tools/embedJs/pull/126

Full Changelog: https://github.com/llm-tools/embedJs/compare/v0.1.4...v0.1.5

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.4

0.1.4 (2024-10-09)

🚀 Features

🩹 Fixes

  • fixed debug message string fro createLoaderFromMimeType (abf5901)

  • remove changelog generation from github release (87abd2b)

  • remove changelog generation from github release (4aa3f18)

  • capitalization on contributing.md (0381453)

  • Adhityan K V

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.3

0.1.3 (2024-10-06)

🚀 Features

  • readded local-path and url loaders (303133c)

🩹 Fixes

  • commit pinecone example package lock file (3d1051e)
  • exclude examples from release process (1382185)
  • downgrade esbuild version to match nx requirements (183308f)

❤️ Thank You

  • Adhityan K V @adhityan

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.2

0.1.2 (2024-10-06)

🚀 Features

  • readded local-path and url loaders (303133c)

🩹 Fixes

  • exclude examples from release process (1382185)

  • downgrade esbuild version to match nx requirements (183308f)

❤️ Thank You

  • Adhityan K V @adhityan

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - 0.1.1 (2024-10-04)

Temporarily disabled dynamic, url and local path loaders as they required install of all modules from the monorepo. Also temporarily removed access to Simple_Models enum. They will be reenabled soon.

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - v0.1.0

0.1.0 (2024-10-03)

This component has been extracted and is now published as part of a workspace monorepo managed by NX. There are many reasons that prompted this move, but the most critical issue was to decouple the need to install all dependencies for a single usecase. While we add (and continue to add) more and more loaders, databases, caches and models - the number of shared dependencies grew a lot. Most projects will not use all these combinations and it made no sense to have them all installed for everyone. Further, issues with dependent packages raised vulnerabilities that affected all projects - clearly something we did not intend.

Now what? Starting with version 0.1.0, We have switched to a monorepo based approach. All packages will have the same version number but changelogs and dependencies will be independent. You only need to install the relevant addons (loaders, models, databases, etc) specific to your usecase. Given the shortage of maintainers, we will not be able to support the non-monorepo version of the library beyond critical bugfixes for the next three months, post which the older version will not receive any security fixes. We strongly recommend upgrading to the newer version as soon as you can.

  • Adhityan K V

- TypeScript
Published by adhityan over 1 year ago

@llm-tools/embedjs - Version 0.0.82

A number of important features and bug fixes make it into this release. Here's a rundown of the top new features -

Loader inference

The library can now infer the type of the loader automatically. You can pass a string and it will use the MimeType (detected using magic numbers) and the file extension if available to decide what is the correct loader to invoke. For example -

```TS .addLoader('https://tesla-info.com/sitemap.xml') // will use sitemap loader .addLoader('https://en.wikipedia.org/wiki/Tesla,_Inc.') // will use the web loader .addLoader('s4pVFLUlx8g') // will detect this is a youtube video id and use the video loader .addLoader('https://lamport.azurewebsites.net/pubs/paxos-simple.pdf') // will use the pdf loader

.addLoader('local/paxos-simple.pdf') // will also use the pdf loader .addLoader('local/data.csv') // will also use the CSV loader ```

You can also pass it a local directory name and it will recursively load all files within it using the most appropirate loader. Note: It will skip files it does not have a loader for.

Alternatively, you can now add loaders by passing in an object with the correct parameters without invoking the loader constructor directly. That is -

```TS //Before .addLoader(new WebLoader({ urlOrContent: 'https://www.biography.com/business-leaders/steve-jobs' }))

//Now .addLoader({ type: 'Web', urlOrContent: 'https://www.biography.com/business-leaders/steve-jobs' }) ```

This makes for simpler reading and is very consistent across all loaders.

List of added loaders

The library now maintains the past list of loaders which were added in its cache. So, you can now get the list of all loaded content even between restarts. This is useful if you want to internalize the state of the RAG application within the library itself.

You can get the list of loaders by calling -

TS await ragApplication.getLoaders()

The list of added loaders will include all loaders, even those that were implicitly invoked by another loader. To understand this better, let's look at theLocalPathLoader. This loader uses the file system API to scan files and directories. Once it infers the file type, it internally calls other loaders to add and process PPT, CSV, HTML, etc. files. When this happens, the getLoaders() method will give you the list of all loaders including LocalPathLoader, CsvLoader, WebLoader, etc with metadata around what each loader worked on.

Note: All the data around this is recorded in the cache attached. Therefore this functionality only works when you have a cache set.

CSV Loader

Now you can add CSV files from both local and web URLs using the CSV loader. To add a Csv file (or URL) to your embeddings, use CsvLoader. The library will parse the Csv and add each row to its vector database.

TS .addLoader(new CsvLoader({ filePathOrUrl: '...' }))

Note: You can control how the CsvLoader parses the file in great detail by passing in the optional csvParseOptions constructor parameter.

Github workflow

The library now uses Github actions to verify if the PR compiles and builds in Node versions 18, 20 and 22. This will be automatically run on every PR.

- TypeScript
Published by adhityan about 2 years ago

@llm-tools/embedjs - Version 0.0.77 (Stable)

The library has now reached stable. It now supports several LLMs (including HuggingFace, Ollama, OpenAI's GPT 4o and Anthropic among others), many Embedding Models and even more Vector Databases (most popular databases are now supported).

Several critical features were added in this time - 1. It has the ability to detect already having processed a type of document and avoid duplication in the vector database. 2. You can automatically filter the embeddings based on a relevance cut-off. This is possible even with databases that do not return a simialrity score. 3. It has extensive debug logging including the ablity to view raw LLM response 4. It returns sources used in a LLM query 5. You can use a variety of caches including Redis

- TypeScript
Published by adhityan about 2 years ago

@llm-tools/embedjs - First alpha release

The package is open sourced under Apache v2 license as of this version.

- TypeScript
Published by adhityan almost 3 years ago