Skip to content

Latest commit

 

History

History
712 lines (627 loc) · 91.4 KB

llm-tools.md

File metadata and controls

712 lines (627 loc) · 91.4 KB
Raw

🏠Home

Open LLM Models

  • Want to know which one is "the best"? Have a look at the 🏆 Leaderboards in the Benchmarking section.
  • llm.extractum.io The LLM Explorer, a Large Language Model Directory with filters for trending, downloads and latest showing details like quantizations, model types and sizes
  • can-it-run-llm Check most Huggingface LLMs and quants for hardware requirements like vram, ram and memory requirements

Tools

Native GUIs

openAI

  • chatgptui/desktop
  • chatbox is a Windows, Mac & Linux native ChatGPT Client
  • BingGPT Desktop application of new Bing's AI-powered chat
  • cheetah Speech to text for remote coding interviews, giving you hints from GTP3/4
  • Chat2DB++ general-purpose SQL & multi DBMS client and reporting tool which uses ChatGPT capabilities to write and optimize Queries
  • ChatGPT-Next-Web Web, Windows, Linux, Mac GUI. Supports: Local LLMs, Markdown, LaTex, mermaid, code, history compression, prompt templates
  • ChatGPT Native Application for Windows, Mac, Android, iOS, Linux

Local LLMs

cpp / ggml / gguf:

  • koboldcpp llama.cpp with a fancy UI, persistent stories, editing tools, memory etc. Supporting ggmlv3 and old ggml, CLBlast and llama, RWKV, GPT-NeoX, Pythia models
  • Serge chat interface based on llama.cpp for running Alpaca models. Entirely self-hosted, no API keys needed
  • faraday.dev using llama.cpp under the hood to run most llama based models, made for character based chat and role play

gpt4all:

ollama:

  • ollama Run, create, and share llms on macOS, win/linux with a simple cli interface and portable modelfile package
  • ollama-ui Simple HTML UI for Ollama
  • ollama-ui ChatGPT-Style Responsive Chat Web UI Client (GUI) for Ollama

others:

  • LM Studio closed-source but very easy to use Native Mac, Windows, Linux GUI, supporting ggml, MPT, StarCoder, Falcon, Replit, GPT-Neu-X, gguf
  • pinokio Template based 1 Click installer for ML inference (LLMs, Audio, Text, Video)
  • Lit-llama training, fine tuning and inference of llama
  • Dalai LLaMA-based ChatGPT for single GPUs
  • ChatLLaMA LLaMA-based ChatGPT for single GPUs
  • mlc-llm, run any LLM on any hardware (iPhones, Android, Win, Linux, Mac, WebGPU, Metal. NVidia, AMD)
  • webllm Web LLM running LLMs with WebGPU natively in the browser using local GPU acceleration, without any backend, demo
  • faraday.dev Run open-source LLMs on your Win/Mac. Completely offline. Zero configuration.
  • ChatALL concurrently sends prompts to multiple LLM-based AI bots both local and APIs and displays the results
  • pyllama hacked version of LLaMA based on Meta's implementation, optimized for Single GPUs
  • gmessage visually pleasing chatbot that uses a locally running LLM server and supports multiple themes, chat history search, text to speech, JSON file export, and OpenAI API compatible Python code
  • selfhostedAI one-click deployment of RWKV, ChatGLM, llama.cpp models for substituting the openAI API to a locally hosted API
  • Lit-GPT run SOTA LLMs, supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed
  • text-generation-inference Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power LLMs api-inference widgets
  • gorilla-cli use natural language in the terminal to assist with command writing, gorilla writes the commands based on a user prompt, while the user just approves them
  • minigpt4.cpp to run minigpt4 using 4-bit quantization with using the ggml library in pure C/C++
  • LocalAI Drop-in OpenAI API replacement with local LLMs, Audio To Text (whisper), Image generation (Stable Diffusion), OpenAI functions and Embeddings
  • Windows AI Studio Visual Studio Code extension for Fine-tuning, RAG development and inference of local models
  • jan an open source alternative to ChatGPT that runs 100% offline on Windows, Intel/Apple Silicon Mac, Linux and Mobile
  • open-interpreter lets LLMs run code (Python, Javascript, Shell, and more) locally. You can chat with Open Interpreter through a ChatGPT-like interface in your terminal
  • ClipboardConqueror a novel omnipresent copilot alternative designed to bring your very own LLM AI assistant to any text field
  • Chat With RTX by NVIDIA using Tensore Cores locally to run LLMs fast with a local RAG workflow

Web GUIs

openAI

Local LLMs

  • Text Generation Webui An all purpose UI to run LLMs of all sorts with optimizations (running LLaMA-13b on 6GB VRAM, HN Thread)
  • Text Generation Webui Ph0rk0z fork supporting all GPTQ versions and max context of 8192 instead of 4096 (because some models support longer context now)
  • dockerLLM TheBloke's docker variant of text-generation-webui
  • lollms-webui former GPT4ALL-UI by ParisNeo, user friendly all-in-one interface, with bindings for c_transformers, gptq, gpt-j, llama_cpp, py_llama_cpp, ggml
  • Alpaca-LoRa-Serve
  • chat petals web app + HTTP and Websocket endpoints for BLOOM-176B inference with the Petals client
  • Alpaca-Turbo Web UI to run alpaca model locally on Win/Mac/Linux
  • FreedomGPT Web app that executes the FreedomGPT LLM locally
  • HuggingChat open source chat interface for transformer based LLMs by Huggingface
  • openplayground enables running LLM models on a laptop using a full UI, supporting various APIs and local HuggingFace cached models
  • RWKV-Runner Easy installation and running of RWKV Models, providing a local OpenAI API, GUI and custom CUDA kernel acceleration. Supports 2gb up to 32gb VRAM
  • BrainChulo Chat App with vector based Long-Term Memory supporting one-shot, few-shot and Tool capable agents
  • biniou a self-hosted webui for 30+ generative ai models for text generation, image generation, audio generation, video generation etc.
  • ExUI simple, lightweight web UI for running local inference using ExLlamaV2
  • ava Air-gapped Virtual Assistant / Personal Language Server with support for local models using llama.cpp as a backend, demo
  • llamafile Distribute and run LLMs with a single file on Windows, macOS, Linux

Model agnostic

  • OpenChat web ui that currently supports openAI but will implement local LLM support, RAG with PDF, websites, confluence, office 365
  • lobe-chat docker image based chat bot framework with plugin and agent support, roles, UI etc
  • LibreChat OpenAI, Assistants API, Vision, Mistral, Bing, Anthropic, OpenRouter, Google Gemini, model switching, langchain, DALL-E, Plugins, OpenAI Functions, Multi-User, Presets

Backends

  • ExLlama a more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights. By ReturningTarzan
  • ExLlamaV2 faster ExLlama
  • transformers huggingface transformers
  • bitsandbytes 8 bit inference
  • AutoGPTQ 4bit inference
  • llama.cpp
  • TensorRT-LLM Python API for running LLMs on GPU with support for MHA, MQA, GQA, Tensor Parallelism, INT4/8 Quantization, GPTQ, AWQ, FP8, RoPE to run Baichuan, BLOOM, ChatGLM, Falcon, GPT-J/NeoX, LLaMA/2,MPT, OPT, SantaCoder, StarCoder etc.
  • tensorrtllm_backend Triton TensorRT-LLM Backend
  • RWKV.cpp CPU only port of BlinkDL/RWKV-LM to ggerganov/ggml. Supports FP32, FP16 and quantized INT4.
  • sherpa llama.cpp on android
  • chatglm.cpp C++ implementation of ChatGLM-6B & ChatGLM2-6B
  • MLX Apple's ML Toolkit supporting Transformers in the MLX format for faster inference

Voice Assistants

openAI

  • datafilik/GPT-Voice-Assistant
  • Abdallah-Ragab/VoiceGPT
  • LlmKira/Openaibot
  • BarkingGPT Audio2Audio by using Whisper+chatGPT+Bark
  • gpt_chatbot Windows / elevenlabs TTS + pinecone long term memory
  • gpt-voice-conversation-chatbot using GPT3.5/4 API, elevenlab voices, google tts, session long term memory
  • JARVIS-ChatGPT conversational assistant that uses OpenAI Whisper, OpenAI ChatGPT, and IBM Watson to provide quasi-real-time tips and opinions.
  • ALFRED LangChain Voice Assistant, powered by GPT-3.5-turbo, whisper, Bark, pyttsx3 and more
  • bullerbot uses GPT and ElevenLabs to join your online meetings, listen for your name and answers questions with your voice
  • RealChar Create, Customize and Talk to your AI Character/Companion in Realtime using GPT3.5/4, Claude2, Chroma Vector DB, Whisper Speech2Text, ElevenLabs Text2Speech
  • gdansk-ai full stack AI voice chatbot (speech-to-text, LLM, text-to-speech) with integrations to Auth0, OpenAI, Google Cloud API and Stripe - Web App, API

Local LLMs

Retrieval Augmented Generation (RAG)

openAI

  • sqlchat Use OpenAI GPT3/4 to chat with your database
  • chat-with-github-repo which uses streamlit, gpt3.5-turbo and deep lake to answer questions about a git repo
  • mpoon/gpt-repository-loader uses Git and GPT-4 to convert a repository into a text format for various tasks, such as code review or documentation generation.
  • chat-your-data Create a ChatGPT like experience over your custom docs using LangChain
  • embedchain python based RAG Framework
  • dataherald a natural language-to-SQL engine built for enterprise-level question answering over structured data. It allows you to set up an API from your database that can answer questions in plain English
  • databerry create proprietary data stores that can be accessed by GPT

Local LLMs

  • Llama-lab home of llama_agi and auto_llama using LlamaIndex
  • PrivateGPT a standalone question-answering system using LangChain, GPT4All, LlamaCpp and embeddings models to enable offline querying of documents
  • Spyglass tests an Alpaca integration for a self-hosted personal search app. Select the llama-rama feature branch. Discussion on reddit
  • local_llama chatting with your PDFs offline. gpt_chatwithPDF alternative with the ultimate goal of using llama instead of chatGPT
  • Sidekick Information retrieval for LLMs
  • DB-GPT SQL generation, private domain Q&A, data processing, unified vector storage/indexing, and support for various plugins and LLMs
  • localGPT a privateGPT inspired document question-answering solution using GPU instead of CPU acceleration and InstructorEmbeddings, which perform better according to leaderboards instead of LlamaEmbeddings
  • LocalDocs plugin for GPT4All
  • annoy_ltm extension to add long term memory to chatbots using a nearest neighbor vector DB for memory retrieval
  • ChatDocs PrivateGPT + Web UI + GPU Support + ggml, transformers, webui
  • PAutoBot document question-answering engine developed with LangChain, GPT4All, LlamaCpp, ChromaDB, PrivateGPT, CPU only
  • AIDE CLI based privateGPT fork, improved, refactored, multiline support, model switch support, non question command support
  • khoj Chat offline with your second brain using Llama 2, supporting multiple data sources, web search etc.
  • secondbrain Multi-platform desktop app to download and run LLMs locally in your computer

Model Agnostic

  • Paper QA LLM Chain for answering questions from documents with citations, using OpenAI Embeddings or local llama.cpp, langchain and FAISS Vector DB
  • BriefGPT document summarization and querying using OpenAI' and locally run LLM's using LlamaCpp or GPT4ALL, and embeddings stored as a FAISS index, built using Langchain.
  • anything-llm document ingestion, supports multiple vector DBs, remote and local LLMs and supports chat and query mode
  • factool factuality Detection in Generative AI
  • opencopilot LLM agnostic, open source Microsoft Copilot alternative to easily built copilot functionality with RAG, Knowledgebase, Conversional History, Eval, UX into your product
  • DocsGPT chat with your project documentation using RAG, supports OpenAI and local LLMs, and also provides a RAG-fine-tuned docsgpt-14b model
  • txtai All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows
  • mindsdb database for datascience and AI centered workloads like local LLM / OpenAI models access, text embeddings, forecasting etc.
  • Swiss Army Llama FastAPI service for semantic text search using precomputed embeddings and advanced similarity measures, with built-in support for various file types through textract
  • Quivr Dump all your files and thoughts into your private GenerativeAI Second Brain and chat with it
  • danswer Model agnostic RAG QA with many advanced features like Hybrid search + Reranking, time extraction, user intent identification, User access level management, document update and connectors for many SaaS tools
  • SecureAI-Tools Chat with local documents through various local or commercial models, supporting user authentication
  • OpenCopilot implement RAG principles with your own LLM supporting API calling of multiple endpoints
  • RAGatouille Retrievel with ColBERT and other implementations of SOTA research for your RAG pipeline
  • QAnything two stage retrieval based on retrieve-and-rerank approach with SOTA performance for EN/CN and planned support for structured and unstructured data and DBs
  • opengpts open source GPTs and Assistants with LangChain, LangServe and LangSmith. LLM agnostic, Prompt Engineering, Tool support, Vector DB agnostic, Various Retrieval Algorithms, Chat History support
  • cognee Memory management for RAG and AI Applications and Agents
  • bionic-gpt LLM deployment with authentication, team and RBAC functionality, RAG pipeline, tenants etc.
  • rawdog CLI assistant that responds by generating and auto-executing a Python script. Recursive Augmentation With Deterministic Output Generations (RAWDOG) is a novel alternative to RAG
  • ADeus RAG Chatbot for everything you say, by using an always on audio recorder and a Web App

Browser Extensions

openAI

  • sider chrome side-bar for chatGPT and OpenAI API supporting custom prompts and text highlighting
  • chathub-dev/chathub
  • Glarity open-source chrome extension to write summaries for various websites including custom ones and YouTube videos. Extensible
  • superpower-chatgpt chrome extension / firefox addon to add missing features like Folders, Search, and Community Prompts to ChatGPT
  • Lumos Chrome Extension with OLlama Backend as a RAG LLM co-pilot for browsing the web

Local LLMs

  • chatGPTBox add useful LLM chat-boxes to github and other websites, supporting self-hosted model (RWKV, llama.cpp, ChatGLM)

Agents / Automatic GPT

openAI

  • Auto GPT
  • AgentGPT Deploy autonomous AI agents, using vectorDB memory, web browsing via LangChain, website interaction and more including a GUI
  • microGPT Autonomous GPT-3.5/4 agent, can analyze stocks, create art, order pizza, and perform network security tests
  • Auto GPT Plugins
  • AutoGPT-Next-Web An AgentGPT fork as a Web GUI
  • AutoGPT Web
  • AutoGPT.js
  • LoopGPT a re-implementation of AutoGPT as a proper python package, modular and extensible
  • Camel-AutoGPT Communicaton between Agents like BabyAGI and AutoGPT
  • BabyAGIChatGPT is a fork of BabyAGI to work with OpenAI's GPT, pinecone and google search
  • GPT Assistant An autonomous agent that can access and control a chrome browser via Puppeteer
  • gptchat a client which uses GPT-4, adding long term memory, can write its own plugins and can fulfill tasks
  • Chrome-GPT AutoGPT agent employing Langchain and Selenium to interact with a Chrome browser session, enabling Google search, webpage description, element interaction, and form input
  • autolang Another take on BabyAGI, focused on workflows that complete. Powered by langchain.
  • ai-legion A framework for autonomous agents who can work together to accomplish tasks.
  • generativeAgent_LLM Generative Agents with Guidance, Langchain, and local LLMs, implementation of the "Generative Agents: Interactive Simulacra of Human Behavior" paper, blogpost
  • gpt-engineer generates a customizable codebase based on prompts using GPT4, and is easy to adapt and extend; runs on any hardware that can run Python.
  • gpt-migrate takes your existing code base and migrates to another framework or language
  • MetaGPT multi agent meta programming framework. takes requirements as input and outputs user stories, analysis, data structures, etc. MetaGPT includes product managers, architects, PMs, engineers and uses SOPs to run, paper
  • aider command-line chat tool that allows you to write and edit code with OpenAI's GPT models
  • AutoChain Build lightweight, extensible, and testable LLM Agents
  • chatdev Develop Custom Software using Natural Language, while an LLM-powered Multi-Agent Team develops the software for you, paper
  • AutoAgents Generate different roles for GPTs to form a collaborative entity for complex tasks, paper
  • RestGPT LLM-based autonomous agent controlling real-world applications via RESTful APIs
  • MemGPT intelligently manages different memory tiers in LLMs to provide extended context, supporting vector DBs, SQL, Documents etc
  • XAgent Autonomous LLM Agent for Complex Task Solving
  • HAAS Hierarchical Autonomous Agent Swarm create a self-organizing and ethically governed ecosystem of AI agents, inspired by ACE Framework
  • agency-swarm agent orchestration framework enabling the creation of a collaborative swarm of agents (Agencies), each with distinct roles and capabilities

Local LLMs

  • Auto Vicuna Butler Baby-AGI fork / AutoGPT alternative to run with local LLMs
  • BabyAGI AI-Powered Task Management for OpenAI + Pinecone or Llama.cpp
  • Agent-LLM Webapp to control an agent-based Auto-GPT alternative, supporting GPT4, Kobold, llama.cpp, FastChat, Bard, Oobabooga textgen
  • auto-llama-cpp fork of Auto-GPT with added support for locally running llama models through llama.cpp
  • AgentOoba autonomous AI agent extension for Oobabooga's web ui
  • RecurrentGPT Interactive Generation of (Arbitrarily) Long Text. Uses LSTM, prompt-engineered recurrence, maintains short and long-term memories, and updates these using semantic search and paragraph generation.
  • SuperAGI open-source framework that enables developers to build, manage, and run autonomous agents. Supports tools extensions, concurrent agents, GUI, console, vector DBs, multi modal, telemetry and long term memory
  • GPT-Pilot writes scalable apps from scratch while the developer oversees the implementation
  • DevOpsGPT Multi agent system for AI-driven software development. Combine LLM with DevOps tools to convert natural language requirements into working software
  • ToRA Tool-integrated Reasoning Agents designed to solve challenging mathematical reasoning problems by interacting with tools, e.g., computation libraries and symbolic solvers, paper
  • ACE Autonomous Cognitive Entities Framework to automatically create autonomous agents and sub agents depending on the tasks at hand

Model agnostic

  • SuperAgent Build, deploy, and manage LLM-powered agents
  • aiwaves-cn/agents Open-source Framework for Autonomous Language Agents with LSTM, Tool Usage, Web Navigation, Multi Agent Communication and Human-Agent interaction, paper
  • autogen framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks, paper
  • openagents an Open Platform for Language Agents in the Wild, paper
  • TaskWeaver code-first agent framework for planning and executing data analytics tasks interpreting user requests and coordinating plugins
  • crewAI framework for orchestrating role-playing, autonomous AI agents
  • phidata toolkit for building AI Assistants using function calling enabling RAG and other workflows
  • FRIDAY Framework for Computer Agents with Self-Improvement on OSX and Linux
  • agentkit Starter-kit to build constrained agents with Nextjs, FastAPI and Langchain

Multi Modal

  • huggingGPT / JARVIS Connects LLMs with huggingface specialized models
  • Langchain-huggingGPT reimplementation of HuggingGPT using langchain
  • OpenAGI AGI research platform, solves multi step tasks with RLTF and supports complex model chains
  • ViperGPT implementation for visual inference and reasoning with openAPI
  • TaskMatrix former visual-chatgpt connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting.
  • PandaGPT combines ImageBind and Vicuna to understand and combine multimodal inputs from text, image, audio, depth, thermal, and IMU.
  • AGiXT agents with memory, model agnostic, docker deployment, plugin extendable, chat feature, speech to text and text to speech, REST api and more
  • SelfTalker Talk with your virtual self using voice cloning, LLMs and computer vision models
  • CoDi Any to any generation via composable diffusion
  • AutoMix Mixing Language Models with Self-Verification and Meta-Verification, paper
  • NExT-GPT Any-to-Any Multimodal LLM for arbitary input-output combinations (any-to-any) for text, image, video, audio and beyond, paper, weights
  • SpeechGPT Empowering LLMs with Intrinsic Cross-Modal Conversational Abilities for speech audio input and output
  • OpenFLamingo-v2 MPT and RedPajama fine tuned on the OpenFLamingo data set for training Autoregressive Vision-Language Models, models
  • Obsidian 3B open source multimodal visual LLM
  • ml-ferret Refer and Ground Anything Anywhere at Any Granularity
  • CogVLM SOTA open visual language model and Agent
  • Video-LLaVA Image and Video dense LLM and MoE-LLaVA 3B sparse Mixture of Expert model outperforming the original dense 7B model
  • MobileAgent Autonomous Multi-Modal Mobile Device Agent with Visual Perception that can execute tasks

Code generation

  • FauxPilot open source Copilot alternative using Triton Inference Server
  • Turbopilot open source LLM code completion engine and Copilot alternative
  • Tabby Self hosted Github Copilot alternative with RAG-based code completion which utilizes repo-level context
  • starcoder.cpp
  • GPTQ-for-SantaCoder 4bit quantization for SantaCoder
  • supercharger Write Software + unit tests for you, based on Baize-30B 8bit, using model parallelism
  • Autodoc toolkit that auto-generates codebase documentation using GPT-4 or Alpaca, and can be installed in a git repository in about 5 minutes.
  • smol-ai developer a personal junior developer that scaffolds an entire codebase with a human-centric and coherent whole program synthesis approach using <200 lines of Python and Prompts.
  • locai kobold/oobabooga -compatible api for vscode
  • oasis local LLaMA models in VSCode
  • aider cli tool for writing and modifying code with GPT-3.5 and GPT-4
  • continue open-source copilot alternative for software development as a VS Code plugin, can use gpt-4 API or local codellama and other models
  • chatgpt-vscode vscode extension to use unofficial chatGPT API for a code context based chat side bar within the editor
  • codeshell-vscode vscode extension to use the CodeShell-7b models
  • localpilot vscode copilot alternative using local llama.cpp/ggml models on Mac
  • sweep AI-powered Junior Developer for small features and bug fixes.

Libraries and Wrappers

openAI

  • acheong08/ChatGPT Python reverse engineerded chatGPT API
  • gpt4free Use reverse engineered GPT3.5/4 APIs of other website's APIs
  • GPTCache, serve cached results based on embeddings in a vector DB, before querying the OpenAI API.
  • kitt TTS + GPT4 + STT to create a conference call audio bot
  • Marvin simplifies AI integration in software development with easy creation of AI functions and bots managed through a conversational interface
  • chatgpt.js client-side JavaScript library for ChatGPT
  • ChatGPT-Bridge use chatGPT plus' GPT-4 as a local API
  • Powerpointer connects to openAPI GPT3.5 and creates a powerpoint out of your content
  • EdgeGPT Reverse engineered API of Microsoft's Bing Chat using Edge browser
  • simpleaichat python package for simple and easy interfacing with chat AI APIs
  • Dotnet SDK for openai chatGPT, Whisper, GPT-4 and Dall-E SDK for .NET

Local LLMs

  • node-llama-cpp TS library to locally run many models supported by llama.cpp, enhanced with many convenient features, like forcing a JSON schema on the model output on the generation level
  • FastLLaMA Python wrapper for llama.cpp
  • WebGPT Inference in pure javascript
  • TokenHawk performs hand-written LLaMA inference using WebGPU, utilizing th.cpp, th-llama.cpp, and th-llama-loader.cpp, with minimal dependencies
  • WasmGPT ChatGPT-like chatbot in browser using ggml and emscripten
  • AutoGPTQ easy-to-use model GPTQ quantization package with user-friendly CLI
  • gpt-llama.cpp Replace OpenAi's GPT APIs with llama.cpp's supported models locally
  • llama-node JS client library for llama (or llama based) LLMs built on top of llama-rs and llama.cpp.
  • TALIS serves a LLaMA-65b API, optimized for speed utilizing dual RTX 3090/4090 GPUs on Linux
  • Powerpointer-For-Local-LLMs connects to oobabooga's API and creates a powerpoint out of your content
  • OpenChatKit open-source project that provides a base to create both specialized and general purpose chatbots and extensible retrieval system, using GPT-NeoXT-Chat-Base-20B as a base model
  • webgpu-torch Tensor computation with WebGPU acceleration
  • llama-api-server that uses llama.cpp and emulates an openAI API
  • CTransformers python bindings for transformer models in C/C++ using GGML library, supporting GPT-2/J/NeoX, StableLM, LLaMA, MPT, Dollyv2, StarCoder
  • basaran GUI and API as a drop-in replacement of the OpenAI text completion API. Broad HF eco system support (not only llama)
  • CodeTF one-stop Python transformer-based library for code LLMs and code intelligence, training and inferencing on code summarization, translation, code generation
  • CTranslate2 provides fast Transformer (llama, falcon and more) inference for CPU and GPU, featuring compression, parallel execution, framework support
  • auto-gptq easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ for GPU inference
  • exllama Memory-Efficient Llama Rewrite in Python/C++/CUDA for 4bit quantized GPTQ weights, running on GPU, faster than llama.cpp (2023-06-13), autoGPTQ and GPTQ-for-llama
  • SimpleAI Self-Hosted Alternative to openAI API
  • rustformer llm Rust-based ecosystem for llms like BLOOM, GPT-2/J/NeoX, LLaMA and MPT offering a CLI for easy interaction and powered by ggml
  • Haven Fine-Tune and Deploy LLMs On Your Own Infrastructure
  • llama-cpp-python Python Bindings for llama.cpp with low level C API interface, python API, openai like API and LangChain compatibility
  • candle a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use
  • tabbyAPI OpenAI API emulation using exllamav2 API that's both lightweight and fast

Model agnostic

  • LangChain Framework for LLM Application Development (example, paolorechia/learn-langchain with vicuna and GPTQ 4 bit support)
  • Langstream a lighter alternative to LangChain
  • LangFlow GUI for Langchain using graphs/flows
  • Toolformer implementation Allows LLMs to use Tools
  • megabots to create LLM bots by providing Q&A, document retrieval, vector DBs, FastAPI, Gradio UI, GPTCache, guardrails, whisper, supports OpenAI API (local LLMs planned)
  • gorilla Enables LLMs to use tools by semantically and syntactically correctly invoking APIs. Reduces hallucination, custom trained model weights based on llama-7b
  • agency A fast and minimal actor model framework allows humans, AIs, and other computing systems to communicate with each other through shared environments called "spaces".
  • Vercel AI SDK a library for building edge-ready AI-powered streaming text and chat UIs in React, Svelte and Vue supporting LangChain, OpenAI, Anthropic and HF
  • tinygrad Geohot's implementation for a PyTorch killer with the target to be 2x faster
  • Xorbits Inference (Xinference) versatile library designed to deploy and serve language, speech recognition, and multimodal models
  • data-juicer zero code, low code and off the shelf data processing for LLMs
  • Microsoft semantic-kernel a lightweight SDK enabling integration of AI Large Language Models (LLMs) with conventional programming languages
  • LlamaIndex provides a central interface to connect your LLM's with external data
  • haystack LLM orchestration framework to connect models, vector DBs, file converters to pipelines or agents that can interact with your data to build RAG, Q&A, semantic search or conversational agent chatbots
  • rivet Visual graph/flow/node based IDE for creating AI agents and prompt chaining for your applications
  • promptflow visual graph/flow/node based IDE for creating AI agents
  • litellm Use OpenAI API call format for any LLM backend (Local, Huggingface, Cohere, TogetherAI, Azure, Ollama, Replicate, Sagemaker, Anthropic, etc)
  • Flowise Drag & drop UI with visual graph/flow/nodes to build your customized LLM app
  • ChainForge visual graph/flow/node based prompt engineering UI for analyzing and evaluating LLM responses
  • LangStream Event-Driven Developer Platform for Building and Running LLM AI Apps, also providing a visual graph/flow/node based UI. Powered by Kubernetes and Kafka
  • activepieces Automation with SaaS tools and GPT using a visual graph/flow/node based workflow
  • kernel-memory Index and query any data using LLM and natural language, tracking sources and showing citations, ideal for RAG pipelines

prompt templating / grammar / engineering:

  • Jsonformer Generate Structured JSON from Language Models by handling JSON synthax, and letting LLM just output the values
  • Microsoft guidance templating / grammar for LLMs, Demo project by paolorechia for local text-generation-webui. reddit thread. guidance fork and llama-cpp-python fork how-to on reddit
  • outlines Guidance alternative templating / grammar for LLM generation to follow JSON Schemas, RegEx, Caching supporting multiple models, model APIs, and HF transformers
  • lmql LMQL templating / grammar language for LLMs based on a superset of Python going beyond constrain-based templating
  • TypeChat templating / grammar for LLMs to enforce constraints for text generation
  • GBNF templating / grammar implementation using Bakus-Naur Form (BNF) in llama.cpp to guide output, BNF Playground
  • sglang structured generation language designed for LLMs with multiple chained generation calls, advanced prompting techniques, control flow, multiple modalities, parallelism, and external interaction
  • DSPy a framework for algorithmically optimizing LM prompts and weights
  • AlphaCodium Automatic Code Generation improvements with Prompt Engineering and Flow Engineering

Fine Tuning & Training

Merging & Quantization

  • unsloth 2-5x faster and 60% less memory local QLoRA finetuning supporting Llama, CodeLlama, Mistral, TinyLlama etc. using Triton
  • mergekit Tools for merging pretrained large language models.
  • MergeLM LLMs are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
  • SLERP Spherical Linear Interpolation Model Merging
  • AutoAWQ

Resources

Data sets

Research

Other awesome resources

Product Showcases

Optimization

  • Petals
  • FlexGen High-throughput Generative Inference of LLMs with a Single GPU
  • XLA Accelerated Linear Algebra is a ML compiler for GPU, CPU and accelerators
  • zipslicer
  • AITemplate a Python framework which renders neural network into high performance CUDA/HIP C++ code
  • Flash-attention Fast and memory-efficient exact attention
  • tokenmonster ungreedy tokenizer increases inference speed and context-length by 35% for pre-training on new LLMs
  • LOMO fuses the gradient computation and the parameter update in one step to reduce memory usage enables the full parameter fine-tuning of a 7B model on a single RTX 3090

Benchmarking

Leaderboards

Benchmark Suites

  • Big-bench a collaborative benchmark featuring over 200 tasks for evaluating the capabilities of llms
  • Pythia interpretability analysis for autoregressive transformers during training
  • AlpacaEval automatic evaluation for instruction following LLMs, validated against 20k human annotations, reddit announcement
  • LMFlow Benchmark automatic evaluation framework for open source LLMs
  • lm-evaluation-harness framework for few-shot evaluation of autoregressive language models from EleutherAI
  • sql-eval evaluation of LLM generated SQL queries
  • ragas RAG assessment: an evaluation framework for Retrieval Augmented Generation pipelines
  • ToolQA an evaluation framework for RAG and Tool LLM pipelines
  • LangCheck Simple, Pythonic building blocks to evaluate LLM applications
  • PromethAI-Memory Open-source framework for building and testing RAGs and Cognitive Architectures, designed for accuracy, transparency, and control
  • PromptBench a Pytorch-based Python package for Evaluation of LLMs providing APIs
  • CanItEdit Evaluating the Ability of Large Language Models to Follow Code Editing Instructions, paper
  • deepeval evaluation framework specialized for unit testing LLM applications based on metrics such as hallucination, answer relevancy, RAGAS, etc.
  • mlflow llm-evaluate use-case specific standard metrics and custom metrics, optional ground truth
  • AgentBoard Evaluation Board of Multi-turn LLM Agents
  • LLM-Uncertainty-Bench Benchmarking LLMs via Uncertainty Quantification

AI DevOps

  • Vicuna FastChat
  • SynapseML (previously known as MMLSpark),an open-source library that simplifies the creation of massively scalable machine learning (ML) pipelines
  • Colossal-AI unified deep learning system that provides a collection of parallel components for distributed deep learning models. Provides data parallelism, pipeline parallelism, and tensor parallelism
  • OpenLLM Run, deploy, and monitor open-source LLMs on any platform
  • skypilot Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution
  • ONNX Runtime cross-platform inference and training machine-learning accelerator compatible with PyTorch, TensorFlow/Keras, scikit-learn, LightGBM, XGBoost, etc. and runs with different hardware, drivers, and operating systems
  • vllm high-throughput and memory-efficient inference and serving engine for LLMs, paper
  • openllmetry observability for your LLM application, based on OpenTelemetry
  • DeepSpeed-FastGen High-throughput Text Generation for LLMs at 2x vLLM speeds
  • DeepSparse Sparsity-aware deep learning inference runtime for CPUs
  • dvc ML Experiments Management with Git
  • S-LoRA Serving Thousands of Concurrent LoRA Adapters
  • PowerInfer Fast LLM Serving with a Consumer-grade GPU leveraging activation locality, PR on llama.cpp, issue on ollama
  • TaskingAI open source platform for AI-native application development
  • inferflow LLM inference serving engine with support for Multi-GPU, Quantization supporting gguf, llama2, safetensors and many model families
  • [LMDeploy](https://github.com/InternLM/lmdeploy multi-model, multi-machine, multi-card inference service for many models
  • powerinfer High-speed Model Inference Serving on Consumer GPU/CPU using activation locality for hot/cold neurons
  • lorax Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Databases for ML

  • Pinecone proprietary vector search for semantic search, recommendations and information retrieval
  • FAISS Library for Efficient Similarity Search and Clustering using vectors
  • Weaviate open source vector DB for services like OpenAI, HF etc for text, image, Q&A etc.
  • vespa.ai one of the only scalable vector DBs that supports multiple vectors per schema field
  • LanceDB free open-source serverless vector DB with support for langchain, llamaindex and multi-modal data
  • Deeplake Vector Database for audio, text, vectors, video
  • milvus open-source cloud-native vector DB focusing on embedding vectors converted from unstructured data
  • chroma open-source embedding database
  • pgvector open-source vector similarity search for Postgres.