Alternatives to ggml-org/llama.cpp

LLM inference in C/C++

Source: ggml-org/llama.cpp · SPDX licence: MIT

Comparable repositories

• nousresearch/nous-llama.cpp · LLM inference in C/C++ - nous research · MIT
• bentoml/bentoml · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more! · Apache-2.0
• bytedance/infinistore · KV cache store for distributed LLM inference · Apache-2.0
• microsoft/llmlingua · [EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss. · MIT
• nousresearch/speculators · A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM · Apache-2.0

Comparison from the dated registry snapshot. Review each project and licence independently.