Overview
local-llm is a reproducible, low-configuration environment for running open-weight language models entirely on the user's own machine. It provisions a complete local inference stack with a single command, using llama.cpp as the inference runtime and opencode as the command-line agent interface, which provides tool use, context management, and a terminal UI. Once models are downloaded, no external API or network access is required during normal operation.
The project is intended as a convenient baseline for experimenting with locally hosted models and for users who would rather not assemble the toolchain by hand. It was tested on hardware with 6 GB of VRAM, so the default models are small, quantized variants of roughly 2B–4B parameters. Nothing in the setup is tied to this configuration, and resource use can be adjusted through Makefile variables and model choice.
Installation and startup are driven by make targets, and the default setup requires no manual editing of configuration files. Dependencies are pinned in uv.lock, and installation is a single scripted procedure. llama.cpp runs in router mode, serving every cached model on demand through one endpoint at http://127.0.0.1:8080. The connection between opencode and the local server is defined in opencode.json.
The project is described as a work in progress, created as a learning exercise to better understand large language models and the tooling used to run them locally. It may contain errors, omissions, or outdated information. Corrections, bug reports, and suggestions are welcome through the issue tracker.
Pricing
Free
Detailed plans have not been confirmed in our catalog. Check the official website for current limits and billing terms.
Visit WebsitePrices and limits may change. Confirm the currency, billing period, seat minimum and usage caps on the official website.
Use Cases
- Teams use local-llm to run open-weight language models entirely on their own machine without external API or network access.
- Users use local-llm to provision a complete local inference stack with a single command.
- Developers use local-llm as a convenient baseline for experimenting with locally hosted models.
- Users use local-llm to serve multiple cached models on demand through one endpoint.
- Users use local-llm to install and start the environment through make targets without manually editing configuration files.
- Users use local-llm to check the state of installed binaries, the server process, and available models.
- Users use local-llm to stop the llama.cpp server or clean up the environment, opencode, llama.cpp, and downloaded models.
- Users use local-llm to extend the agent with optional Model Context Protocol servers for terminal operations or web search and fetch.
Features
Run open-weight language models locally
Set up reproducible model environments
Use llama.cpp for inference
Use opencode as agent harness
Manage workflows through make targets
Reviews
No reviews yet. Be the first to share your experience!