serge-chat /serge

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

★ 5,692⑂ 385👁 47⑃ 1,617● 180▤ 27
SvelteApache-2.0Created 2023-03-19serge.chat
下载:0.9.0 - Updates to dependencies and UI fixes0.9.0 ↓0.9.02024-02-14
-1
日增长
5,692
stars
385
forks
47
watchers
1,617
prs
180
issues
27
releases

README

Serge - LLaMA made easy 🦙

!License


Serge is a chat interface crafted with llama.cpp for running LLM models. No API keys, entirely self-hosted!

- 🌐 SvelteKit frontend
- 💾 Redis for storing chat history & parameters
- ⚙️ FastAPI + LangChain for the API, wrapping calls to llama.cpp using the python bindings

🎥 Demo:

demo.webm

 ⚡️ Quick start

🐳 Docker:

bash
docker run -d \
    --name serge \
    -v weights:/usr/src/app/weights \
    -v datadb:/data/db/ \
    -p 8008:8008 \
    ghcr.io/serge-chat/serge:latest


🐙 Docker Compose:

yaml
services:
  serge:
    image: ghcr.io/serge-chat/serge:latest
    container_name: serge
    restart: unless-stopped
    ports:
      - 8008:8008
    volumes:
      - weights:/usr/src/app/weights
      - datadb:/data/db/

volumes:
  weights:
  datadb:


Then, just visit http://localhost:8008, You can find the API documentation at http://localhost:8008/api/docs

 🌍 Environment Variables

The following Environment Variables are available:

| Variable Name         | Description                                             | Default Value                        |
|-----------------------|---------------------------------------------------------|--------------------------------------|
| SERGE_DATABASE_URL  | Database connection string                              | sqlite:////data/db/sql_app.db      |
| SERGE_JWT_SECRET    | Key for auth token encryption. Use a random string      | uF7FGN5uzfGdFiPzR                  |
| SERGE_SESSION_EXPIRY| Duration in minutes before a user must reauthenticate   | 60                                 |
| NODE_ENV            | Node.js running environment                             | production                         |

 🖥️ Windows

Ensure you have Docker Desktop installed, WSL2 configured, and enough free RAM to run models.

 ⚠️ Memory Usage

LLaMA will crash if you don't have enough available memory for the model

 💬 Support

Need help? Join our Discord

 🧾 License

Nathan Sarrazin and Contributors. Serge is free and open-source software licensed under the MIT License and Apache-2.0.

 🤝 Contributing

If you discover a bug or have a feature idea, feel free to open an issue or PR.

To run Serge in development mode:

bash
git clone https://github.com/serge-chat/serge.git
cd serge/
docker compose -f docker-compose.dev.yml up --build


The solution will accept a python debugger session on port 5678. Example launch.json for VSCode:

json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote Debug",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}/api",
                    "remoteRoot": "/usr/src/app/api/"
                }
            ],
            "justMyCode": false
        }
    ]
}

Top issues

Latest release

0.9.0 - Updates to dependencies and UI fixes 0.9.0 2024-02-14
🚀 New Features:

- Add support for IPv6 (1055)
- Update llama-cpp-python to v0.2.41 (1133)
- Add support for sidebar menu (1075)

 🧠 Models

- Update OpenChat model version (1046)
- Add support for Phi 2 (1030)
- Add Mixtral 2.7, MedicineLLM, MistralLite, and Notux (1026)

 🧹 Updates:

- Updates to UI components, cleanup of unused dependencies (1136)
- Update llama-cpp-python to v0.2.41 (1133)
- Update NodeJS dependencies (1132)
- Update llama-cpp-python from 0.2.38 to 0.2.39 (1119)
- Update README.md (1113)
- Update llama-cpp-python to v0.2.38 (1062)
- Bump pytest from 7.4

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON