bentoml /BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

★ 8,941⑂ 1,065👁 80⑃ 4,312● 1,186▤ 195
PythonApache-2.0Created 2019-04-02bentoml.com
TA的项目
下载:v1.4.40v1.4.40 ↓1.4.402026-10-23
+0
日增长
8,941
stars
1,065
forks
80
watchers
4,312
prs
1,186
issues
195
releases

README

Unified Model Serving Framework

🍱 Build model inference APIs and multi-model serving systems with any open-source or custom AI models. 👉 Join our forum!






 What is BentoML?

BentoML is a Python library for building online serving systems optimized for AI apps and model inference.

- 🍱 Easily build APIs for Any AI/ML Model. Turn any model inference script into a REST API server with just a few lines of code and standard Python type hints.
- 🐳 Docker Containers made simple. No more dependency hell! Manage your environments, dependencies and model versions with a simple config file. BentoML automatically generates Docker images, ensures reproducibility, and simplifies how you deploy to different environments.
- 🧭 Maximize CPU/GPU utilization. Build high performance inference APIs leveraging built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline and multi-model inference-graph orchestration.
- 👩‍💻 Fully customizable. Easily implement your own APIs or task queues, with custom business logic, model inference and multi-model composition. Supports any ML framework, modality, and inference runtime.
- 🚀 Ready for Production. Develop, run and debug locally. Seamlessly deploy to production with Docker containers or Kubernetes.

 Getting started

Install BentoML:


 Requires Python≥3.9
pip install -U bentoml


Define APIs in a service.py file.

python
import bentoml

@bentoml.service(
    image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"),
)
class Summarization:
    def __init__(self) - None:
        import torch
        from transformers import pipeline

        device = "cuda" if torch.cuda.is_available() else "cpu"
        self.pipeline = pipeline('summarization', device=device)

    @bentoml.api(batchable=True)
    def summarize(self, texts: list[str]) - list[str]:
        results = self.pipeline(texts)
        return [item['summary_text'] for item in results]


 💻 Run locally

Install PyTorch and Transformers packages to your Python virtual environment.

bash
pip install torch transformers   additional dependencies for local run


Run the service code locally (serving at http://localhost:3000 by default):

bash
bentoml serve


You should expect to see the following output.


[INFO] [cli] Starting production HTTP BentoServer from "service:Summarization" listening on http://localhost:3000 (Press CTRL+C to quit)
[INFO] [entry_service:Summarization:1] Service Summarization initialized


Now you can run inference from your browser at http://localhost:3000 or with a Python script:

python
import bentoml

with bentoml.SyncHTTPClient('http://localhost:3000') as client:
    summarized_text: str = client.summarize([bentoml.__doc__])[0]
    print(f"Result: {summarized_text}")


 🐳 Deploy using Docker

Run bentoml build to package necessary code, models, dependency configs into a Bento - the standardized deployable artifact in BentoML:

bash
bentoml build


Ensure Docker is r

Top issues

Discussions

Latest release

v1.4.40 v1.4.40 2026-10-23
Security: eMudhra intermediate CA compromise

- \certifi=2026.10.19\ is now required. It drops trust in the compromised eMudhra/emSign CAs.
- Generated Dockerfiles and local \bentoml build\ install \uv==0.13.4\ with \--require-hashes\ whenever uv is missing or older (uv < 0.13.4 bundles the compromised CAs). pip 26.2.2 is used where supported. (5757)

Action for users: run \bentoml build\ again with 1.4.40, then redeploy, so your bento image picks up the fixed certifi and uv.

 Also in this release

- Agent skills for creating BentoML projects and for deploying to Kubernetes and EC2 (5705, 568

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON