treeverse /lakeFS
lakeFS - Data version control for your data lake | Git for data
★ 5,589⑂ 491👁 40⑃ 6,516● 3,990▤ 252
+0
日增长
5,589
stars
491
forks
40
watchers
6,516
prs
3,990
issues
252
releases
README
lakeFS is Data Version Control (Git for Data) lakeFS is a source-available tool that transforms your object storage into a Git-like repository. It enables you to manage your data lake the way you manage your code. With lakeFS you can build repeatable, atomic, and versioned data lake operations - from complex ETL jobs to data science and analytics. lakeFS supports AWS S3, Azure Blob Storage, and Google Cloud Storage as its underlying storage service. It is API compatible with S3 and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, DuckDB, and Presto. For more information, see the documentation. Getting Started You can spin up a standalone sandbox instance of lakeFS: bash pip install lakefs python -m lakefs.quickstart Once you've got lakeFS running, open http://127.0.0.1:8000/ in your web browser. Getting Started with Docker Alternatively, you can run lakeFS a server using Docker: bash docker run --pull always \ --name lakefs \ -p 8000:8000 \ treeverse/lakefs:latest \ run --quickstart Quickstart 👉🏻 For a hands-on walk through of the core functionality in lakeFS head over to the quickstart to jump right in! Make sure to also have a look at the lakeFS samples. These are a rich resource of examples of end-to-end applications that you can build with lakeFS. Why Do I Need lakeFS? ETL Testing with Isolated Dev/Test Environment When working with a data lake, it’s useful to have replicas of your production environment. These replicas allow you to test these ETLs and understand changes to your data without impacting downstream data consumers. Running ETL and transformation jobs directly in production without proper ETL Testing is a guaranteed way to have data issues flow into dashboards, ML models, and other consumers sooner or later. The most common approach to avoid making changes directly in production is to create and maintain multiple data environments and perform ETL testing on them. Dev environment to develop the data pipelines and test environment where pipeline changes are tested before pushing it to production. With lakeFS you can create branches, and get a copy of the full production data, without copying anything. This enables a faster and easier process of ETL testing. Reproducibility Data changes frequently. This makes the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state. This has a negative impact on the work, as it becomes hard to: Debug a data issue. Validate machine learning training accuracy (re-running a model over different data gives different results). Comply with data audits. In comparison, lakeFS exposes a Git-like interface to data that allows keeping track of more than just the current state of data. This makes reproducing its state at any point in time straightforward. Write-Audit-Publish Data pipelines feed processed data from data lakes to downstream con
Top issues
#2512[Discovery] Commit only a subset of the staging area9 comments · 2021-10-03#8401Serve OpenAPI spec off a server endpoint8 comments · 2024-11-28#6584Currently lakectl local doesn't support upload with content type.8 comments · 2023-09-12#5334[Bug]: UI issue with Airflow Hooks with "wait_for_dag: true" config8 comments · 2023-02-24#2092Compaction: Common prefix listing on a branch is linear in # of uncommitted entries8 comments · 2021-06-09#9600[Bug]: Missing permissions don't return to the user upon unauthorized request.7 comments · 2025-10-23#9391Add `apply()` function to `Change` model.7 comments · 2025-08-06#8112Document `pmpu` APIs as stabilised7 comments · 2024-08-29
Discussions
#10169Support for GCS PMPU?1 comments · 2026-02-20#10072How to configure LinkAddressTime expiration with API (e.g. 24h)1 comments · 2026-01-29#9333Local lakefs auto attach existing s3 repos?4 comments · 2025-07-23#9002Integration with Apache Ozone for s3-compatible data store1 comments · 2025-04-29#8467LakeFS - Support for Databricks DLT1 comments · 2025-01-06#8115About the S3 gateway implementation1 comments · 2024-08-30#8069LakeFS1 comments · 2024-08-08#6469`lakefs-spec` - a Python `fsspec` implementation for lakeFS1 comments · 2023-08-23
Latest release
v1.88.0 v1.88.0 2026-10-01Important note: Action hooks can no longer send requests to internal addresses (loopback, private, link-local) by default. Allow the internal hosts your hooks need with actions.network.allowed_hosts. Hook requests no longer use the HTTP_PROXY/HTTPS_PROXY environment variables. Security Fixes - Fixed a vulnerability that allowed users who can write action files to make the lakeFS server send requests to internal addresses (GHSA-223c-qwgx-wwcg) - S3 gateway now rejects SigV4 requests with unsigned x-amz- headers (GHSA-96f7-c79c-crg8)