Turn the Parquet and open table formats (delta, iceberg, ducklake, ..) data you already have into fast, typed HTTP APIs — without standing up a warehouse or writing a service layer. Safe - Fast - Easy.
You have the data. Getting to it is the hard part.
Curated datasets land in object storage as Parquet or open table formats. But before anyone can use them, someone has to build and run a service — usually a separate team the owners hand off to.
- Owners hand off to an engineering team to build a Flask / FastAPI wrapper
- Schemas get hand-coded and drift out of sync
- Data gets copied into a database "just to serve it"
- The handoff blurs governance — access & ownership split across teams
- Every consumer waits on that handoff to finish
The gap is a thin, dependable publication layer — and it keeps getting rebuilt by hand.
One config file. A production API.
Point DataPress at your files. It infers the schema and serves a versioned HTTP API in seconds.
pip install datap-rs # or: cargo install datapress
Light enough for one instance per dataset — capable enough to serve many from one. No data movement, no service to maintain, no schema to hand-write.
From bespoke glue to a data product
Today
- A different contract per dataset — or one rigid API forced on them all
- Hand-coded schemas & SQL
- Publishing gated by a central team
- Ad-hoc auth, no metrics
- Weeks of engineering
With DataPress
- One consistent contract, per-dataset limits & policy
- Schema inferred automatically
- Each domain publishes its own data
- Built-in auth, metrics, probes
- Minutes to publish
One layer, four audiences
Data engineers
Publish curated data in Parquet or open table formats straight from pipelines — own your domain's data product without a central team in the loop.
App developers
A stable HTTP contract instead of embedding storage readers and SQL engines in every app.
Analytics engineers
Arrow-native bulk pulls for notebooks and jobs, plus lightweight JSON for inspection.
Business teams
Governed files become reusable, documented, discoverable data products.
A real API, not just a file dump
Discover
List datasets & inspect schemas at runtime.
Query
Filter, sort, group, aggregate, paginate — over HTTP JSON.
Count
Count matching rows without fetching them.
Stream
Arrow IPC for fast, bulk analytics pulls.
Reload
Swap in new data without restarting.
Secure
OIDC / OAuth2 bearer scopes when you need them.
Two engines, one identical API
Same request and response shapes — switch engines with a single config line and compare under your own workload.
| DuckDB | Arrow + DataFusion | |
|---|---|---|
| Best at | Rich SQL, joins, huge/growing data | Low-latency lookups, dense filters |
| Storage | Lazy reads from object storage, resident in RAM as needed | Resident Arrow batches in RAM, lazy reads when needed |
| Data held | Lazy — streamed on demand, scales past RAM - Eager available | Eager — cached in memory, warm and ready - Lazy available |
| Use when | Datasets are large or grow over time | Consumers need ms latency — e.g. market tickers |
| Startup | Milliseconds | Builds an in-memory index, lazy start available |
| Footprint | Bundled engine | Lean, pure-Rust static binary |
Go lazy for big or growing data, eager (in-memory) when every millisecond counts — pick the right default, keep the option open, no rewrite required.
Precompute once. Serve instantly. Refresh automatically.
Declare a dataset as a SELECT over other datasets. DataPress runs the SQL, keeps the result in memory (or on storage), and serves it exactly like a Parquet file — no query-path branching.
Save processing time
- Joins, aggregations & window functions run once, not per request
- Chain query datasets on top of query datasets
- Optional equality index &
sort_byfor fast lookups & pruning
Fresh when upstream changes
on_upstream_reload— rebuild when a dependency publishes- Scheduled
intervalrefresh with jitter & backoff - Cascade in topological order; keep-last-good on failure
[[dataset]]
name = "state_daily_severity"
[dataset.source]
kind = "query"
sql = "SELECT state, avg(severity) FROM accidents GROUP BY state"
depends_on = ["accidents"]
[dataset.refresh]
interval = "15m" # scheduled re-materialization
on_upstream_reload = true # rebuild when "accidents" publishes
Fast by construction, operable by default
Built for speed
Rust + actix-web, columnar formats, mature query engines, projection & predicate pushdown, Arrow IPC to skip JSON overhead.
Runs like a service
Liveness/readiness probes, version metadata, graceful shutdown, Prometheus metrics, hot reloads.
Meets you where you are
Local files, S3-compatible buckets, Parquet partitions, Lakehouse/Open Table Formats — through one predictable API.
Easy to adopt
A pip wheel for Python, a single lightweight binary for ops — run one per dataset or many in one, with embedded docs & Swagger.
Concrete use cases
- Internal data products — publish governed datasets for many teams to reuse
- Powering app features — back search, filters, and tables with a stable JSON API
- Notebook & pipeline feeds — Arrow IPC straight into Polars, DuckDB, pandas
- Replacing one-off services — retire bespoke Flask/FastAPI wrappers
- Lake-side serving — expose S3/lakehouse data without copying into a database
Launch from Python
from datap_rs.datapress import DataPress, DataPressConfig, DatasetConfig
server = DataPress(
DataPressConfig(backend="duckdb", port=8000),
datasets=[DatasetConfig(name="events",
source="s3://lake/events/*.parquet")],
)
# one dataset here — add more to serve them from a single instance
# server.run() → a typed HTTP API over your lake data
Configure in Python, or declare datasets in a single datasets.toml and run the binary — one instance per dataset, or many datasets per instance.
Not a warehouse. Not a BI tool.
DataPress is the thin, fast publication layer between your columnar data and the people and systems that consume it.
It complements your lakehouse and database, and lets each domain publish its own data — it doesn't replace them.
datap-rs
docs.datap-rs.org