How notebooks and pipelines get scoped, short-lived access to geospatial tables on any S3-compatible object storage, without ever holding an S3 credential.
Short version: instead of handing out keys, we sign every S3 request in the catalog at the moment it is dispatched. No AWS STS, no vended credentials, nothing that works on only one hyperscaler. The catalog decides per request and logs it, which makes it the complete audit trail as well.
Nobody should be holding an S3 key
Handing long-lived, full-scope S3 credentials to end users is a bad idea. That might sound obvious, but it is what most setups still do, because the alternatives are awkward. A notebook needs to read a Zarr array, so someone pastes in an access key. Now that key is in a notebook, in a Slack thread, and in a .env file on a laptop. It does not expire, and it is scoped to whatever the bucket allows rather than to what that person is allowed to see.
The fix is not better key hygiene. It is not handing out keys at all. Which means permissions have to live in one place instead. Not S3 bucket policies here, IAM roles there, and a spreadsheet of who holds which key somewhere else. One place where every permission is visible and every access is auditable.
For us that place is the catalog: Lakekeeper, the open source Apache Iceberg REST catalog.
The hard part is data that isn’t a table
Iceberg tables are the easy case, and even there it is still a struggle to make it work across every engine. It gets properly hard the moment the data is not a table: Zarr arrays of gridded weather data, Parquet dumps, raw blobs. That is exactly the data our geospatial and meteorological workloads run on, and it is the data notebooks and DAGs reach for most often.
The standard answer is STS (AWS Security Token Service). The cloud vendor vends short-lived credentials, the client talks to S3 directly. That works on AWS. It does not exist on most other object storage, and it moves the decision back out of the catalog. For the whole lifetime of that credential, the catalog has no say.
So we used remote request signing
Remote request signing means the client builds an S3 request, the catalog signs it, and only then does it go out. Every request, every retry, signed fresh. The client never holds a key. The platform runs wherever you have Kubernetes and S3-compatible storage, and the catalog holds a per-request record of who touched what.
The auth flow, end to end
- Your code asks the catalog for a table or blob, presenting its OAuth token. In a notebook that token comes from SSO and we refresh it in the background. In a pipeline it belongs to the DAG’s own service identity.
- Lakekeeper checks the OpenFGA grants for that identity and returns the metadata and the object location. Generic tables put Zarr, Parquet and raw blobs under the same grants as Iceberg tables.
- Your code asks the catalog to sign one specific request: method, URI, headers.
- Lakekeeper checks the grants again, logs the access, and returns the signed URI and headers. The check happens per request, not once per session. That is what makes the log a complete record rather than a sample.
- Your code sends the bytes straight to object storage. Lakekeeper never touches them.
Steps 3 to 5 repeat for every request: every ranged GET, every PUT, every LIST, every multipart part, every retry. Your code never holds an S3 key, and no vendor ever vends one, which is why the same flow works off hyperscaler.
Two pieces were missing
→ Lakekeeper only signed requests for Iceberg tables. We contributed a PR extending remote signing to generic tables, now merged upstream. We are waiting on a release that includes it. Any non-Iceberg object can now sit under catalog governance.
→ obstore, the Rust-backed object store underneath Zarr in Python, had no way to delegate signing at all. We added a RemoteSignedS3Store: ranged GETs, PUTs, LIST, multipart uploads, every retry signed afresh. Running from a fork for now, upstreaming is on the list.
What this looks like in a notebook
On Databaas everyone gets an isolated notebook environment and is already signed in through SSO. We keep the token fresh in the background, so nobody sees a login prompt, let alone a credential. Which makes the whole thing one line. The same call works from an Airflow DAG, using the pipeline’s identity instead of yours.
from databaas_auth import get_remote_signed_s3_store, open_zarr_group
# a governed S3 folder: read and write files directly
store = get_remote_signed_s3_store("climate", "raw_grib")
store.put("2026-08-28.grib2", data)
# the same mechanism, opened as a Zarr array
group = open_zarr_group("climate", "temperature_grid")Namespace read, table found, signing wired up, permissions enforced and logged by the catalog as that specific user. They get a Zarr array back and start working.
Common questions
Does this need AWS? No. That is the point. Remote signing works against any S3-compatible storage, so the same setup runs on Scaleway, OVHcloud, STACKIT, Leafcloud or your own MinIO cluster.
Is it open source? Lakekeeper is, and the generic tables PR is merged upstream. The obstore store runs from a fork until we upstream it. The open source building blocks are almost always there. Our work is finding the combination that holds up when you are not on a hyperscaler. Every gap we close is one less reason to default to one.
Next stop: getting DuckDB working with Iceberg and remote signing.
Want to try this yourself? Book a meeting with our founders to get access to a demo environment.
