Querying Amazon S3 in place: a read-only, zero-copy access layer for data pipelines and AI agents
· By Rhombus AI Engineering Team · 9 min read
Overview
Your data is already in S3. Your next question should not have to wait for a migration.
Which products are driving growth? Where are costs rising? What changed this week? You should be able to ask the question, find the right data and get to work — not set up another storage destination and wait for a sync job first.
With Rhombus AI, you connect your S3 bucket and describe what you want to achieve. Rhombo discovers the relevant files, queries them where they live and helps turn the analysis into a repeatable pipeline. From CSV and Excel to large Parquet datasets, the work comes to your data, not the other way around.
Your data never leaves your account until you choose a destination.
Behind that experience is a native S3 access layer: read-only connections, a searchable metadata catalogue and a router that matches each workload to DuckDB, Amazon Athena or AWS Glue. This post walks through how those pieces fit together.
- Start with the question, not the migration. Work with the files already in your bucket, without building a second data lake first.
- Keep your source under your control. Read-only access leaves original files untouched, with permissions scoped to your chosen bucket or folder.
- Go from exploration to execution. Use the right engine for a quick question, a large analytical query or a distributed pipeline.
- Let Rhombo do the heavy lifting. Discover sources, analyse data and build pipelines through a catalogue of authorised assets, without passing AWS credentials to the model.
Why copy-first ETL is a problem for data lakes
Your data is already in S3. It has a home, an access policy and teams keeping it up to date. Yet many platforms ask you to move it into their storage before you can ask your first question. That turns analysis into an ingestion project.
Suppose you want last season’s revenue by product category. The sales file contains tens of gigabytes, but the answer may be a dozen rows. A copy-first workflow makes you ingest and maintain another copy of the source before producing that small result. The question is small. The prerequisite should not be an entire data migration.
- You wait for movement, not insight. Initial loads and sync jobs sit between the question and the answer. Adding a new source means another transfer to configure and monitor.
- You pay to maintain the same data twice. The original remains in S3 while a second copy adds storage, transfer and ingestion work — before the analysis itself begins.
- Your answers depend on the last successful sync. A corrected file in S3 does not automatically correct its copy. Teams end up checking whether the data arrived, whether the sync finished and which version the answer used.
- Your data footprint grows with every copy. Each replica brings another set of access rules, retention policies and deletion responsibilities. The analysis tool becomes another place to govern the source data.
Native access removes that prerequisite. Catalogue the files where they live, run the joins and aggregations against the source, and deliver the result to the destination you choose. S3 remains the source of truth; a second data lake is not the price of asking a question.
Architecture at a glance
Rhombus AI maintains the metadata catalogue and orchestrates discovery, queries and pipelines. The data access layer, query engines and intermediate results stay within your AWS account, alongside your source data. You choose when and where the finished results are delivered.
Rhombus AI
DuckDB
Amazon Athena
AWS Glue
Your S3 bucket1. Connecting to an S3 bucket with read-only access
Connecting a bucket starts with its name and region, plus an optional folder prefix and KMS key. The connector generates the bucket policy, or a CloudFormation template to help apply it.
The setup grants the compute roles read-only S3 access, with additional KMS permissions for customer-managed encryption keys:
| Permission | Why |
|---|---|
| s3:GetBucketLocation | Confirm the bucket is in the declared region |
| s3:ListBucket | Discover objects, optionally limited to the configured prefix |
| s3:GetObject | Read objects in the authorised scope |
| kms:Decrypt and kms:DescribeKey, when needed | Inspect the key and decrypt objects encrypted with a customer-managed key |
Our CloudFormation template merges the required statements into your existing bucket policy.
Verification is a real read, not a checkbox. The connector checks the region, looks for a supported file within the configured scope, inspects its metadata and attempts a one-byte read. That small read verifies access, including decryption for encrypted objects, before cataloguing begins.
Customer S32. Cataloguing S3 data without copying the lake
Before Rhombo can answer a question, it needs to find the right files and understand how they fit together. The catalogue connects file discovery with query planning, using metadata rather than a second copy of your datasets.
Rhombus AI builds a searchable metadata index across the bucket or prefix you connect. It brings together file names, formats, sizes, worksheets and schemas across CSV, TSV, Parquet and Excel. The source files stay unchanged in your AWS account; the catalogue holds the information needed to discover and work with them.
- Find sources without navigating the whole bucket. Full-text and fuzzy search turn a request such as “the claims file from March” into a catalogue lookup. Discovery works against the index instead of listing S3 again for every question.
- Give Rhombo context before it writes a query. Column names, data types and source capabilities help the agent identify join keys, choose relevant fields and plan aggregations. It starts with the structure of the data, not guesses based on a file name.
- Connect discovery to execution. Each source has a stable asset reference that people, agents and pipelines can use. The router uses its format, size and capabilities to select the appropriate execution path inside your AWS account.
Refreshing a connection updates this index, not a second copy of the lake. Organisation-scoped authorisation and Postgres row-level security keep each organisation’s catalogue isolated.
3. Routing each workload to DuckDB, Athena or Glue
A preview, a cross-source analysis and a production pipeline are different workloads. Sending all three through the same engine either adds unnecessary startup overhead or asks a lightweight execution path to do distributed work. The router starts with the task, then considers source format, size and query capabilities.
DuckDBDirect S3 reads
Amazon AthenaServerless SQL
AWS Glue PySparkReads S3 in place
DuckDB: keep exploration lightweight
Exploration is a sequence of small decisions: inspect columns, check a sample, test a filter, refine the question. DuckDB reads S3 directly and runs inside the customer-account execution environment, without a separate distributed job to start for each interaction. The router uses this path when a single-node engine fits the work, keeping the feedback loop short.
Athena: push the analysis into SQL
When the question calls for large scans or joins across S3 sources, Rhombo pushes the work into Athena rather than loading the inputs into a local DataFrame. Filters, joins and aggregations execute together in SQL; the analysis receives the resulting answer set, not every source row. Athena runs in your AWS account, so query execution and results stay alongside the data.
Glue: distribute the pipeline, not every question
A repeatable pipeline can involve several stages of transformation, joins and output preparation. When that workload benefits from distributed execution, Rhombus AI compiles it to PySpark and runs it on AWS Glue in your account. Spark spreads processing across workers, while intermediate results remain in your designated storage. That startup and coordination overhead serves a full pipeline run rather than being imposed on every preview or exploratory question.
4. Security model
Rhombo can propose a query. It cannot grant itself access. Authorisation, credentials and write permissions are enforced by the platform and the execution environment, independently of the model’s instructions.
| Boundary | How it works |
|---|---|
| Source access | Organisation permissions determine which sources each operation can access. |
| Credentials | IAM roles perform S3 operations; credentials stay out of the model’s context. |
| Output delivery | Source access is read-only. Writes use separately authorised storage and destinations. |
Frequently asked questions
Where does my data live?
Your source data and processing stay within your AWS account. Results are delivered only to the destinations you choose.
Can I connect a single folder instead of the whole bucket?
Yes. Choose a folder prefix when connecting, and scope the generated permissions to that location. You control which part of the bucket Rhombus AI can access.
Which S3 file formats are supported?
CSV, TSV, Parquet, JSONL and Excel (XLSX). You can discover and work with these sources through the same catalogue.
Do I need to write SQL or choose a query engine?
You can describe the analysis or pipeline you want in plain English. Rhombo plans the work, and the router selects an execution path suited to the sources and workload.
Will Rhombo change or delete my original files?
No. Source access is read-only. Pipeline outputs are written separately to the destination you configure, leaving the original files unchanged.
What happens when a bucket is disconnected?
Rhombus AI stops using that connection to access the source. Disconnecting does not delete or modify the files in your S3 bucket.
Your next answer is already in your S3 bucket
Bring the question you have been putting off. Connect your S3 bucket, tell Rhombo what you want to find out and turn the analysis into a pipeline you can use again. Start with your data, not a migration. Get started with Rhombus AI.
