Retrieval-augmented generation (RAG) has become the default way to put an organization’s own documents in front of a large language model without retraining it. The model gets the attention, but the layer that decides whether a RAG system stays accurate, affordable and governable over time is the storage underneath it. A pipeline that ingests, chunks, embeds and retrieves millions of documents has specific expectations of where those documents live, and storage chosen for a traditional application often fails to meet them.
This matters because RAG deployments rarely stay small. A pilot over a few thousand PDFs becomes a corpus spanning wikis, ticketing systems, contracts and years of email, each with its own access rules. Storage decisions made at the pilot stage are hard to unwind once the corpus, its embeddings and the retrieval logic all depend on them. This article covers how a retrieval pipeline touches storage, why RAG datasets grow, what that implies for requirements, how the main architectures compare, and where the vector database fits relative to the source corpus.
What does storage actually do in a RAG pipeline?
Ingestion
Source documents (PDFs, office files, HTML pages, transcripts, structured exports) are collected from their systems of origin and landed in a central repository. This is a bulk write workload of files in many sizes, arriving in batches or continuously as connectors sync. Each document must be tagged with its origin, owner, timestamp and access classification.
Chunking
Documents are split into passages of a few hundred tokens so retrieval returns a focused piece of text rather than a whole report. This multiplies the object count: a thousand-page manual becomes several thousand small objects, each carrying a reference to its parent document, its position, and the metadata inherited from it.
Embedding
Each chunk is passed through an embedding model that produces a dense vector. The pipeline reads every chunk in parallel and writes back vectors. Re-embedding happens whenever the model is upgraded, so the pipeline must be able to re-read the entire corpus efficiently, not just new arrivals.
Vector indexing
Vectors are loaded into a vector database, where approximate nearest-neighbor structures make similarity search fast. The index is derived data: it is rebuilt from the vectors, and the vectors are rebuilt from the chunks.
Retrieval at inference
When a user asks a question, the query is embedded, the index returns the top matching chunk identifiers, and the application fetches the chunk text (and often the parent document for a citation) to assemble the prompt. This is the latency-sensitive path: many concurrent users, each triggering small reads that must complete in tens of milliseconds.
Across these stages the storage layer sees bulk writes, full-corpus parallel scans, and a constant stream of small concurrent reads. Few architectures are comfortable with all three.
Why do RAG datasets keep growing?
Teams often size storage for the source documents and are surprised when the footprint is several times larger a year later. Growth comes from several directions at once.
- Source documents accumulate as more repositories are connected and existing ones keep producing content. Very little is deleted, because a RAG system is only as complete as its corpus.
- Chunks and embeddings add a second copy of the text plus a vector per chunk. Suppose an organization holds 2 TB of documents that chunk into 400 million passages; at a common embedding dimension stored as 32-bit floats, the vectors alone would occupy well over a terabyte, and every re-embedding with a new model creates another full set.
- Versions multiply everything, because reproducibility demands that a team can say which version of which document, embedded with which model, produced a given answer.
- Logs and evaluation data grow with usage. Prompts, retrieved chunks, responses and feedback are retained to debug quality and audit outputs, and in a busy deployment can rival the corpus itself.
RAG storage must therefore scale in object count as much as in capacity, without a migration each time the corpus doubles.
What storage requirements does RAG impose?
Rich, queryable metadata
Every chunk needs to know its source, version, classification, owner and embedding model. Storing that as first-class metadata on the object, rather than in a side database that drifts out of sync, also enables filtered retrieval (only documents this user may see, only the current version) before the expensive vector search.
S3 API access from pipeline tooling
Nearly every data engineering, orchestration and machine learning framework used to build RAG pipelines speaks the S3 API natively. A platform that exposes S3 lets loaders, embedding jobs and vector databases run without adapters or staging copies.
Scale in capacity and object count
Billions of small chunk objects alongside large source files on a single namespace, with no ceiling that forces partitioning across volumes.
Concurrency of many small reads
Inference-time retrieval is dominated by small parallel GETs from many clients. Operations per second matter more than raw bandwidth, and tail latency matters more than average latency.
Durability and versioning
The corpus is a long-lived asset that is expensive to rebuild. It must survive disk, node and site failures, and versioning should be native rather than an application convention.
Governance and access control
RAG surfaces information through natural language, which makes it easy to leak a document a user should never have seen. Object-level policies, identity integration, audit logging and immutability at the storage tier provide a control point independent of application code.
Placement under the organization’s control
RAG corpora often contain the most sensitive information an organization holds, and regulatory or contractual obligations frequently require that it stay in a particular jurisdiction or on infrastructure the organization operates. Data sovereignty is a design constraint from day one, not an add-on.
How do the storage architectures compare for RAG?
Block storage and SAN presents raw volumes to a server. It offers low-latency performance for a single host and is the right home for a database engine, but it has no metadata beyond what the file system provides, scales by adding volumes rather than as one namespace, and is awkward to share across many pipeline workers.
Scale-out NAS and file storage provides a shared hierarchical namespace over NFS or SMB and handles large sequential reads well. Its limits appear in object count (directories with millions of small files degrade), in metadata (fixed POSIX attributes rather than arbitrary tags), and in the fact that most RAG tooling expects S3 rather than a mounted path.
On-premises or private object storage stores each document or chunk as an object in a flat namespace with arbitrary key-value metadata, accessed over S3. It scales horizontally in capacity and object count, handles high concurrency of small requests, supports versioning and object lock natively, and keeps data on infrastructure the organization controls. It is not a substitute for the vector database; it is the durable foundation the vector database reads from.
Public cloud object storage offers the same S3 semantics and scale with no hardware to operate. The concerns for RAG are cost and control: retrieval generates a very large volume of small requests, and pulling chunks to on-premises inference or another provider incurs egress fees that grow with adoption. Placement, key custody and jurisdiction are governed by the provider’s terms.
| Architecture | Scalability | Access pattern fit | Metadata | Cost model and egress | Data control |
|---|---|---|---|---|---|
| Block / SAN | Scales per volume; single-host oriented | Strong for one database engine; poor for shared many-client reads | File-system attributes only | Capital cost per array; no egress | Full, but hard to share across pipeline nodes |
| Scale-out NAS / file | Good capacity scaling; strained by billions of small files | Good for large sequential reads; NFS/SMB rather than S3 | Fixed POSIX attributes | Capital or subscription; no egress | Full; on-premises placement |
| Object storage (on-premises / private) | Horizontal in capacity and object count on one namespace | Designed for concurrent small GETs and bulk parallel scans over S3 | Arbitrary key-value tags; native versioning | Predictable per-capacity cost; no egress | Full; sovereign placement, policy and immutability under the organization |
| Public cloud object storage | Effectively unlimited | Same S3 fit; latency depends on distance to inference | Arbitrary tags; native versioning | Per GB plus per request, with egress on reads leaving the region or provider | Governed by provider terms; jurisdiction and key custody constrained |
For a RAG system that will run for years, grow into billions of chunks and serve inference from infrastructure the organization operates, on-premises or private object storage lines up most directly with the requirements above. Public cloud object storage remains reasonable when inference is co-located in the same region and the corpus has no residency constraints.
Where does the vector database fit relative to the source corpus?
A common design mistake is to treat the vector database as the place where the knowledge lives. It is better understood as an index. It holds embeddings, the nearest-neighbor index built over them, and identifiers pointing back at the chunk each vector came from. Some deployments also store chunk text inside it for convenience, but that copy is a cache. The object store holds the source documents, the chunks derived from them, the version history and the metadata tying everything together. The source corpus in the object store is the system of record; the vector index is a derived artifact that can be rebuilt from it at any time.
Keeping that distinction clear pays off in several ways:
- Re-embedding with a new model becomes a batch job that reads the corpus from S3 and writes a fresh index, with no risk to the underlying data.
- Access control is enforced on the objects themselves, so a retrieved chunk identifier resolves to text only if the requesting identity may read it.
- Citations and audit trails point at a durable, versioned object rather than at an index row that may have been rebuilt since.
The vector database answers "which chunks resemble this query" as fast as possible; the object store answers "what exactly did that chunk say, where did it come from, and may this user see it" with durability and governance.
What should teams ask before choosing RAG storage?
- Can the pipeline tooling read and write it directly over S3 without staging copies?
- What are the sustained small-GET operations per second and tail latency under concurrent inference load?
- Is per-object metadata arbitrary and searchable, and is versioning native?
- Can access policy, audit logging and immutability be applied at the storage tier independently of the application?
- Where does the data physically reside, who holds the encryption keys, and can that be proven to an auditor?
How Scality ADI and RING support RAG storage
Scality RING is software-defined, scale-out object and file storage that organizations run at petabyte to exabyte scale on their own infrastructure, exposing the S3 API that RAG tooling expects. Its flat namespace, per-object metadata and native versioning make it a single durable system of record for documents, chunks and history, accessible by many pipeline workers and inference nodes concurrently.
Scality ADI, the company’s Autonomous Data Infrastructure platform, brings that storage foundation together with AI data connectivity, end-to-end cyber resilience and sovereign control under one platform. For RAG this means the corpus can be placed on infrastructure the organization operates, protected with the same immutability and access controls used for backup data, and kept in the jurisdiction the data owner requires, while remaining accessible to embedding jobs and vector databases through standard interfaces.
More on how Scality approaches storage for AI workloads is on the AI core use case page and in the AI storage glossary.
The practical takeaway: decide where the system of record for your RAG corpus will live before the pilot becomes production, and make sure that layer can scale, serve concurrent small reads over S3, and stay under your control.
