AI data preprocessing should happen where each transformation can run efficiently, within the data’s access and location constraints. A useful starting point is to validate and filter data near its source, run reusable preparation on compute close to shared storage, and keep transformations that change with each training run near the training environment. The right placement depends on how much data a step moves, whether its output can be reused, and how quickly the next stage needs it.
This is rarely a choice between putting everything beside the data or everything beside the GPUs. A document pipeline may extract text on CPU workers, generate embeddings on separate accelerators, and store both outputs for later use. An image pipeline may filter files centrally while leaving random crops and other augmentation to the training process.
The infrastructure question is therefore more specific: which work should happen once, which work must happen again, and what needs to cross the network between those steps?
Within the broader AI data pipeline architecture and stages, preprocessing turns source material into inputs that a model or retrieval system can use. Depending on the workload, it includes validating files, removing duplicates, extracting text, resizing images, normalizing fields, tokenizing text, or assembling records into batches. For retrieval-augmented generation, preparation also commonly includes splitting documents into chunks and generating embeddings for indexing.
These operations place different demands on infrastructure. Parsing a document may depend heavily on CPU time and memory, while an embedding model may benefit from GPU acceleration. Reading millions of small files creates a different storage workload from streaming a few large dataset shards.
Treating all of this as one “preprocessing job” hides those differences. Break the pipeline into stages before deciding where to run it. For each stage, identify the input, output, compute requirement, expected reuse, and permitted processing locations.
Use the following placements as starting points to test against your workload. “Near storage” means compute with an appropriate network path to the data, such as workers in the same data center or cloud region. It does not mean installing arbitrary preprocessing jobs on storage servers.
| Preprocessing work | Useful starting location | Main reason |
|---|---|---|
| File validation and early filtering | Near the source or ingestion point | Avoid moving rejected or irrelevant data downstream |
| Reusable parsing, cleaning and deduplication | Shared compute near the durable dataset | Prepare a controlled output that multiple jobs can reuse |
| Embedding generation or compute-heavy extraction | CPU or GPU workers suited to the operation | Match the transformation to available processing capacity |
| Random augmentation and batch assembly | Near the training workers | Produce inputs that depend on the current run |
| Request-time input preparation | Near the inference service | Limit added latency in the response path |
A single deployment may use every row. The purpose is to place each step deliberately and make its output an explicit handoff to the next stage. That also gives infrastructure teams a clearer way to isolate performance problems.
Early validation prevents predictable failures from consuming expensive resources later. An ingestion worker can reject unreadable files, identify unsupported formats, or quarantine records that fail basic checks. Where filtering removes a large share of the input, running it before a constrained network link can also reduce transfer volume.
Keep enough information to explain those decisions. A rejected document should have an identifier and a recorded reason, so a pipeline owner can distinguish intentional exclusion from missing data. When filtering rules change, the team needs a way to determine which inputs require reprocessing.
Early filtering has limits. A decision that requires comparing records across the entire corpus may belong in a shared processing stage. Source systems may also lack the spare compute needed for heavier preparation.
Stable transformations are good candidates for a shared preprocessing service. Examples include extracting text with a fixed parser version, applying agreed cleaning rules, or creating a curated image dataset. When several downstream jobs need the same result, persisting that output can avoid repeating the work.
Separate CPU workers can also help when preprocessing is starving training workers of inputs. Ray’s documentation describes adding CPU-only nodes for this situation and separating preprocessing tasks when training nodes face CPU or memory pressure. Whether that helps depends on the measured bottleneck. Ray data loading and preprocessing.
The additional workers still need sufficient storage throughput and network capacity. Moving a parser to another machine can free resources on the training host while introducing a new transfer delay. Judge the change by the rate of usable data delivered to training.
Some transformations should remain part of the training process because their output changes across runs or passes through the dataset. Random image augmentation is a common example. Persisting one augmented copy would not provide the same variation as generating fresh transformations during training.
Batch assembly also depends on the model and training configuration. Tokenizer versions, sequence lengths and sampling strategies can change, so a prepared dataset must clearly identify the configuration it supports. A reusable output is only reusable within those boundaries.
A practical split is to retain a validated, consistently formatted dataset and apply run-specific transformations as it is consumed. Test that split with the actual data loader. The right boundary is the one that preserves the intended training behavior while keeping input delivery reliable.
Moving preprocessing upstream does not always reduce network traffic. Filtering can make a dataset smaller, but decoding compressed images or video can produce much larger outputs. Converting compact records into a verbose intermediate format can have a similar effect.
Consider an illustrative pipeline with 10 TB of source documents. Suppose extraction and filtering produce 2 TB of approved text, and the training environment sits across a constrained network connection. Processing near the source would send the 2 TB output across that connection instead of the full 10 TB input.
At a sustained effective transfer rate of 1 GB per second, using decimal units, those transfers take roughly 33 minutes and 2 hours 47 minutes respectively. These are transfer-only estimates, calculated by dividing bytes by throughput. They exclude preprocessing time, retries and contention, so they do not establish the faster end-to-end design by themselves.
Now reverse the relationship: a compressed dataset expands substantially when decoded. Sending decoded output across the same connection may be slower than transferring the compressed source and decoding near the consumer. Measure bytes entering and leaving each stage before assuming that earlier preprocessing improves data locality.
Persisting every intermediate result increases capacity requirements and creates more copies to govern. Recomputing everything increases processing time and can delay recovery after a failed job. The useful middle ground is to retain outputs that are expensive to reproduce, reused frequently, or necessary to explain a result.
Extracted document text may serve several embedding experiments without repeating extraction. An embedding dataset, however, depends on the embedding model and its version; changing that model usually requires generating new embeddings. A temporary decoded batch may be cheaper to recreate than to retain.
For each persisted output, record the source versions, transformation version, relevant parameters and completion status. Write to a new output location and publish a manifest only after the dataset passes validation. This prevents downstream jobs from treating a partially written dataset as complete.
Versioning also needs a retirement process. Assign an owner and retention rule to intermediate datasets, including cached copies. Otherwise, yesterday’s useful optimization becomes an unexplained collection of stale training inputs.
In a RAG pipeline, preparation determines what information becomes searchable. Parsing quality, chunk boundaries and embedding configuration affect the retrieved material. The placement decision must account for how quickly source changes need to reach the retrieval system.
For a frequently updated knowledge base, process changed documents incrementally where possible. Preserve the relationship between each source document, its chunks and its index entries. That relationship allows the pipeline to replace or remove derived material when a document changes or is deleted.
Document permissions also need to survive the handoff. A preprocessing worker may have permission to read the whole repository, while the person querying the application may only be entitled to a subset. Carry the necessary access metadata forward and enforce it in the retrieval path.
This makes freshness an operational measure. Track the delay between a source change and the corresponding index update, including permission changes and deletions. A fast embedding job is only one part of that interval.
Data location constraints apply to the processing path as well as the primary repository. An on-premises dataset may still leave its intended boundary through an external extraction service, a temporary work directory, an embedding API or diagnostic logs. Map those destinations before choosing where a stage runs.
Derived data deserves its own review. Extracted text can retain sensitive information, and embeddings should not be assumed anonymous simply because people cannot read them as ordinary text. Determine which outputs can move and which must remain within the approved environment.
Give each processing stage access to the inputs and destinations it needs. Training workers consuming approved datasets may not need access to the full raw repository. That separation makes access decisions easier to review and limits the reach of a compromised worker.
Scality RING provides a shared object storage foundation with S3 access and a MultiScale architecture that supports independent scaling across dimensions including capacity and performance. In an AI pipeline, that makes it a candidate for the durable repository holding source data and reusable prepared datasets. Scality RING.
An architecture using RING can keep those durable datasets available to separate preprocessing and training environments. CPU workers read source objects and publish curated outputs; downstream jobs consume a defined dataset version. Local storage or a performance tier can hold temporary working data where the workload benefits from it.
The preprocessing software still owns parsing, cleaning, transformation logic and job scheduling. Storage design should support those operations without making a temporary compute cluster the only place a valuable prepared dataset exists. Validate the complete path with representative object sizes, concurrency and access patterns.
Compare two plausible placements using the same representative input and equivalent transformation settings. Include large and small files, difficult formats and malformed inputs, rather than selecting only the easiest records. Run the comparison under realistic concurrency so that shared storage and network contention are visible.
Capture a small set of measures:
Repeat the test with an empty cache and with reusable outputs available. A design that performs well only after everything has been staged may still miss the required startup window. Compare processing cost and retained capacity alongside elapsed time.
Choose the placement that meets the workload’s delivery and freshness requirements with acceptable movement, cost and recovery effort. Validate and reduce data early where that helps, retain preparation that earns its reuse, and keep changing transformations close to their consumer. The result should be a pipeline whose boundaries follow the work each stage performs.