An AI platform usually starts with one team and one dataset, and access control is a formality because everyone involved is allowed to see everything. By the time there are six teams, data from three business units and a dataset that arrived with contractual limits on who may use it, the same arrangement has become a problem that is awkward to fix while people are working.
The separation that matters is not only about permission. It also covers capacity, so one project cannot consume the storage another was relying on; performance, so a heavy job does not degrade everyone else; and audit, so it is possible to establish afterwards who read what. Designing for all four at the start costs very little. Retrofitting them means moving data that is in active use.
Four requirements usually apply, and they are independent enough to be worth stating separately. Confidentiality means a team can only read the datasets it is entitled to. Attribution means consumption and cost can be traced to a project. Containment means one project's activity does not starve another. Accountability means access can be reconstructed later, which becomes a hard requirement as soon as personal data or licensed third party data is involved.
Organizations frequently implement the first and assume the rest follow. They do not. A permission model can be correct while a single team's training job saturates the shared throughput, and a well behaved environment can still be unable to answer which project read a particular dataset last quarter.
It is also worth being explicit about what separation is not for. The goal is not to prevent teams from using each other's data, which is usually valuable, but to make sharing a decision that someone makes rather than a default that nobody reviewed.
In object storage, the structural choice is how buckets and prefixes map onto teams, projects and datasets. Two patterns dominate. A bucket per project gives clean policy boundaries, simple quota attribution and easy deletion when a project ends, at the cost of more buckets to manage and awkward sharing of common datasets. A shared bucket with a prefix per project keeps common data easy to reach and pushes all the separation into policy, which is more flexible and easier to get wrong.
A workable middle position uses buckets to separate by sensitivity class rather than by team: one for data with contractual restrictions, one for data containing personal information, one for general purpose corpora, each with prefixes per project inside. That way the strongest controls are applied at a boundary that reflects the actual obligation, and team membership changes do not require data to move.
Whichever pattern is chosen, decide it before large datasets land. Reorganizing prefixes later means copying petabytes, and the reorganization tends to be deferred indefinitely for exactly that reason.
Access should be granted to roles rather than to individuals, and to prefixes rather than to whole buckets, so that a new team member inherits the right access and a new dataset does not become visible to everyone by default. Training jobs should use their own identities rather than a person's credentials, which is what makes audit records meaningful and what allows a departing employee's access to be removed without breaking running pipelines.
| Boundary | Mechanism | Failure it prevents |
|---|---|---|
| Who can read a dataset | Role-based policy scoped to a prefix | Restricted data used in an unlicensed project |
| Who can write or delete | Separate write role, granted narrowly | A job overwriting another team's source data |
| How much capacity a project uses | Quota per bucket or per prefix | One project consuming the shared pool |
| How much throughput a job takes | Rate limits or separate endpoints | A single run degrading everyone else |
| Which job read which data | Distinct identity per pipeline, access logging | Being unable to answer an audit question |
| Data that must not be combined | Separate buckets, no cross-policy grants | Two datasets merged in breach of their terms |
The last row is the one with legal weight. Datasets frequently arrive with terms that permit use for one purpose and not another, or that forbid combination with other sources. A technical boundary that makes the prohibited combination difficult is more reliable than a policy document that assumes everyone remembers, and it is the kind of control that an audit will ask to see evidence of.
Quotas are the simplest and most effective control, and they are frequently omitted because they feel bureaucratic. Without them, the first project to grow consumes the headroom that everyone's forecast assumed, and the shortfall appears as an urgent purchase rather than as a project that exceeded its allocation.
Throughput separation is harder because most storage systems share bandwidth by design. Where the workloads differ enough, the practical approach is to separate at the access layer: distinct endpoints for the training cluster and for interactive analysis, so that a large batch job and an exploratory notebook are not competing for the same connection pool. Where they must share, rate limits per identity prevent a single misconfigured job from taking everything.
Both controls need visibility to be useful. Per-project consumption and per-project throughput should be reported routinely, because the value of a quota is mostly in the conversation that happens before it is reached.
Scality RING presents one S3 namespace with separate accounts, buckets and policies, so multiple teams can share a single large system without each one needing its own infrastructure. Policies scoped to buckets and prefixes handle the confidentiality boundary, distinct identities per pipeline make access records attributable, and the single namespace means common datasets do not have to be duplicated for each team that needs them.
Running one large system rather than several smaller ones is usually the more efficient arrangement, since capacity headroom is shared and growth is absorbed by adding nodes rather than by provisioning another platform. The trade is that the separation has to be enforced in policy rather than by physical isolation, which makes the namespace and policy design the piece worth spending time on, particularly where operational assumptions are being set for the first time.
The practical measure is how long it takes to give a new project its own space with the right access, a quota and its own audit trail. If that is a documented procedure that takes an hour, the design is working. If it requires a discussion about where the data should go and which existing permissions to extend, the separation exists on paper rather than in the system.
Test the other direction too. Remove a project and see whether its data, identities and quotas can be cleanly retired. Environments that make onboarding easy and offboarding impossible accumulate orphaned data and stale access, and both of those are the findings that show up in a review long after the team that created them has moved on.