Most on-premises object storage deployments do not fail at installation. They fail a year later, when the first rebuild saturates the network, when an application that "supports S3" depends on one unsupported call, or when the cluster fills with tiny objects the sizing spreadsheet never anticipated. The technology works; the trouble comes from carrying file-storage habits into a system built on different assumptions.
Object storage is now the default landing zone for backup, archives, media, analytics and AI data, and running it on-premises is attractive for cost predictability and sovereignty. But its operational model differs from block and NAS in ways that are easy to underestimate. This field guide covers the mistakes that recur in on-prem projects, each with symptom and fix, plus a summary table and a checklist.
Object storage stores data as immutable objects in flat buckets, addressed by key rather than path, and accessed over an HTTP API (in practice, the S3 API). Objects are written whole, replaced whole, and protected by replication or erasure coding across many nodes. There is no hierarchy, no locking, and no in-place modification.
The naive view is that this is "a big shared drive with a different protocol." It is not. Scale-out object storage is a distributed system, and the team inherits its responsibilities: metadata capacity, east-west traffic, failure domains, rebuild behavior and access policy. Nearly every pitfall below is one of those left to default.
Symptom: raw capacity is plentiful, yet latency climbs, listings crawl, and the metadata tier runs out of flash long before the data disks fill.
Fix: size on object count, not just terabytes. Every object carries metadata, and erasure coding a 4 KB object across a wide stripe creates per-fragment overhead that can exceed the payload. Suppose an organization plans 2 PB for a backup application writing 1 MB chunks: that is roughly two billion objects, and the metadata tier must be planned for that number. Profile real object sizes, ask whether small objects can be packed, and size the metadata flash accordingly.
Symptom: three years in, the original nodes are end of life, and adding denser nodes is blocked by a homogeneity requirement, forcing a forklift migration.
Fix: treat refresh as a day-one design input. Choose a platform that supports heterogeneous node generations and rebalances as capacity is added and retired, so an old generation drains out in a rolling operation.
Symptom: behind an S3-to-file gateway, renaming a large "directory" takes minutes, two clients overwrite each other with no lock error, and appends silently rewrite whole objects.
Fix: know which POSIX semantics object storage does not provide: no byte-range writes, no append, no locking, and rename is a copy plus delete because keys are not paths. Applications that depend on these behaviors need object-native patterns (multipart upload, conditional writes, versioning) or should stay on a file system. If the platform offers both, use the file interface where it is genuinely needed.
Symptom: client throughput collapses the first time a node fails, because rebuild traffic competes with clients on the same links.
Fix: design for internal traffic, not just client ingress. Erasure-coded writes fan out to every node in the stripe, and a rebuild pulls fragments from many nodes at once. Reserve capacity for the back-end network, size uplinks for rebuild plus client load, and confirm the platform can throttle rebuilds.
Symptom: the scheme was picked for efficiency, but a rack power event takes more fragments offline than it tolerates, or a node rebuild takes days and the cluster runs degraded longer than the risk model assumed.
Fix: select erasure coding parameters against three questions: which failure domain must be survivable (drive, node, rack, site), how long the largest rebuild takes at realistic network speed, and how much efficiency is worth trading for shorter exposure. Place fragments across independent domains and test by pulling a node before production.
Symptom: a site-level event (fire, cooling failure, ransomware reaching the management plane) would take every copy offline together.
Fix: decide explicitly how much geographic protection each data class requires, from asynchronous replication to a second site through to stretched erasure coding across three sites that survives the loss of one. Match the choice to recovery objectives and bandwidth; geo-protection is not a substitute for immutability.
Symptom: buckets grow without bound, stale versions and expired backups consume capacity, and nobody can say what is safe to delete.
Fix: define lifecycle policies per bucket before data arrives: expiration for transient data, cleanup of incomplete multipart uploads, noncurrent version expiry, and object lock retention modes where immutability is required. These are far easier to set on an empty bucket than to retrofit on a full one.
Symptom: one root key sits in every application config, never rotated, and a single compromised host can delete every bucket.
Fix: treat S3 credentials as privileged identities. Create per-application identities with least-privilege bucket policies scoped to prefixes and actions, rotate on a schedule, and integrate with a central identity provider. Enable object lock on backup buckets so a stolen key cannot erase recovery data.
Symptom: the platform passes a generic conformance suite, but the backup application fails on object lock, or a multipart edge case appears only under load.
Fix: the S3 API is large and implemented in subsets. Run the actual applications at representative scale in a proof of concept, prefer platforms validated by the application vendors that matter, and confirm the specific features in use: versioning, object lock modes, tagging and notifications.
| Pitfall | Symptom | Fix |
|---|---|---|
| Undersizing for small objects | Metadata tier exhausted while raw capacity is free | Size on object count; flash for metadata |
| Treating it like a NAS | Locking, rename and append misbehave | Object-native patterns, or keep the workload on file |
| Ignoring network design | Throughput collapses during rebuilds | Dedicated back-end network sized for rebuild plus clients |
| Erasure coding chosen by efficiency alone | Long degraded windows; a rack event exceeds tolerance | Choose by failure domain and rebuild time; test it |
| Single-site deployment | All copies share one site-level risk | Replication or stretched erasure coding across sites |
| No lifecycle or bucket policies | Unbounded growth; unclear what can be deleted | Define retention, expiry and object lock before go-live |
| Shared root keys | One compromised host can delete everything | Per-application identities, least privilege, rotation |
| Untested S3 compatibility | Application fails on a specific API behavior in production | Proof of concept with real applications; vendor validation |
Some pitfalls are placement mistakes. Object storage trades per-operation latency and in-place modification for throughput, capacity and durability. It is the wrong primary tier for:
Scality RING is software-defined, scale-out object and file storage used at petabyte to exabyte scale. Its data protection can be placed across independent failure domains, including stretched multi-site configurations that survive the loss of a full site, and it supports mixed hardware generations so refresh is a rolling operation rather than a migration. Native file access alongside S3 gives POSIX-dependent workloads a place to live.
Scality ARTESCA is S3 object storage built for backup, with object lock immutability and validation with major backup applications. That validation speaks directly to the compatibility pitfall, and immutability protects recovery data when a credential is compromised. Both support multi-site deployment within Scality's cyber resilience approach.
The practical takeaway: before the first node is racked, write down the object count, the failure domain, the rebuild time and the bucket policies, and treat any blank on that list as the first problem to solve.