Multiple backup applications are the norm in large organizations. Virtual machines are protected by one product, databases by their native tools, Kubernetes by another, Microsoft 365 by a fourth and a legacy estate by whatever was bought fifteen years ago. Each traditionally brought its own storage target. Most of them can now write to S3, which raises the possibility of pointing all of them at one object store. Consolidation is attractive, and it works when the store is set up so that the applications share infrastructure without sharing anything else.
This post covers how several backup applications coexist on one object store: what has to be separated, how their workloads differ and what to watch as they are added.
Why estates end up with several backup products
Nobody plans to run five backup products. They accumulate through acquisitions, through platform teams choosing tools suited to their technology and through older systems that the main product does not support. Standardizing on one product is a multi-year project with its own risks. Standardizing on one storage target is quicker and delivers much of the operational benefit: one system to secure, one capacity plan, one place to enforce immutability.
What must stay separate
The first principle is that each application should be unable to see or affect another's data. On a multi-tenant object store the clean way to achieve that is an account per backup application, each with its own users, access keys and buckets. Credentials for one product then give no access to another's backups, and a compromise of one backup server is contained to that product's data.
| Element | Shared | Separate per application |
|---|---|---|
| Hardware, capacity pool, network | Yes | |
| Accounts and administrators | Yes | |
| Access keys | Yes, and per backup server where possible | |
| Buckets | Yes, never shared between products | |
| Object Lock mode and retention | Yes, set to each product's requirements | |
| Lifecycle rules | Usually none, the backup product manages expiry | |
| Monitoring and alerting | Yes, with per-account views |
The lifecycle row deserves a note. Backup applications track their own objects and expect to delete them on their own schedule. A lifecycle rule or versioning setting applied to a backup bucket without the product's guidance can remove objects the catalog still references or keep ones it believes are gone. Each product documents the bucket configuration it expects, and that documentation should be followed exactly. The practices in the post on backup credential security apply to every account.
Different products, different workloads
Backup applications use S3 in different ways, and a shared store sees the sum of them. Some write large objects of tens or hundreds of megabytes with few requests. Others write many objects of around a megabyte and issue frequent metadata operations. Database tools may stream very large multipart uploads. Kubernetes and SaaS backup tools tend to produce very large numbers of small objects.
The implication is that capacity alone does not describe the load. Object count and request rate matter for the metadata tier, and they are driven by the products with small objects even when those products hold a minority of the bytes. Sizing should total each application's capacity, object count and peak request rate separately, as set out in the post on object storage sizing, and the effect of small objects is covered in small object storage.
Immutability per application
Every backup product that supports S3 Object Lock applies it in its own way. Some set retention on each object and extend it as chains grow. Some require compliance mode, others accept governance. Some need versioning enabled without a default retention on the bucket. Because the settings are per bucket and per object, they coexist without conflict as long as each product has its own buckets.
The shared consequence is capacity. Locked data cannot be removed early, so the store must hold every product's locked data for its full period. One application with a long lock and a high change rate can consume headroom that others were relying on. The post on immutable backup capacity explains how to estimate the overhead, and it should be estimated per application and then added.
Overlapping windows
Left to their defaults, backup products all start at the same time in the evening. On dedicated targets that did not matter. On a shared store, the combined ingest peak can exceed what the network or the store handles comfortably, even though the total daily volume is well within capacity. Staggering start times is usually enough. Where it is not, concurrency limits in each product spread the load.
Restore is the harder case. A wide incident triggers restores from several products at once, each reading as fast as it can. The store's aggregate read throughput under that combined load is what sets recovery time, and it should be tested with more than one product restoring simultaneously. The post on mass restore planning covers how to sequence a large recovery.
Onboarding an application
A repeatable process keeps a shared store orderly as products are added. Each new application gets an account, a named owner, a capacity and object-count estimate, its documented bucket settings and a test that covers backup, restore and a refused delete of a locked object. Vendor compatibility statements are useful, and a short test on the actual store is more useful still, since products differ in which S3 operations they depend on.
Growth should then be reported per account. When one application's consumption departs from its estimate, the conversation is with that product's owner, with figures to hand.
Where Scality RING fits
Scality RING exposes an S3 interface and is multi-tenant, with accounts, buckets and policy-based isolation, so each backup application can be given its own account and credentials on shared infrastructure. RING supports versioning and S3 Object Lock, set per bucket, which allows each product's immutability requirements to be met independently.
RING distributes data across the nodes in the system, so ingest and restore load from several applications is spread across many devices, and nodes can be added to a live system as applications are onboarded. It emits operational metrics and logs that the operator keeps locally, so capacity, object counts and request rates can be followed per account.
Checks for a shared backup store
- Does each backup application have its own account, keys and buckets?
- Are bucket settings exactly as each product's documentation specifies?
- Has locked capacity been estimated per application and totalled?
- Are backup windows staggered, and has a multi-product restore been tested?
- Is growth reported per account, with an owner for each?
One object store for many backup applications reduces the number of systems to secure and operate. It delivers that benefit when the applications are treated as separate tenants of a shared platform, each isolated, each measured and each tested.