Nonstop, exponentially increasing, data storage growth is a well known fact of modern life. This fact applies equally well to government agencies as to the private sector. The challenges of data growth have reached a point where they are overwhelming the capabilities of traditional storage systems. These challenges are far worse with big unstructured data (videos, photos, biometric data, etc.) than with structured data (DBMS), and government unstructured data growth continues to mirror that of other domains.
Key Challenges for Government Data Storage
Traditional storage is multi-user, not multitenant, which may lead to problems sharing data across different government agencies. Security breaches are another top concern for government agencies, yet traditional storage systems rarely encrypt the data saved to or retrieved from storage or replication.
Many government agencies must also be available 7 x 24 x 365. Traditional storage is incapable being always available without incurring incredibly high costs that are impractical for government today. During a tech refresh, data migration alone severely impacts storage uptime and availability. In those cases where traditional storage can meet the challenges of scaling, security, and availability, it does so at such an exceedingly high cost, that it becomes financially untenable in times of economic crisis.
To resolve 21st century government data storage challenges, traditional storage systems must overcome the following handicaps:
- Incapacity to scale to levels required by collected by government agenciesIn their attempt to cope with exponential data growth, traditional solutions lead to storage system sprawl, with excessive expenditures of time and money on management, infrastructure, and ongoing data migrations.
- Incompatibility with requirements for multitenancy and “pay-per-use” billingGovernment departments or agencies with different security levels have access only to their own data, not that of other agencies.
- Excessive HA and DR costs and complicationsWith traditional storage, high availability and disaster recovery are predicated on storing multiple copies of data. To accommodate the explosion in data, expensive hardware and data center additions increase storage system complexity and lead ultimately to operator and end user frustration.
- Excessively high TCOTotal cost of ownership is based on a model that does not work for government organizations in an age of austerity.
Traditional or legacy storage does not make life very easy for government agency storage administrators. It tends to be overly complex, and difficult to expand or scale capacity while maintaining high performance levels. This is especially true when scaling up to the levels commonly required by government agencies. SAN, NAS, and DAS storage technologies were developed at a time when storage growth was nowhere near today’s exponential data expansion rates. This makes it a bit more than a Rubik’s cube type challenge when it comes to meeting contemporary government data scaling requirements.
Essential Government Storage Requirements
Unstructured data stored by government agencies can run to billions of files and occupy petabytes of disk space. Considering that potential users range into the millions, extensive scalability is an absolute must. Government agencies of all types no longer have the option for unsecured data. Internal and external threats continue to multiply, requiring both “in flight” and “at rest” security, often mandated by law or regulation. Another mandatory requirement is “uptime, all the time”, meaning no downtime is allowed for scheduled or unscheduled events.
The need to provide scalability, security, performance, high availability, and disaster recovery in a single package is ever more urgent. Effective storage systems for government agencies must meet the following minimal requirements:
- Ability to scale to billions of objects while maintaining performance for all users without disruptionGovernment data storage systems must be able to provision PBs to EBs of capacity for billions of objects or files, and provide excellent performance regardless of where users are located.
- Provision of secure multitenancy with in-flight and at-rest data encryptionMultitenancy simplifies release management and permits cost savings through resource consolidation. The data mining advantages of multitenancy must be complemented by stringent data encryption protections.
- Always available and onlineFive nines (99.999%) availability all the time.
- Provision of a low TCOReduced budgets are placing ever increasing pressure on government storage assets and resources.
Storage is clearly one of the Government’s largest cost components, with an outsized effect on whether or not requirements can be met. It must be adaptive, flexible, and always online, by providing for transparent technology refreshes and enabling all scheduled and unscheduled maintenance to be performed without user disruption.
Severe budget tightening has put increasing pressure on all governmental agencies to meet or exceed their storage requirements without breaking the bank. This means storage must deliver all of the requirements at the lowest possible cost. With unstable Government funding, and year-to-year variability, a pay-per-use storage system is a better fit than the upfront pricing of traditional storage models (pricing for fixed, preset storage capacity).
The Solution: Scality RING™ Organic Storage
Scality RING Organic Storage is architected from the ground up to meet and exceed all government storage requirements. It scales capacity into the exabytes, files or objects well into the billions, and provides superior performance to tens of thousands of users. This scalability of the RING is the direct result of its unique Distributed Hash Table (DHT). DHT is an extraordinarily efficient lookup methodology that enables very large numbers of storage nodes, capacity, performance, and files/objects.
Scality Organic RING Storage Solution Diagram
Secure multitenancy is built into Scality RING Organic Storage. Scality RING supports in-flight encryption through its RS2 SSL connectors, and at-rest government-grade encryption through the Droplet API.
Scality RING Organic Storage also provides unparalleled data, nodal, and system availability by leveraging its distinctive industry hardened, carrier-grade peer-to-peer technology. It comes with unequalled built-in system data resilience similar to an organic immune system. Every node constantly monitors a limited number of its peers, and automatically rebalances replicas and load to make the system fully self-healing without human intervention. Consistent hashing guarantees that only a small subset of keys is ever affected by a node failure or removal.
The RING rebalances the data load automatically when a node fails, is removed or upgraded, or when new nodes are added. RING makes technology refresh a simple, online process with no application disruptions, eliminating data migration, long nights, and sleepless weekends. The result is a very high level of fault tolerance because the system stays reliable even with nodes joining or leaving the ring.
Just as importantly, costs are kept low by using standard off-the-shelf commodity server nodes, and Scality delivers the storage on a pay-as-you-go (pay-per-use) model that more closely aligns with the way government agency funding models. Unlike traditional storage, agencies pay on a used capacity basis, not on raw storage capacity, thereby benefiting from the lowest possible storage TCO.
© 2012 Scality. All rights reserved. Specifications are subject to change without notice. Scality, the Scality logo, Organic Storage, RING, RING Organic Storage, are trademarks or registered trademarks of Scality. in the United States and/or other countries.