Big Data is a big problem for most IT organizations. There are many storage definitions for Big Data based on arbitrary capacity, performance, or file/object scalability. In general, Big Data refers to any data set the size of which is beyond the capability of traditional storage systems to manage without a significant increase in processing latency.
Big Data is breaking traditional storage systems, causing system sprawl, runaway costs, and a pressing need for suitable resolutions.
Key Challenges for Big Data Storage
Organizations are creating more data today than ever before, with a 62% compound annual growth rate (CAGR), according to the IDC. Unstructured data (such as photos, videos, CAD/CAM diagrams, etc.) represent a highly significant proportion of the new data growth.
Traditional storage is simply not architected to handle this "Internet scale" data. In its attempt to manage this data, traditional storage systems frequently resort to byzantine workarounds that inevitably break, causing significant data loss and loss in productivity. These workarounds also tend to produce system sprawl, leading to exponential increases in operational complexity.
The following obstacles pose enormous difficulties for traditional storage systems when it comes to handling the challenges of Big Data:
- Incapacity to scale storage to levels required by Big Data demand
Traditional solutions lead to storage system sprawl, with excessive expenditures of time and money on management, infrastructure, and ongoing data migrations—all in an ultimately futile effort to chase down the root causes of data storage and retrieval problems.
- Incompatibility with distributed and shared geographic access
Storage solutions must support a wide variety of workgroups, contractors, or customers located in dispersed geographic locations with concurrent local access to verify, edit, change, add, convert, rework, or manipulate content.
- Excessive HA and DR costs and complications
With traditional storage, high availability and disaster recovery are predicated on storing multiple copies of data. Expensive hardware and data center doubling (or even tripling) to accommodate the explosion in data add to storage system complexity and lead ultimately to operator and customer frustration.
- Exceedingly high TCO
Total cost of ownership is based on a model that does not work for Big Data storage.
The "rip-out-and-replace" architecture of traditional storage systems typically requires a tech refresh every 3 to 4 years, with disruptive data migrations, software license upgrades, and the necessary replacement of obsolete or degraded storage capacity. The resulting system sprawl of traditional storage solutions greatly increases the costs of power, cooling, network equipment, rack and floor space, operator retraining, etc.—all contributing to an extraordinarily high TCO. It doesn’t have to be this way.
Essential Private Cloud Storage Requirements
Small, medium, and large enterprises are seeing unprecedented storage consumption today. Just 250 TBs of storage will grow to over a PB of data in less than three years based on the IDC-calculated CAGR of 62%. Something is bound to break. A new approach to data storage is required to meet the needs of exponential data growth. Cloud computing and object storage form the basis of the most promising new approach.
Cloud solutions can be public, private (using low cost off-the-shelf servers, but also able to interface with legacy storage media), or a hybridized mix of public and private servers. Object storage systems should be able to provide extremely fast metadata indexing with striping of file data across several disks and policy mechanisms for automatic tiering and managing data redundancy, and for optimal cost efficiency, erasure coding. They must also be self-healing, automatically reapportioning object keys in the event of disk failures, and avoiding any single points of failure.
At a minimum, a new approach to Big Data storage must also fulfill the following requirements:
- Ability to scale to billions of objects while maintaining performance for all users without disruption
Big Data storage means that storage systems must be able to accommodate PBs to EBs of capacity, and billions of objects or files, with satisfactory performance for millions of concurrent users.
- Support for ongoing nondisruptive tech refreshes
Storage systems must be able to handle equipment updates online, without downtime or manually intensive data migrations.
- Always available and online
Five nines (99.999%) availability all the time.
- Competitive TCO
Storage has become the largest line item for IT budgets, while an increasing climate of austerity is placing increasingly stringent demands to significantly reduce storage costs.
Rip-out-and-replace architectures and their resultant data migrations cause too many headaches that are only getting worse with the growth of Big Data. They must become a thing of the past.
Downtime, scheduled or unscheduled, is no longer acceptable in a 7 x 24 x 365 world, with data that must be protected and available all the time. Ultimately the storage TCO must decline—even as capacity, performance, and file/object storage and retrieval demands are continue to escalate.
The Solution: Scality RING™ Organic Storage
Scality RING Organic Storage is architected from the ground up to meet and exceed all IT organization storage requirements. It scales capacity into the exabytes, files or objects into the billions, and performance that remains steady as new nodes are added. It is designed for Big Data, with no known limits at this time. The scalability of the RING solution is the direct result of its unique Distributed Hash Table (DHT). DHT is an extraordinarily efficient lookup methodology that enables storage and retrieval of very large numbers of files or objects at a very high level of performance.
Scality Organic RING Storage Solution Diagram
Scality RING Organic Storage provides unparalleled data, nodal, and system availability by leveraging its distinctive industry-hardened, carrier-grade peer-to-peer technology. The RING also comes with unequalled built-in system data resilience similar to an organic immune system. Every node constantly monitors a limited number of its peers, automatically rebalancing replicas and load to make the system fully self-healing without human intervention. Consistent hashing guarantees that only a small subset of keys is ever affected by a node failure or removal.
The RING also rebalances the data load automatically when a node fails, is removed or upgraded, or when new nodes are added. RING makes technology refresh a simple, online process with no application disruptions, eliminating data migration, long nights, and sleepless weekends. The result is a very high level of fault tolerance because the system stays reliable even with nodes joining or leaving the ring. Scality RING keeps costs low by enabling the use of standard off-the-shelf commodity server nodes, and through the use of a paradigm-shifting pay-by-the-drink pricing model. Unlike traditional storage, Scality RING charges are based on used capacity, not raw storage capacity, thereby assuring the lowest possible storage TCO.
ABOUT SCALITY
Scality is the developer of RING, a software platform enabling cloud storage to easily scale up to exabytes using commodity server hardware with direct attached storage. Scality delivers the performance and reliability of a SAN- or NAS-based architecture without the aggravations of volume management—at one third to one half the cost.
Business uses for Scality RING include all enterprise storage, Rich Media, Web 2.0, Email (for millions of users), Storage-as-a-Service, and other cloud computing service applications. Deployments manage billion of files while meeting or exceeding the high performance expectations of application users. Scality RING is based on a patented object storage technology that delivers high availability, ease of operations, and total data control.
PARTNER FOR SUCCESS
Partnering with Scality means gaining access to Scality’s extensive storage and market expertise, innovative technologies, pay-as-you-go pricing, intuitive integrated billing, training, business planning, best practices, branding, business tools, and even lead generation. Best of all, service providers are able to continue using the applications and tools at their disposal, enabling and facilitating uninterrupted market growth. This permits investment to match revenue models, providing an immediate ROI, without the limitations of traditional storage solutions.
© 2012 Scality. All rights reserved. Specifications are subject to change without notice. Scality, the Scality logo, Organic Storage, RING, RING Organic Storage, are trademarks or registered trademarks of Scality. in the United States and/or other countries.