Glossary

Erasure Coding

Erasure coding is a data protection method that divides data into fragments, creates additional parity fragments, and distributes them across multiple storage devices or locations. If some fragments become unavailable because of drive, node, or other infrastructure failures, the original data can be reconstructed from the remaining fragments.

The technique is widely used in distributed storage systems because it can provide high levels of data durability with less storage overhead than traditional replication. This makes erasure coding particularly useful for large-scale object storage, where organizations may need to protect petabytes of data while managing infrastructure capacity and cost.

How does erasure coding work?

Erasure coding applies a mathematical algorithm to a set of data fragments to generate additional parity fragments. These fragments are distributed across independent failure domains, such as drives, storage nodes, racks, or sites.

An erasure coding configuration is commonly expressed using two values: the number of data fragments and the number of parity fragments. For example, a 10+4 erasure coding scheme divides data into 10 data fragments and creates four parity fragments, producing 14 fragments in total.

In this example, the storage system can reconstruct the original data as long as any 10 of the 14 fragments remain available. Up to four fragments can therefore become unavailable without causing data loss.

The basic process includes:

1. Data segmentation: The storage system divides an object or data set into multiple fragments.
2. Parity calculation: Mathematical algorithms generate additional parity fragments from the original data.
3. Fragment distribution: Data and parity fragments are placed across separate storage resources according to the system’s protection policy.
4. Failure detection: The storage platform identifies missing or inaccessible fragments.
5. Data reconstruction: When necessary, the system uses the surviving fragments to reconstruct the missing data.

The exact implementation varies by storage architecture, but the underlying objective is consistent: maintain recoverability without storing multiple complete copies of every object.

What is the difference between erasure coding and replication?

Replication protects data by maintaining multiple complete copies. A three-way replication policy, for example, stores three copies of the same data. This approach is relatively straightforward and can provide fast access to another copy when hardware fails, but it requires substantially more raw storage capacity.

Erasure coding instead stores data fragments plus parity information. Because the parity fragments are smaller than additional full copies of the data, erasure coding can achieve comparable or higher levels of durability with lower capacity overhead.

For example, three-way replication requires 3 TB of raw capacity to store 1 TB of logical data. A 10+4 erasure coding scheme requires approximately 1.4 TB of raw capacity for every 1 TB of logical data, excluding other system overhead.

The tradeoff is additional computational and network activity. Encoding data and reconstructing unavailable fragments require processing, and recovery may involve retrieving fragments from several storage nodes.

For large object storage environments, the capacity efficiency of erasure coding can make these tradeoffs worthwhile.

Why is erasure coding important for object storage?

Object storage systems are designed to manage large quantities of unstructured data across distributed infrastructure. At this scale, hardware failures are expected operational events rather than exceptional conditions.

Erasure coding allows an object storage platform to distribute protection across this infrastructure. When a drive or node fails, the system can use surviving fragments to reconstruct affected data rather than relying on a complete secondary copy.

This approach provides several benefits.

Storage efficiency

Erasure coding can significantly reduce the raw capacity required for data protection compared with multiple full replicas. The savings become increasingly important as storage environments grow into hundreds of terabytes or petabytes.

Data durability

By distributing fragments across independent failure domains, erasure coding allows data to remain recoverable when multiple infrastructure components become unavailable.

Failure-domain protection

Storage systems can place fragments across drives, nodes, racks, or geographic locations. Proper placement helps prevent a single infrastructure failure from eliminating enough fragments to make an object unrecoverable.

Scalability

Erasure coding is well suited to distributed architectures because protection can span many storage resources rather than depend on dedicated primary and secondary copies.

Cost efficiency

Lower capacity overhead can reduce the amount of storage hardware, rack space, power, and cooling required to maintain a given level of data protection.

What are erasure coding schemes?

An erasure coding scheme defines how many data and parity fragments are created.

A scheme represented as k+m contains:

• k data fragments, which contain the encoded original data
• m parity fragments, which provide redundancy

A 10+4 configuration therefore creates 14 total fragments. A 6+3 configuration creates nine.

Increasing the number of parity fragments generally increases failure tolerance, but it also increases storage overhead. Storage architects therefore select schemes based on durability requirements, infrastructure topology, performance considerations, and the types of failures the system needs to tolerate.

The coding scheme alone does not determine resilience. Fragment placement is equally important. Four parity fragments provide limited protection if too many required fragments reside within the same failure domain.

What happens when a drive or storage node fails?

When infrastructure containing one or more fragments becomes unavailable, the storage system identifies the affected data.

If enough fragments remain accessible, objects can still be read or reconstructed. The system can also generate replacement fragments and place them on healthy storage resources, restoring the intended protection level.

This repair process is sometimes called rebuild, reconstruction, or healing.

In large storage environments, rebuild behavior is an important architectural consideration. A system must recover protection efficiently without allowing reconstruction traffic to place excessive pressure on production workloads.

Distributed object storage architectures can perform reconstruction across multiple nodes and drives, allowing recovery work to use the resources of the broader storage cluster.

What are the limitations of erasure coding?

Erasure coding provides strong protection against infrastructure failures, but it does not address every data protection requirement.

Encoding and reconstruction consume CPU, network, and storage resources. Workloads involving very small objects or frequent modifications may also behave differently from workloads dominated by large, immutable objects.

Erasure coding also protects primarily against data unavailability caused by infrastructure failure. It does not inherently protect against logical threats such as accidental deletion, application corruption, compromised credentials, or ransomware.

For that reason, erasure coding is typically one component of a broader data protection strategy. Capabilities such as immutability, versioning, access controls, authentication, backup, geographic protection, and cyber resilience address different failure and threat scenarios.

Erasure coding and data durability

Data durability describes the probability that stored data will remain intact over time. Erasure coding contributes to durability by allowing data to survive the loss of multiple fragments.

Actual durability depends on several factors, including:

• The selected erasure coding scheme
• Fragment placement across failure domains
• Hardware failure rates
• Failure detection speed
• Reconstruction performance
• The probability of correlated failures
• Geographic distribution where applicable

As a result, evaluating an erasure-coded storage system requires looking beyond the number of parity fragments. The overall storage architecture and its ability to detect, isolate, and repair failures are also important.

How does Scality use erasure coding?

Scality uses erasure coding as part of its approach to protecting data across distributed object storage infrastructure.

With Scality RING, data can be distributed across multiple storage nodes and failure domains using configurable data protection policies. Erasure coding helps organizations maintain data durability while reducing the capacity overhead associated with storing multiple complete replicas.

The distributed architecture also allows storage resources across the system to participate in data protection and recovery. When infrastructure fails, affected data can be reconstructed from surviving fragments and redistributed to restore the required protection level.

Erasure coding works alongside other data protection and cyber resilience capabilities within the Scality architecture. This allows organizations to address hardware resilience, data durability, infrastructure efficiency, and protection against broader data threats as separate but complementary requirements.

For organizations managing large volumes of unstructured data, erasure coding provides a capacity-efficient foundation for maintaining durable data across distributed storage infrastructure.