FastTrack RING Expansions General Principles
This KB article will focus on the expansion during which servers are added to the RING following one of the FastTrack paths.
- A RING is composed of several storage servers running six node processes each.
- Each node process of each server is responsible for a slice of the RING, called the key range of a node.
- The sum of the nodes’ key ranges of each server is proportional to the capacity of each server. For example, the sum of the key ranges of a server with 500TB of disk capacity will be twice the sum of the key ranges of a server with 250TB of capacity.
- The key ranges of the nodes composing a RING take into account the different failure scenarios defined during the implementation (or after a configuration change) so data is still available in case one of those scenarios may happen. For example, in ARC 9+3 with 12 servers, we make sure we do not lose any data after the loss of up to three servers (or three disks on three different servers) at the same time.
For further definitions, refer to the Scality Glossary.
When performing an expansion, the key ranges assigned to the new and existing nodes must respect:
- The failure scenarios
- The storage capacity of each server
During an expansion, the nodes of the new servers are added to the RING and take a part of the key ranges of the existing nodes. Then occurs “balances”: data is moved through the network from one server to another automatically by the node processes to balance the data across the different servers. At the end of an expansion, all the servers have the same percentage of data stored.
Depending on the amount of data to be moved, balancing can last multiple weeks but this does not impact the availability of the data or the service. The new capacity added is immediately available for the applications or end users.
When adding new servers to a RING, a very specific process must be respected.
Expansion scenario is validated by Scality teams (number of servers added + capacity of those servers).
New keyspace is computed and validated by Scality teams.
New servers are installed and configured (OS, Network, RAID controllers, disks, etc.). See the Requirements for more details.
Software is deployed and configured by Scality teams.
Environment is validated (hardware, network, software, OS, etc) by Scality teams.
New servers are added to the RING and the keys of the nodes are assigned by Scality teams.
The following can be monitored during an expansion:
- The disk usage of all servers must be checked to make sure some servers are not filling up abnormally: the storage used on the servers should not exceed 85%. If that went to happen, contact Scality support team so they can confirm the normal behavior or take actions if necessary.
- The latency at the application level must be checked. If an unexpected higher latency is observed, a throttling of the tasks can be applied to limit such an impact. This can be done directly in the Administration tab of the RING Administration Interface.
If this does not solve your issues please get in touch with support