Copying data to the RING using Rsync or similar synchronization tool may increase drastically the metadata RING disk usage and could lead to a service interruption (P1). The main reason is for each directory update, tools are rewritting metadata records which expire by default after seven days. When migrating data to the RING, Scality recommend to lower the purge period on the META RING and to monitor the evolution over the first hours/day.

In this article we will understand:

• How to prepare and monitor a data migration
• How to copy with rsync tool
• How to react if we are already at risk

Preparing and monitoring a data migration

Before migrating data to the RING, it is important to know how many content and their size will be copied. You may know that the ratio between small files (<64KB) and biggest files, files will be stored differently with an overhead depending on the RING’s level of data protection. No matter which tool will be used for this, you need to monitor the evolution of your RING’s disk usage.

Monitor the capacity:

• Through the Supervisor Interface -> Overview: RING Capacity for both META and DATA.
• Through the Advanced Montiroring: RING Keys for both META and DATA.
• Through the used tool Rsync or similar synchronization tool).

Copying With rsync Tool

Rsync is a tool that helps to copy content from a source to a destination. It is frequently used for any kind of project or need. Scality recommends the usage of the option –inplace.

Here is the description of the recommended option –inplace, extracted from rsync manpage:

This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file.

Rsync’s default file copy method implies creating and deleting temporary metadata keys in the metadata RING. With the different protection provided by the Scality solution with the retention and relocation mechanisms, data are not instantanously deleted. Meaning that an important copy could quickly fill the metadata RING and may lead to a service interruption (P1) without the recommanded option –inplace

Reacting if we are already at risk

If the metadata ring is almost full due to this specific behavior, you need to:

• Stop the copy.
• Contact Scality Support. Some internal tasks tuning could be performed in order to make them more aggressive and go back to a nominal situation.
• Once extraneous metadata has been purged from the disks, restart the copy with –inplace the rsync option.