fbpx
CUSTOMER TIPS

Copying with Rsync or similar tool

Description

Copying data to the RING using Rsync or similar synchronization tool may increase drastically the metadata RING disk usage and could lead to a service interruption (P1). The main reason is for each directory update, tools are rewritting metadata records which expire by default after seven days. When migrating data to the RING, Scality recommend to lower the purge period on the META RING and to monitor the evolution over the first hours/day.

In this article we will understand:

  • How to prepare and monitor a data migration
  • How to copy with rsync tool
  • How to react if we are already at risk

Preparation

Preparing and monitoring a data migration

Before migrating data to the RING, it is important to know how many content and their size will be copied. You may know that the ratio between small files (<64KB) and biggest files, files will be stored differently with an overhead depending on the RING’s level of data protection. No matter which tool will be used for this, you need to monitor the evolution of your RING’s disk usage.

Monitor the capacity:

  • Through the Supervisor Interface -> Overview: RING Capacity for both META and DATA.
  • Through the Advanced Montiroring: RING Keys for both META and DATA.
  • Through the used tool Rsync or similar synchronization tool).

Solution

Copying With rsync Tool

Rsync is a tool that helps to copy content from a source to a destination. It is frequently used for any kind of project or need. Scality recommends the usage of the option –inplace.

Here is the description of the recommended option –inplace, extracted from rsync manpage:

This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file.

Rsync’s default file copy method implies creating and deleting temporary metadata keys in the metadata RING. With the different protection provided by the Scality solution with the retention and relocation mechanisms, data are not instantanously deleted. Meaning that an important copy could quickly fill the metadata RING and may lead to a service interruption (P1) without the recommanded option –inplace

Reacting if we are already at risk

If the metadata ring is almost full due to this specific behavior, you need to:

  • Stop the copy.
  • Contact Scality Support. Some internal tasks tuning could be performed in order to make them more aggressive and go back to a nominal situation.
  • Once extraneous metadata has been purged from the disks, restart the copy with –inplace the rsync option.

SAN FRANCISCO, USA

149 New Montgomery Street, Suite 607
San Francisco, CA, 94105
Email: sales.us@scality.com
Telephone: +1 (650) 356-8500
Fax: +1 (650) 356-8501
Toll Free: +1 (855) 722-5489


PARIS, FRANCE

11 rue Tronchet
75008 Paris, France
Email: sales.eu@scality.com
Telephone: +33 1 78 09 82 70


WASHINGTON, D.C., USA

43777 Central Station Drive, Suite 410
Ashburn, VA 20147, USA
Email: sales.us@scality.com
Toll Free: +1 (855) 722-5489

TOKYO, JAPAN

Otemachi Bldg. 4F, 1-6-1, Otemachi
Chiyoda-ku Tokyo, 100-0004 Japan
Email: sales.japan@scality.com
Telephone: +81-3-4405-5400


LONDON, UNITED KINGDOM

Quadrant House, Floor 6
4 Thomas More Square
London, United Kingdom, E1W 1YW
Email: sales.eu@scality.com

Products

The most powerful data storage platform.

Protect, search and manage your data on any cloud.

RING

ON PREMISES, PRIVATE CLOUD STORAGE SOLUTION

ZENKO

DATA MANAGEMENT ACROSS
MULTIPLE CLOUDS

NAS ARCHIVER

TIER INACTIVE DATA FROM NAS TO SCALITY RING

Hardware Alliances

Delivering fully integrated solutions.

Learn More

Read about Scality data storage and management solutions.