Configure On-Premise StorageVaults for downloading files and folders in bulk

This guide describes how to enable the Download files and folders in bulk for your On-premise StorageVault.

In order to enable the feature the On-Premise Storage Connector 3.3 or newer has to be installed. See Checking Storage Connector version to understand what version you are currently running.

1. Determine the installation/upgrade scenario

Once the user chooses which files to download in bulk and starts the download, the connector generates a zip file: gathers all the files in a single archive and stores them in a dedicated location on the backend storage. Zip creation consumes hardware resources on the Storage Connector virtual machine. In order for the zip creation not to affect the normal connector operation and user experience, the following must be considered:

1. Whether to deploy dedicated Storage Connector instances to handle only zip creation.
2. How many zips can be created in parallel on a single connector.

Both of these depend on expected load of the Storage Connectors.

For small volume of operations (file uploads, downloads, bulk downloads), where the total expected number of requests in parallel is less than 300 times the number of deployed Storage connectors, an existing Storage Connector infrastructure can be reused. For bigger volume of operations either adjust the number of existing connector instances accordingly, or consider deploying dedicated Storage Connectors to handle only zip archive creation.

When determining the maximum number of zips a single connector can handle in parallel, please refer to Storage Connector configuration parameters and look up the syncplicity.bulkdownload.maxThreads setting.

2. Upgrade to or install the latest version of the Storage Connector

Deployment and upgrade of Storage connectors is described in details in the following article: Deploy the Syncplicity On-Premise Storage Connector. All existing Storage Connectors have to be upgraded to 3.3 or newer version (if available).

By default:

• the feature is enabled on the connectors
• created zip archives are being stored in the data bucket

3. Configure the bulk download settings for the connectors

The following additional configurations are needed.

If zip creation is handled by dedicated Storage Connectors

1. Deploy the expected number of the dedicated Storage Connectors. These shouldn't be behind the load balancer, in order not to receive other operation requests.

2. On the dedicated connectors: disable the cleanup and configure the bulk download operation parameters in /etc/syncp-storage/syncp-storage.yml

   syncplicity:
      bulkdownload:
        enabled: true
        maxThreads: 3
        interval: 30
        compressionLevel: NO_COMPRESSION
   syncplicity.cleanup.enabled: false

   Information on the parameters is available in Storage Connector configuration parameters. maxThreads should be adjusted depending on the instructions.

3. On the rest of the Storage Connectors - the ones that will continue to handle the rest of the operation load - disable the archive creation. In /etc/syncp-storage/syncp-storage.yml:
   

   syncplicity:
      bulkdownload:
        enabled: false

If zip creation is not handled by dedicated Storage Connectors

The only required changes are:

• to adjust syncplicity.bulkdownload.maxThreads setting.
• to revise and adjust the overall number of Storage Connectors to handle the expected load.

Configure storage location for the zip archives

It is highly recommended to store the zip archives on a separate storage location from the default one. In order to do so please:

1. Determine where to store and create a storage location, similar to what has been created for data, image or irm buckets.
2. On all Storage Connector instances, configure the archive storage location: syncplicity.storage.<storage_type>.archive.<property_name>.
3. Please refer to Storage Connector configuration parameters for detailed guidance on the available config properties.
4. The available location should have enough free space to store the created zip archives.

(Optional) Enhance your monitoring solution

In case you're already monitoring the connector health metrics, new metrics are available for this feature. Please check "Bulk Download" section in Storage Connector health metrics definitions.