When transferring data between a source and destination there are a number of factors which can limit the transfer speed. Most cloud providers have rate limitations which reduce the transfer rate but if those limits are account based and it supports impersonation DataHub can create a pool of accounts that issues commands in a round-robin format across all of the accounts connected to the pool. Any modifications to the connection pool will used on the next job run.
For example, if a connection pool has 2 accounts all commands will be alternated between them. If a third account is added to the pool, the next run of the job will use all three accounts.
| Not Supported: "My Computer" and Network File Share (NFS) connections are not supported with connection pooling |
| Connections using OAuth 2 authentication may experience bandwidth throttling when using connection pooling. |
REST API - Creating a Connection Pool
Instructions
- Create connections which should be in a connection pool. Keep track of connection ID's.
-
Using one of the connection IDs from previous step, create connection pool using REST API:
Create Connection Pool
|
|
Note the pool:id in the response to the above command. This is the new connection Pool ID to be used when creating new jobs and to add more connections to the pool.
-
Add more connections (one by one) to the pool via REST API
Add Another Connection to the Pool
|
|
- Create transfer job using {{connectionPool_id}} as the connection id.
REST API - Editing Connection Pool
Connection pool can be queried and deleted using same REST API as regular connections.
Connection can be added per steps above or deleted via REST API:
|
|