Define Job Schedule
During job creation, the final step is to define when the job will run and what criteria will define when it stops.
- Save job will launch the job scheduler
- Save job and run it right now will trigger the job to start immediately. It will run every 15 mins after the last execution completes
Schedule Stop Policies
Stop policies determine when a job should stop running. If none of the stop policies are enabled, a scheduled job will continue to run until it is manually stopped or removed.
The options for the stop policy are:
Stop after a number of total runs
The number of total executions before the job will move to "complete" status
Stop after a number of runs with no changes
The job has run and detected no further changes; all content has transferred successfully.
If new content is added to the source and the job runs again, this will not increment your stop policy count. However, job executions that detect no changes do not need to be consecutive to increment your stop policy count.
Stop after a number of failures
This option stops the job after a set number of failures have been logged during a job run. Most failures are resolved through automatic retries. If the retries fail to resolve the failures, the failures are logged in the audit log. Manual intervention is required. This policy takes the job out of rotation so the issue can be investigated. Job executions that detect failures do not need to be consecutive to increment your stop policy count.
This stop policy does not include when jobs fail to start; only failures logged during the job run count toward the stop policy. When a job fails to start, it is generally the result of a configuration issue.
Stop after a specific date
This identifies the last date the job should run. The job will run as scheduled on the specified date and transition to "complete" when the job meets the other scheduled criteria for that day (if set). Setting a stop date does not stop the job from running on the date specified; it prevents subsequent runs after that date.
For example, if you schedule a job to run at 12:00 AM and set the "stop after date" to be January 1, the job will run at 12:00 AM on January 1 and then transition to "complete." Similarly, if you have a job set to run every 15 minutes with a "stop after date" of January 1, the job will run every 15 minutes on January 1 and transition to "complete" when the last job in the schedule runs.
Related:
Stop Policy - DataHub API
Stop Policy allows the user to decide when a scheduled job should stop running based on a defined set of criteria
- "on_success" = No Changes in the DataHub Job Overview User Interface
- "on_failure" = Failures in the DataHub Job Overview User Interface
Global Configuration - Environment Overrides and Precedence
All settings can be set through the command line (--settingname=value) or through (SKYSYNC_settingname=value)
When using environment variables, replace the colon (:) with double underscore (__). So server:port could be set using an environment variable named SKYSYNC_server__port=value. The order of precedence before the database is online is command-line, then environment variables, then appSettings.json. However, once the license and database are online, the order of precedence changes to the license, then the database (Configuration table), then command-line, then environment variables, then appSettings.json.
- License first
- Then database
- Then command-line
- Then environment variables
- Then appSettings.json
Global Stop Policy Configuration
|
Job Configuration
For stop policy purposes success means that during job's executions no actions were taken – i.e. no files/folders were copied and no errors occurred. This typically would happen after previous job executions have copied everything successfully and there are no new files to be copied.
Option |
Description |
Default Value |
---|---|---|
jobs:default_stop_policy:on_success |
The number of successful executions before terminating a job by default
"stop_policy": { "on_success": 3 }
|
(null) |
jobs:default_stop_policy:on_failure |
The number of failed executions before terminating a job by default
"stop_policy": { "on_failure": 3 }
|
(null) |
jobs:default_stop_policy:on_execute |
The number of total executions before terminating a job by default. This is different from "on_failure" and "on_success" in that it will terminate after the number of executions regardless of the status of the executions.
"stop_policy": { "on_execute": 3 }
|
|
jobs:convention_stop_policy:on_success |
The number of successful convention executions before terminating a convention job by default Example:
"stop_policy": { "on_success": 3 }
|
(null) |
jobs:convention_stop_policy:on_failure |
The number of failed convention executions before terminating a convention job by default Example:
"stop_policy": { "on_failure": 3 }
|
(null) |
jobs:convention_stop_policy:on_execute |
The number of total executions before terminating a convention job by default.
"stop_policy": { "on_execute": 3 }
|
A job will be in a failure state if any of the items attempted to be transferred log one or more error level categories in the audit log. Note, the default failure policy is to log error and continue with the execution.
See:
- Job Reports | Item Report > Transfer Audit Category
- Job Failure Policy
Example Job JSON
The following copy job will stop future executions after 3 successful executions. Alternatively "on_failure" can be provided instead of "on_success" to stop executions after given amount of failed executions.
POST v1/jobs
|
How to Reset Job Stop Policy
After a job has reached its stop policy, use one of the following to reset the stop policy. This will allow you to execute the job again.
For example, if you have defined a job stop policy to stop execution after 3 executions, DataHub will stop executing the job after 3 executions. If you want to run the job after the stop policy has been applied, use one of the following options to reset the number of executions and continue to run the job until your job has hit another 3 executions
- Start - start will clear the stop policy and immediately trigger the job to start
- Pause - pause will clear the stop policy and immediately set the job to a paused state
- Resume - resume will clear the stop policy and put the job back into the scheduler queue based on the job's normal schedule
Reset Job Stop Policy | Start Job
|
Reset Job Stop Policy | Pause Job
|
Reset Job Stop Policy | Resume Job
|