job.proto
path mgmt/v1alpha1/job.proto
package mgmt.v1alpha1
Messages
ActivityFailure
Name | Type | Description |
---|---|---|
message | string |
ActivityOptions
Config that contains various timeouts that are configured in the underlying temporal workflow(s) and activities
Name | Type | Description |
---|---|---|
schedule_to_close_timeout | optional int64 | Total time that a workflow is willing to wait for an activity to complete, including retries. Measured in seconds |
start_to_close_timeout | optional int64 | Max time of a single Temporal Activity execution attempt. This timeout should be as short as the longest psosible execution of any activity (e.g. table sync). Important to know that this is per retry attempt. Defaults to the schedule to close timeout if not provided. Measured in seconds |
retry_policy | RetryPolicy | Optionally define a retry policy for the activity If max attempts is not set, the activity will retry indefinitely until the start to close timeout lapses |
AwsS3DestinationConnectionOptions
Name | Type | Description |
---|
AwsS3SourceConnectionOptions
Name | Type | Description |
---|---|---|
connection_id | string |
CancelJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
CancelJobRunResponse
Name | Type | Description |
---|
CreateJobDestination
Name | Type | Description |
---|---|---|
connection_id | string | |
options | JobDestinationOptions |
CreateJobDestinationConnectionsRequest
Name | Type | Description |
---|---|---|
job_id | string | |
destinations | repeated CreateJobDestination |
CreateJobDestinationConnectionsResponse
Name | Type | Description |
---|---|---|
job | Job |
CreateJobRequest
Name | Type | Description |
---|---|---|
account_id | string | The unique account identifier that this job will be associated with |
job_name | string | The unique, friendly name of the job. This is unique per account |
cron_schedule | optional string | Optionally provide a cron schedule. Goes into effect if the job status is set to enabled |
mappings | repeated JobMapping | |
source | JobSource | |
destinations | repeated CreateJobDestination | |
initiate_job_run | bool | Initially trigger a run of this job regardless of its status or cron schedule |
workflow_options | WorkflowOptions | Specify timeouts and other workflow options for the underlying temporal workflow |
sync_options | ActivityOptions | Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table |
CreateJobResponse
Name | Type | Description |
---|---|---|
job | Job |
CreateJobRunRequest
Name | Type | Description |
---|---|---|
job_id | string |
CreateJobRunResponse
Name | Type | Description |
---|
DeleteJobDestinationConnectionRequest
Name | Type | Description |
---|---|---|
destination_id | string |
DeleteJobDestinationConnectionResponse
Name | Type | Description |
---|
DeleteJobRequest
Name | Type | Description |
---|---|---|
id | string |
DeleteJobResponse
Name | Type | Description |
---|
DeleteJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
DeleteJobRunResponse
Name | Type | Description |
---|
GenerateSourceOptions
Name | Type | Description |
---|---|---|
schemas | repeated GenerateSourceSchemaOption | |
fk_source_connection_id | optional string |
GenerateSourceSchemaOption
Name | Type | Description |
---|---|---|
schema | string | |
tables | repeated GenerateSourceTableOption |
GenerateSourceTableOption
Name | Type | Description |
---|---|---|
table | string | |
row_count | int64 |
GetJobNextRunsRequest
Name | Type | Description |
---|---|---|
job_id | string |
GetJobNextRunsResponse
Name | Type | Description |
---|---|---|
next_runs | JobNextRuns |
GetJobRecentRunsRequest
Name | Type | Description |
---|---|---|
job_id | string |
GetJobRecentRunsResponse
Name | Type | Description |
---|---|---|
recent_runs | repeated JobRecentRun |
GetJobRequest
Name | Type | Description |
---|---|---|
id | string |
GetJobResponse
Name | Type | Description |
---|---|---|
job | Job |
GetJobRunEventsRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
GetJobRunEventsResponse
Name | Type | Description |
---|---|---|
events | repeated JobRunEvent | |
is_run_complete | bool |
GetJobRunLogsStreamRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string | |
window | LogWindow | |
should_tail | bool | |
max_log_lines | optional int64 |
GetJobRunLogsStreamResponse
Name | Type | Description |
---|---|---|
log_line | string |
GetJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
GetJobRunResponse
Name | Type | Description |
---|---|---|
job_run | JobRun |
GetJobRunsRequest
Name | Type | Description |
---|---|---|
job_id | string | |
account_id | string |
GetJobRunsResponse
Name | Type | Description |
---|---|---|
job_runs | repeated JobRun |
GetJobStatusRequest
Name | Type | Description |
---|---|---|
job_id | string |
GetJobStatusResponse
Name | Type | Description |
---|---|---|
status | JobStatus |
GetJobStatusesRequest
Name | Type | Description |
---|---|---|
account_id | string |
GetJobStatusesResponse
Name | Type | Description |
---|---|---|
statuses | repeated JobStatusRecord |
GetJobsRequest
Name | Type | Description |
---|---|---|
account_id | string |
GetJobsResponse
Name | Type | Description |
---|---|---|
jobs | repeated Job |
IsJobNameAvailableRequest
Name | Type | Description |
---|---|---|
name | string | |
account_id | string |
IsJobNameAvailableResponse
Name | Type | Description |
---|---|---|
is_available | bool |
Job
Name | Type | Description |
---|---|---|
id | string | The unique identifier of the job |
created_by_user_id | string | |
created_at | google.protobuf.Timestamp | |
updated_by_user_id | string | |
updated_at | google.protobuf.Timestamp | |
name | string | The unique, friendly name of the job |
source | JobSource | |
destinations | repeated JobDestination | |
mappings | repeated JobMapping | |
cron_schedule | optional string | |
account_id | string | The account identifier that a job is associated with |
sync_options | ActivityOptions | Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table |
workflow_options | WorkflowOptions | Specify timeouts and other workflow options for the underlying temporal workflow |
JobDestination
Name | Type | Description |
---|---|---|
connection_id | string | |
options | JobDestinationOptions | |
id | string |
JobDestinationOptions
Name | Type | Description |
---|---|---|
postgres_options | PostgresDestinationConnectionOptions | |
aws_s3_options | AwsS3DestinationConnectionOptions | |
mysql_options | MysqlDestinationConnectionOptions |
JobMapping
Name | Type | Description |
---|---|---|
schema | string | |
table | string | |
column | string | |
transformer | JobMappingTransformer |
JobMappingTransformer
Name | Type | Description |
---|---|---|
source | string | |
config | TransformerConfig |
JobNextRuns
Name | Type | Description |
---|---|---|
next_run_times | repeated google.protobuf.Timestamp |
JobRecentRun
Name | Type | Description |
---|---|---|
start_time | google.protobuf.Timestamp | |
job_run_id | string |
JobRun
Name | Type | Description |
---|---|---|
id | string | The id of the job run. This will currently be equivalent to the temporal workflow id |
job_id | string | |
name | string | |
status | JobRunStatus | |
started_at | google.protobuf.Timestamp | |
completed_at | optional google.protobuf.Timestamp | |
pending_activities | repeated PendingActivity | Pending activities are only returned when retrieving a specific job run and will not be returned when requesting job runs in list format |
JobRunEvent
Name | Type | Description |
---|---|---|
id | int64 | |
type | string | |
start_time | google.protobuf.Timestamp | |
close_time | google.protobuf.Timestamp | |
metadata | JobRunEventMetadata | |
tasks | repeated JobRunEventTask |
JobRunEventMetadata
Name | Type | Description |
---|---|---|
sync_metadata | JobRunSyncMetadata |
JobRunEventTask
Name | Type | Description |
---|---|---|
id | int64 | |
type | string | |
event_time | google.protobuf.Timestamp | |
error | JobRunEventTaskError |
JobRunEventTaskError
Name | Type | Description |
---|---|---|
message | string | |
retry_state | string |
JobRunSyncMetadata
Name | Type | Description |
---|---|---|
schema | string | |
table | string |
JobSource
Name | Type | Description |
---|---|---|
options | JobSourceOptions |
JobSourceOptions
Name | Type | Description |
---|---|---|
postgres | PostgresSourceConnectionOptions | |
aws_s3 | AwsS3SourceConnectionOptions | |
mysql | MysqlSourceConnectionOptions | |
generate | GenerateSourceOptions |
JobSourceSqlSubetSchemas
Name | Type | Description |
---|---|---|
postgres_subset | PostgresSourceSchemaSubset | |
mysql_subset | MysqlSourceSchemaSubset |
JobStatusRecord
Name | Type | Description |
---|---|---|
job_id | string | |
status | JobStatus |
MysqlDestinationConnectionOptions
Name | Type | Description |
---|---|---|
truncate_table | MysqlTruncateTableConfig | |
init_table_schema | bool |
MysqlSourceConnectionOptions
Name | Type | Description |
---|---|---|
halt_on_new_column_addition | bool | |
schemas | repeated MysqlSourceSchemaOption | |
connection_id | string |
MysqlSourceSchemaOption
Name | Type | Description |
---|---|---|
schema | string | |
tables | repeated MysqlSourceTableOption |
MysqlSourceSchemaSubset
Name | Type | Description |
---|---|---|
mysql_schemas | repeated MysqlSourceSchemaOption |
MysqlSourceTableOption
Name | Type | Description |
---|---|---|
table | string | |
where_clause | optional string |
MysqlTruncateTableConfig
Name | Type | Description |
---|---|---|
truncate_before_insert | bool |
PauseJobRequest
Name | Type | Description |
---|---|---|
id | string | |
pause | bool | |
note | optional string |
PauseJobResponse
Name | Type | Description |
---|---|---|
job | Job |
PendingActivity
Name | Type | Description |
---|---|---|
status | ActivityStatus | |
activity_name | string | |
last_failure | optional ActivityFailure |
PostgresDestinationConnectionOptions
Name | Type | Description |
---|---|---|
truncate_table | PostgresTruncateTableConfig | |
init_table_schema | bool |
PostgresSourceConnectionOptions
Name | Type | Description |
---|---|---|
halt_on_new_column_addition | bool | |
schemas | repeated PostgresSourceSchemaOption | |
connection_id | string |
PostgresSourceSchemaOption
Name | Type | Description |
---|---|---|
schema | string | |
tables | repeated PostgresSourceTableOption |
PostgresSourceSchemaSubset
Name | Type | Description |
---|---|---|
postgres_schemas | repeated PostgresSourceSchemaOption |
PostgresSourceTableOption
Name | Type | Description |
---|---|---|
table | string | |
where_clause | optional string |
PostgresTruncateTableConfig
Name | Type | Description |
---|---|---|
truncate_before_insert | bool | |
cascade | bool |
RetryPolicy
Defines the retry policy for an activity
Name | Type | Description |
---|---|---|
maximum_attempts | optional int32 | Maximum number of attempts. When exceeded the retries stop even if not expired yet. If not set or set to 0, it means unlimited, and rely on activity ScheduleToCloseTimeout to stop. |
SetJobSourceSqlConnectionSubsetsRequest
Name | Type | Description |
---|---|---|
id | string | |
schemas | JobSourceSqlSubetSchemas |
SetJobSourceSqlConnectionSubsetsResponse
Name | Type | Description |
---|---|---|
job | Job |
SetJobSyncOptionsRequest
Name | Type | Description |
---|---|---|
id | string | The unique identifier of the job |
sync_options | ActivityOptions | The sync options object. The entire object must be provided and will fully overwrite the previous result |
SetJobSyncOptionsResponse
Name | Type | Description |
---|---|---|
job | Job |
SetJobWorkflowOptionsRequest
Name | Type | Description |
---|---|---|
id | string | The unique identifier of the job |
worfklow_options | WorkflowOptions | The workflow options object. The entire object must be provided and will fully overwrite the previous result |
SetJobWorkflowOptionsResponse
Name | Type | Description |
---|---|---|
job | Job |
TerminateJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
TerminateJobRunResponse
Name | Type | Description |
---|
UpdateJobDestinationConnectionRequest
Name | Type | Description |
---|---|---|
job_id | string | |
connection_id | string | |
options | JobDestinationOptions | |
destination_id | string |
UpdateJobDestinationConnectionResponse
Name | Type | Description |
---|---|---|
job | Job |
UpdateJobScheduleRequest
Name | Type | Description |
---|---|---|
id | string | |
cron_schedule | optional string |
UpdateJobScheduleResponse
Name | Type | Description |
---|---|---|
job | Job |
UpdateJobSourceConnectionRequest
Name | Type | Description |
---|---|---|
id | string | |
source | JobSource | |
mappings | repeated JobMapping |
UpdateJobSourceConnectionResponse
Name | Type | Description |
---|---|---|
job | Job |
WorkflowOptions
Config that contains various timeouts that are configured in the underlying temporal workflow More options will come in the future as needed
Name | Type | Description |
---|---|---|
run_timeout | optional int64 | The timeout for a single workflow run. Measured in seconds |
Enums
ActivityStatus
Name | Number | Description |
---|---|---|
ACTIVITY_STATUS_UNSPECIFIED | 0 | |
ACTIVITY_STATUS_SCHEDULED | 1 | |
ACTIVITY_STATUS_STARTED | 2 | |
ACTIVITY_STATUS_CANCELED | 3 | |
ACTIVITY_STATUS_FAILED | 4 |
JobRunStatus
Name | Number | Description |
---|---|---|
JOB_RUN_STATUS_UNSPECIFIED | 0 | |
JOB_RUN_STATUS_PENDING | 1 | |
JOB_RUN_STATUS_RUNNING | 2 | |
JOB_RUN_STATUS_COMPLETE | 3 | |
JOB_RUN_STATUS_ERROR | 4 | |
JOB_RUN_STATUS_CANCELED | 5 | |
JOB_RUN_STATUS_TERMINATED | 6 | |
JOB_RUN_STATUS_FAILED | 7 |
JobStatus
Name | Number | Description |
---|---|---|
JOB_STATUS_UNSPECIFIED | 0 | |
JOB_STATUS_ENABLED | 1 | |
JOB_STATUS_PAUSED | 3 | |
JOB_STATUS_DISABLED | 4 |
LogWindow
Name | Number | Description |
---|---|---|
LOG_WINDOW_NO_TIME_UNSPECIFIED | 0 | |
LOG_WINDOW_FIFTEEN_MIN | 1 | |
LOG_WINDOW_ONE_HOUR | 2 | |
LOG_WINDOW_ONE_DAY | 3 |
Services
JobService
GetJobs
Method | GetJobs |
---|---|
Request | GetJobsRequest |
Response | GetJobsResponse |
Description |
GetJob
Method | GetJob |
---|---|
Request | GetJobRequest |
Response | GetJobResponse |
Description |
CreateJob
Method | CreateJob |
---|---|
Request | CreateJobRequest |
Response | CreateJobResponse |
Description |
DeleteJob
Method | DeleteJob |
---|---|
Request | DeleteJobRequest |
Response | DeleteJobResponse |
Description |
IsJobNameAvailable
Method | IsJobNameAvailable |
---|---|
Request | IsJobNameAvailableRequest |
Response | IsJobNameAvailableResponse |
Description |
UpdateJobSchedule
Method | UpdateJobSchedule |
---|---|
Request | UpdateJobScheduleRequest |
Response | UpdateJobScheduleResponse |
Description |
UpdateJobSourceConnection
Method | UpdateJobSourceConnection |
---|---|
Request | UpdateJobSourceConnectionRequest |
Response | UpdateJobSourceConnectionResponse |
Description |
SetJobSourceSqlConnectionSubsets
Method | SetJobSourceSqlConnectionSubsets |
---|---|
Request | SetJobSourceSqlConnectionSubsetsRequest |
Response | SetJobSourceSqlConnectionSubsetsResponse |
Description |
UpdateJobDestinationConnection
Method | UpdateJobDestinationConnection |
---|---|
Request | UpdateJobDestinationConnectionRequest |
Response | UpdateJobDestinationConnectionResponse |
Description |
DeleteJobDestinationConnection
Method | DeleteJobDestinationConnection |
---|---|
Request | DeleteJobDestinationConnectionRequest |
Response | DeleteJobDestinationConnectionResponse |
Description |
CreateJobDestinationConnections
Method | CreateJobDestinationConnections |
---|---|
Request | CreateJobDestinationConnectionsRequest |
Response | CreateJobDestinationConnectionsResponse |
Description |
PauseJob
Method | PauseJob |
---|---|
Request | PauseJobRequest |
Response | PauseJobResponse |
Description |
GetJobRecentRuns
Method | GetJobRecentRuns |
---|---|
Request | GetJobRecentRunsRequest |
Response | GetJobRecentRunsResponse |
Description | Returns a list of recently invoked job runs baseds on the Temporal cron scheduler. This will return a list of job runs that include archived runs |
GetJobNextRuns
Method | GetJobNextRuns |
---|---|
Request | GetJobNextRunsRequest |
Response | GetJobNextRunsResponse |
Description | Returns a list of runs that are scheduled for execution based on the Temporal cron scheduler. |
GetJobStatus
Method | GetJobStatus |
---|---|
Request | GetJobStatusRequest |
Response | GetJobStatusResponse |
Description |
GetJobStatuses
Method | GetJobStatuses |
---|---|
Request | GetJobStatusesRequest |
Response | GetJobStatusesResponse |
Description |
GetJobRuns
Method | GetJobRuns |
---|---|
Request | GetJobRunsRequest |
Response | GetJobRunsResponse |
Description | Returns a list of job runs by either account or job |
GetJobRunEvents
Method | GetJobRunEvents |
---|---|
Request | GetJobRunEventsRequest |
Response | GetJobRunEventsResponse |
Description |
GetJobRun
Method | GetJobRun |
---|---|
Request | GetJobRunRequest |
Response | GetJobRunResponse |
Description | Returns a specific job run, along with any of its pending activities |
DeleteJobRun
Method | DeleteJobRun |
---|---|
Request | DeleteJobRunRequest |
Response | DeleteJobRunResponse |
Description |
CreateJobRun
Method | CreateJobRun |
---|---|
Request | CreateJobRunRequest |
Response | CreateJobRunResponse |
Description |
CancelJobRun
Method | CancelJobRun |
---|---|
Request | CancelJobRunRequest |
Response | CancelJobRunResponse |
Description |
TerminateJobRun
Method | TerminateJobRun |
---|---|
Request | TerminateJobRunRequest |
Response | TerminateJobRunResponse |
Description |
GetJobRunLogsStream
Method | GetJobRunLogsStream |
---|---|
Request | GetJobRunLogsStreamRequest |
Response | GetJobRunLogsStreamResponse stream |
Description | Returns a stream of logs from the worker nodes that pertain to a specific job run |
SetJobWorkflowOptions
Method | SetJobWorkflowOptions |
---|---|
Request | SetJobWorkflowOptionsRequest |
Response | SetJobWorkflowOptionsResponse |
Description | Set any job workflow options. Must provide entire object as is it will fully override the previous configuration |
SetJobSyncOptions
Method | SetJobSyncOptions |
---|---|
Request | SetJobSyncOptionsRequest |
Response | SetJobSyncOptionsResponse |
Description | Set the job sync options. Must provide entire object as it will fully override the previous configuration |