File: //usr/lib/python3/dist-packages/boto/datapipeline/__pycache__/layer1.cpython-310.pyc
o
ckF[^q � @ sX d dl Z d dlmZ d dlmZ d dlmZ d dlmZ d dl m
Z
G dd� de�Z dS ) � N)�json)�AWSQueryConnection)�
RegionInfo)�JSONResponseError)�
exceptionsc s e Zd ZdZdZdZdZdZdZe Z
e j e j
e je je jd�Z� fdd�Zd d
� Zd d � Zd.dd�Zdd� Z
d/dd�Zdd� Zdd� Zd.dd�Zd.dd�Z
d/d d �Z d d � Z
d0d d!�Z d"d#� Z
d/d$d%�Z d&d'� Z!
d0d(d)�Z"d*d+� Z#d,d-� Z$� Z%S )1�DataPipelineConnectiona/
This is the AWS Data Pipeline API Reference . This guide provides
descriptions and samples of the AWS Data Pipeline API.
AWS Data Pipeline is a web service that configures and manages a
data-driven workflow called a pipeline. AWS Data Pipeline handles
the details of scheduling and ensuring that data dependencies are
met so your application can focus on processing the data.
The AWS Data Pipeline API implements two main sets of
functionality. The first set of actions configure the pipeline in
the web service. You call these actions to create a pipeline and
define data sources, schedules, dependencies, and the transforms
to be performed on the data.
The second set of actions are used by a task runner application
that calls the AWS Data Pipeline API to receive the next task
ready for processing. The logic for performing the task, such as
querying the data, running data analysis, or converting the data
from one format to another, is contained within the task runner.
The task runner performs the task assigned to it by the web
service, reporting progress to the web service as it does so. When
the task is done, the task runner reports the final success or
failure of the task to the web service.
AWS Data Pipeline provides an open-source implementation of a task
runner called AWS Data Pipeline Task Runner. AWS Data Pipeline
Task Runner provides logic for common data management scenarios,
such as performing database queries and running data analysis
using Amazon Elastic MapReduce (Amazon EMR). You can use AWS Data
Pipeline Task Runner as your task runner, or you can write your
own task runner to provide custom data management.
The AWS Data Pipeline API uses the Signature Version 4 protocol
for signing requests. For more information about how to sign a
request with this protocol, see `Signature Version 4 Signing
Process`_. In the code examples in this reference, the Signature
Version 4 Request parameters are represented as AuthParams.
z
2012-10-29z us-east-1z$datapipeline.us-east-1.amazonaws.com� DataPipeline)�PipelineDeletedException�InvalidRequestException�TaskNotFoundException�PipelineNotFoundException�InternalServiceErrorc sJ |� dd �}|st| | j| j�}|j|d< tt| �jdi |�� || _d S )N�region�host� ) �popr �DefaultRegionName�DefaultRegionEndpoint�endpoint�superr �__init__r )�self�kwargsr �� __class__r �:/usr/lib/python3/dist-packages/boto/datapipeline/layer1.pyr V s �
z DataPipelineConnection.__init__c C s dgS )Nzhmac-v4r )r r r r �_required_auth_capability_ s z0DataPipelineConnection._required_auth_capabilityc C � d|i}| j dt�|�d�S )a�
Validates a pipeline and initiates processing. If the pipeline
does not pass validation, activation fails.
Call this action to start processing pipeline tasks of a
pipeline you've created using the CreatePipeline and
PutPipelineDefinition actions. A pipeline cannot be modified
after it has been successfully activated.
:type pipeline_id: string
:param pipeline_id: The identifier of the pipeline to activate.
�
pipelineId�ActivatePipeline��action�body�� make_requestr �dumps�r � pipeline_id�paramsr r r �activate_pipelineb s �z(DataPipelineConnection.activate_pipelineNc C s. ||d�}|dur
||d< | j dt�|�d�S )a(
Creates a new empty pipeline. When this action succeeds, you
can then use the PutPipelineDefinition action to populate the
pipeline.
:type name: string
:param name: The name of the new pipeline. You can use the same name
for multiple pipelines associated with your AWS account, because
AWS Data Pipeline assigns each new pipeline a unique pipeline
identifier.
:type unique_id: string
:param unique_id: A unique identifier that you specify. This identifier
is not the same as the pipeline identifier assigned by AWS Data
Pipeline. You are responsible for defining the format and ensuring
the uniqueness of this identifier. You use this parameter to ensure
idempotency during repeated calls to CreatePipeline. For example,
if the first call to CreatePipeline does not return a clear
success, you can pass in the same unique identifier and pipeline
name combination on a subsequent call to CreatePipeline.
CreatePipeline ensures that if a pipeline already exists with the
same name and unique identifier, a new pipeline will not be
created. Instead, you'll receive the pipeline identifier from the
previous attempt. The uniqueness of the name and unique identifier
combination is scoped to the AWS account or IAM user credentials.
:type description: string
:param description: The description of the new pipeline.
)�name�uniqueIdN� description�CreatePipeliner r# )r r* � unique_idr, r( r r r �create_pipelinet s
�z&DataPipelineConnection.create_pipelinec C r )a�
Permanently deletes a pipeline, its pipeline definition and
its run history. You cannot query or restore a deleted
pipeline. AWS Data Pipeline will attempt to cancel instances
associated with the pipeline that are currently being
processed by task runners. Deleting a pipeline cannot be
undone.
To temporarily pause a pipeline instead of deleting it, call
SetStatus with the status set to Pause on individual
components. Components that are paused by SetStatus can be
resumed.
:type pipeline_id: string
:param pipeline_id: The identifier of the pipeline to be deleted.
r �DeletePipeliner r# r& r r r �delete_pipeline� s �z&DataPipelineConnection.delete_pipelinec C s> ||d�}|dur
||d< |dur||d< | j dt�|�d�S )a�
Returns the object definitions for a set of objects associated
with the pipeline. Object definitions are composed of a set of
fields that define the properties of the object.
:type pipeline_id: string
:param pipeline_id: Identifier of the pipeline that contains the object
definitions.
:type object_ids: list
:param object_ids: Identifiers of the pipeline objects that contain the
definitions to be described. You can pass as many as 25 identifiers
in a single call to DescribeObjects.
:type evaluate_expressions: boolean
:param evaluate_expressions: Indicates whether any expressions in the
object should be evaluated when the object descriptions are
returned.
:type marker: string
:param marker: The starting point for the results to be returned. The
first time you call DescribeObjects, this value should be empty. As
long as the action returns `HasMoreResults` as `True`, you can call
DescribeObjects again and pass the marker value from the response
to retrieve the next set of results.
)r � objectIdsN�evaluateExpressions�marker�DescribeObjectsr r# )r �
object_idsr' r4 �evaluate_expressionsr( r r r �describe_objects� s ��z'DataPipelineConnection.describe_objectsc C r )a�
Retrieve metadata about one or more pipelines. The information
retrieved includes the name of the pipeline, the pipeline
identifier, its current state, and the user account that owns
the pipeline. Using account credentials, you can retrieve
metadata about pipelines that you or your IAM users have
created. If you are using an IAM user account, you can
retrieve metadata about only those pipelines you have read
permission for.
To retrieve the full pipeline definition instead of metadata
about the pipeline, call the GetPipelineDefinition action.
:type pipeline_ids: list
:param pipeline_ids: Identifiers of the pipelines to describe. You can
pass as many as 25 identifiers in a single call to
DescribePipelines. You can obtain pipeline identifiers by calling
ListPipelines.
� pipelineIds�DescribePipelinesr r# )r � pipeline_idsr( r r r �describe_pipelines� s �z)DataPipelineConnection.describe_pipelinesc C s |||d�}| j dt�|�d�S )a�
Evaluates a string in the context of a specified object. A
task runner can use this action to evaluate SQL queries stored
in Amazon S3.
:type pipeline_id: string
:param pipeline_id: The identifier of the pipeline.
:type object_id: string
:param object_id: The identifier of the object.
:type expression: string
:param expression: The expression to evaluate.
)r �objectId�
expression�EvaluateExpressionr r# )r r' r> � object_idr( r r r �evaluate_expression� s ��z*DataPipelineConnection.evaluate_expressionc C s, d|i}|dur ||d< | j dt�|�d�S )a�
Returns the definition of the specified pipeline. You can call
GetPipelineDefinition to retrieve the pipeline definition you
provided using PutPipelineDefinition.
:type pipeline_id: string
:param pipeline_id: The identifier of the pipeline.
:type version: string
:param version: The version of the pipeline definition to retrieve.
This parameter accepts the values `latest` (default) and `active`.
Where `latest` indicates the last definition saved to the pipeline
and `active` indicates the last definition of the pipeline that was
activated.
r N�version�GetPipelineDefinitionr r# )r r' rB r( r r r �get_pipeline_definition s �z.DataPipelineConnection.get_pipeline_definitionc C s( i }|dur
||d< | j dt�|�d�S )a5
Returns a list of pipeline identifiers for all active
pipelines. Identifiers are returned only for pipelines you
have permission to access.
:type marker: string
:param marker: The starting point for the results to be returned. The
first time you call ListPipelines, this value should be empty. As
long as the action returns `HasMoreResults` as `True`, you can call
ListPipelines again and pass the marker value from the response to
retrieve the next set of results.
Nr4 �
ListPipelinesr r# )r r4 r( r r r �list_pipelines s �z%DataPipelineConnection.list_pipelinesc C �< d|i}|dur ||d< |dur||d< | j dt�|�d�S )a�
Task runners call this action to receive a task to perform
from AWS Data Pipeline. The task runner specifies which tasks
it can perform by setting a value for the workerGroup
parameter of the PollForTask call. The task returned by
PollForTask may come from any of the pipelines that match the
workerGroup value passed in by the task runner and that was
launched using the IAM user credentials specified by the task
runner.
If tasks are ready in the work queue, PollForTask returns a
response immediately. If no tasks are available in the queue,
PollForTask uses long-polling and holds on to a poll
connection for up to a 90 seconds during which time the first
newly scheduled task is handed to the task runner. To
accomodate this, set the socket timeout in your task runner to
90 seconds. The task runner should not call PollForTask again
on the same `workerGroup` until it receives a response, and
this may take up to 90 seconds.
:type worker_group: string
:param worker_group: Indicates the type of task the task runner is
configured to accept and process. The worker group is set as a
field on objects in the pipeline when they are created. You can
only specify a single value for `workerGroup` in the call to
PollForTask. There are no wildcard values permitted in
`workerGroup`, the string must be an exact, case-sensitive, match.
:type hostname: string
:param hostname: The public DNS name of the calling task runner.
:type instance_identity: dict
:param instance_identity: Identity information for the Amazon EC2
instance that is hosting the task runner. You can get this value by
calling the URI, `http://169.254.169.254/latest/meta-data/instance-
id`, from the EC2 instance. For more information, go to `Instance
Metadata`_ in the Amazon Elastic Compute Cloud User Guide. Passing
in this value proves that your task runner is running on an EC2
instance, and ensures the proper AWS Data Pipeline service charges
are applied to your pipeline.
� workerGroupN�hostname�instanceIdentity� PollForTaskr r# )r � worker_grouprI �instance_identityr( r r r �
poll_for_task3 s ,�z$DataPipelineConnection.poll_for_taskc C � ||d�}| j dt�|�d�S )aj
Adds tasks, schedules, and preconditions that control the
behavior of the pipeline. You can use PutPipelineDefinition to
populate a new pipeline or to update an existing pipeline that
has not yet been activated.
PutPipelineDefinition also validates the configuration as it
adds it to the pipeline. Changes to the pipeline are saved
unless one of the following three validation errors exists in
the pipeline.
#. An object is missing a name or identifier field.
#. A string or reference field is empty.
#. The number of objects in the pipeline exceeds the maximum
allowed objects.
Pipeline object definitions are passed to the
PutPipelineDefinition action and returned by the
GetPipelineDefinition action.
:type pipeline_id: string
:param pipeline_id: The identifier of the pipeline to be configured.
:type pipeline_objects: list
:param pipeline_objects: The objects that define the pipeline. These
will overwrite the existing pipeline definition.
�r �pipelineObjects�PutPipelineDefinitionr r# �r �pipeline_objectsr' r( r r r �put_pipeline_definitiong s ��z.DataPipelineConnection.put_pipeline_definitionc C sN ||d�}|dur
||d< |dur||d< |dur ||d< | j dt�|�d�S )a�
Queries a pipeline for the names of objects that match a
specified set of conditions.
The objects returned by QueryObjects are paginated and then
filtered by the value you set for query. This means the action
may return an empty result set with a value set for marker. If
`HasMoreResults` is set to `True`, you should continue to call
QueryObjects, passing in the returned value for marker, until
`HasMoreResults` returns `False`.
:type pipeline_id: string
:param pipeline_id: Identifier of the pipeline to be queried for object
names.
:type query: dict
:param query: Query that defines the objects to be returned. The Query
object can contain a maximum of ten selectors. The conditions in
the query are limited to top-level String fields in the object.
These filters can be applied to components, instances, and
attempts.
:type sphere: string
:param sphere: Specifies whether the query applies to components or
instances. Allowable values: `COMPONENT`, `INSTANCE`, `ATTEMPT`.
:type marker: string
:param marker: The starting point for the results to be returned. The
first time you call QueryObjects, this value should be empty. As
long as the action returns `HasMoreResults` as `True`, you can call
QueryObjects again and pass the marker value from the response to
retrieve the next set of results.
:type limit: integer
:param limit: Specifies the maximum number of object names that
QueryObjects will return in a single call. The default value is
100.
)r �sphereN�queryr4 �limit� QueryObjectsr r# )r r' rV r4 rW rX r( r r r �
query_objects� �
)�z$DataPipelineConnection.query_objectsc C r )a�
Updates the AWS Data Pipeline service on the progress of the
calling task runner. When the task runner is assigned a task,
it should call ReportTaskProgress to acknowledge that it has
the task within 2 minutes. If the web service does not recieve
this acknowledgement within the 2 minute window, it will
assign the task in a subsequent PollForTask call. After this
initial acknowledgement, the task runner only needs to report
progress every 15 minutes to maintain its ownership of the
task. You can change this reporting time from 15 minutes by
specifying a `reportProgressTimeout` field in your pipeline.
If a task runner does not report its status after 5 minutes,
AWS Data Pipeline will assume that the task runner is unable
to process the task and will reassign the task in a subsequent
response to PollForTask. task runners should call
ReportTaskProgress every 60 seconds.
:type task_id: string
:param task_id: Identifier of the task assigned to the task runner.
This value is provided in the TaskObject that the service returns
with the response for the PollForTask action.
�taskId�ReportTaskProgressr r# )r �task_idr( r r r �report_task_progress� s �z+DataPipelineConnection.report_task_progressc C rG )a�
Task runners call ReportTaskRunnerHeartbeat every 15 minutes
to indicate that they are operational. In the case of AWS Data
Pipeline Task Runner launched on a resource managed by AWS
Data Pipeline, the web service can use this call to detect
when the task runner application has failed and restart a new
instance.
:type taskrunner_id: string
:param taskrunner_id: The identifier of the task runner. This value
should be unique across your AWS account. In the case of AWS Data
Pipeline Task Runner launched on a resource managed by AWS Data
Pipeline, the web service provides a unique identifier when it
launches the application. If you have written a custom task runner,
you should assign a unique identifier for the task runner.
:type worker_group: string
:param worker_group: Indicates the type of task the task runner is
configured to accept and process. The worker group is set as a
field on objects in the pipeline when they are created. You can
only specify a single value for `workerGroup` in the call to
ReportTaskRunnerHeartbeat. There are no wildcard values permitted
in `workerGroup`, the string must be an exact, case-sensitive,
match.
:type hostname: string
:param hostname: The public DNS name of the calling task runner.
� taskrunnerIdNrH rI �ReportTaskRunnerHeartbeatr r# )r �
taskrunner_idrL rI r( r r r � report_task_runner_heartbeat� s �z3DataPipelineConnection.report_task_runner_heartbeatc C s |||d�}| j dt�|�d�S )ap
Requests that the status of an array of physical or logical
pipeline objects be updated in the pipeline. This update may
not occur immediately, but is eventually consistent. The
status that can be set depends on the type of object.
:type pipeline_id: string
:param pipeline_id: Identifies the pipeline that contains the objects.
:type object_ids: list
:param object_ids: Identifies an array of objects. The corresponding
objects can be either physical or components, but not a mix of both
types.
:type status: string
:param status: Specifies the status to be set on all the objects in
`objectIds`. For components, this can be either `PAUSE` or
`RESUME`. For instances, this can be either `CANCEL`, `RERUN`, or
`MARK_FINISHED`.
)r r2 �status� SetStatusr r# )r r6 rd r' r( r r r �
set_status s ��z!DataPipelineConnection.set_statusc C sN ||d�}|dur
||d< |dur||d< |dur ||d< | j dt�|�d�S )ak
Notifies AWS Data Pipeline that a task is completed and
provides information about the final status. The task runner
calls this action regardless of whether the task was
sucessful. The task runner does not need to call SetTaskStatus
for tasks that are canceled by the web service during a call
to ReportTaskProgress.
:type task_id: string
:param task_id: Identifies the task assigned to the task runner. This
value is set in the TaskObject that is returned by the PollForTask
action.
:type task_status: string
:param task_status: If `FINISHED`, the task successfully completed. If
`FAILED` the task ended unsuccessfully. The `FALSE` value is used
by preconditions.
:type error_id: string
:param error_id: If an error occurred during the task, this value
specifies an id value that represents the error. This value is set
on the physical attempt object. It is used to display error
information to the user. It should not start with string "Service_"
which is reserved by the system.
:type error_message: string
:param error_message: If an error occurred during the task, this value
specifies a text description of the error. This value is set on the
physical attempt object. It is used to display error information to
the user. The web service does not parse this value.
:type error_stack_trace: string
:param error_stack_trace: If an error occurred during the task, this
value specifies the stack trace associated with the error. This
value is set on the physical attempt object. It is used to display
error information to the user. The web service does not parse this
value.
)r\ �
taskStatusN�errorId� errorMessage�errorStackTrace�
SetTaskStatusr r# )r r^ � task_status�error_id�
error_message�error_stack_tracer( r r r �set_task_status! r[ z&DataPipelineConnection.set_task_statusc C rO )a�
Tests the pipeline definition with a set of validation checks
to ensure that it is well formed and can run without error.
:type pipeline_id: string
:param pipeline_id: Identifies the pipeline whose definition is to be
validated.
:type pipeline_objects: list
:param pipeline_objects: A list of objects that define the pipeline
changes to validate against the pipeline.
rP �ValidatePipelineDefinitionr r# rS r r r � validate_pipeline_definitionT s ��z3DataPipelineConnection.validate_pipeline_definitionc
C s� d| j |f | jjdtt|��d�}| jdddi ||d�}| j|d dd�}|�� �d �}t j
� |� |j d
kr@|r>t
�|�S d S t
�|�}|�d d �}| j�|| j�} | |j |j|d ��)
Nz%s.%szapplication/x-amz-json-1.1)z X-Amz-Target�Hostz Content-TypezContent-Length�POST�/)�method�path� auth_pathr( �headers�data�
)�sender�override_num_retrieszutf-8�� �__type)r" )� TargetPrefixr r �str�len�build_base_http_request�_mexe�read�decode�boto�log�debugrd r �loads�get�_faults�
ResponseError�reason)
r r! r" ry � http_request�response�
response_body� json_body�
fault_name�exception_classr r r r$ i s0
���
�
�z#DataPipelineConnection.make_request)N)NN)NNN)&�__name__�
__module__� __qualname__�__doc__�
APIVersionr r � ServiceNamer� r r� r r r
r r r
r� r r r) r/ r1 r8 r< rA rD rF rN rU rZ r_ rc rf rp rr r$ �
__classcell__r r r r r sV '�
%
�(
�4&
�3
�'
�3r ) r� � boto.compatr �boto.connectionr �boto.regioninfor �boto.exceptionr �boto.datapipeliner r r r r r �<module> s