File: //usr/lib/python3/dist-packages/duplicity/backends/__pycache__/_boto_multi.cpython-310.pyc
o
��`�& � @ s d dl mZ d dlmZ e�� d dlmZ d dlZd dlZd dl Z d dl
Z
d dl Z d dl Z d dl
Z
d dlZd dlmZ d dlmZ d dlmZ d dlT d d lmZ d
d lmZ d
d lmZ d
Ze jdvrtd dl m Z e� de j � nd dl Z G dd� de j �Z G dd� de�Zdd� Z!dS )� )�division)�standard_library)�rangeN)�config)�log)�progress)�*)� FileChunkIO� )� BotoBackend)�get_connectionz2.1.1)�darwin�linux2)�dummyzAMultiprocessing is not supported on %s, will use threads instead.c s( e Zd ZdZ� fdd�Zdd� Z� ZS )�ConsumerThreadz�
A background thread that collects all written bytes from all
the pool workers, and reports it to the progress module.
Wakes up every second to check for termination
c s0 t t| ��� d| _d| _i | _|| _|| _d S )NTF)�superr �__init__�daemon�finishr �queue�total)�selfr r �� __class__� �@/usr/lib/python3/dist-packages/duplicity/backends/_boto_multi.pyr > s
zConsumerThread.__init__c
C s� d}| j s@z| j�|d�}|d | j|d < d}W n! tjy: } zt�t| j�� �| j� d}W Y d }~nd }~ww | j rd S d S )NTr
r F) r r �getr �Empty�report_transfer�sum�valuesr )r �wait�args�er r r �runF s ���zConsumerThread.run)�__name__�
__module__� __qualname__�__doc__r r$ �
__classcell__r r r r r 8 s r c @ s2 e Zd ZdZdd� Zdd� Zdd� Zd d d
�ZdS ) r a�
Backend for Amazon's Simple Storage System, (aka Amazon S3), though
the use of the boto module, (http://code.google.com/p/boto/).
To make use of this backend you must set aws_access_key_id
and aws_secret_access_key in your ~/.boto or /etc/boto.cfg
with your Amazon Web Services key id and secret respectively.
Alternatively you can export the environment variables
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
c C s6 t �| |� zdd l}W n ty � w | �� d S )Nr )�BotoSingleBackendr �boto� ImportError� _setup_pool)r �
parsed_urlr+ r r r r _ s � zBotoBackend.__init__c C sf t j}|s tjdd�}t| dd�r#t�d� | j�� | j� � d | _t�d| � t
j |d�| _d S )NF)�logical�_poolz8A process pool already exists. Destroying previous pool.z;Setting multipart boto backend process pool to %d processes)� processes) r �s3_multipart_max_procs�psutil� cpu_count�getattrr �Debugr0 � terminate�join�multiprocessing�Pool)r �number_of_procsr r r r- g s
zBotoBackend._setup_poolc C s, t �| � t�d� | j�� | j�� d S )Nz Closing pool)r* �_closer r6 r0 r7 r8 )r r r r r< v s
zBotoBackend._closeNc C s� dd l }tj}|tjk rt�dtj|f � tj}tj�|�}||k r&d}n || }|| r2|d7 }t� d||f � | j
j |j |tj
d�}d } tjr\t�� }
|
�� } t| |�} | �� g } t|�D ]#}
|�| j�}| j| j|| j|j||
|tj| g
}| � | j � t |�� qbt� dt | � � | r�z)| d j!tj"d� | d �#� r�| d �$� r�| d= nt� d� | �%� W n tj&�W n tj&y� t� d t | � � | �%� Y nw | s�t� d
� tjr�d | _'| �(� t | �dks�t |�)� �|k r�|�*� t+d ��|�,� S )
Nr z+Minimum chunk size is %d, but %d specified.r
z Uploading %d bytes in %d chunks)� encrypt_keyz2Waiting for the pool to finish processing %s tasks)�timeoutz6Part upload not successful, aborting multipart upload.z^%s tasks did not finish by the specified timeout,aborting multipart upload and resetting pool.z.Done waiting for the pool to finish processingTz!Multipart upload failed. Aborted.)-r+ r �s3_multipart_chunk_size� s3_multipart_minimum_chunk_sizer �Warn�os�path�getsizer6 �bucket�initiate_multipart_upload�key�
s3_use_sser r9 �Manager�Queuer �startr � storage_uri� boto_uri_str�schemer. � bucket_name�id� num_retries�appendr0 � apply_async�multipart_upload_worker�lenr! �s3_multipart_max_timeout�ready�
successfulr- � TimeoutErrorr r8 �
get_all_parts�
cancel_upload�BackendException�complete_upload)r �filenamerG �headersr+ �
chunk_size�bytes�chunks�mpr �manager�consumer�tasks�nrL �paramsr r r �upload| sr
�
�
����
zBotoBackend.upload)N)r% r& r' r( r r- r< ri r r r r r S s r c
s4 ��fdd��� �������� �
f
dd�� � |�S )z�
Worker method for uploading a file chunk to S3 using multipart upload.
Note that the file chunk is read into memory, so it's important to keep
this number reasonably small.
c s< t �� j}t�d|| |f � �d ur ��� | g� d S d S )Nz%s: Uploaded %s/%s bytes)r9 �current_process�namer r6 �put)�uploadedr � worker_name)�offsetr r r �_upload_callback� s
�z1multipart_upload_worker.<locals>._upload_callbackc
s� t �� j}t�d|�d f � z�t��� �}|���}|�� D ]o}|j�kr�t �d�� �d��T}t
�
� }z|j |�d �t dd� d �d� W n t
jyc } z t�|j� W Y d }~nd }~ww t
�
� }t�d j|�d �t dt|| �� d
�� W d � n1 s�w Y nq |�� d }d }~W n9 ty� } z-t�� | r�t�d |�d | d f � � | d �W Y d } ~ S t�d |�d f � | �d } ~ ww t�d
|�d f � d S )Nz%s: Uploading chunk %dr
�r)ro ra � � i )�cb�num_cbz>{name}: Uploaded chunk {chunk} at roughly {speed} bytes/second)rk �chunk�speedz8%s: Upload of chunk %d failed. Retrying %d more times...z*%s: Upload of chunk %d failed. Aborting...z %s: Upload of chunk %d complete)r9 rj rk r r6 r �lookup�list_multipart_uploadsrP r �time�upload_part_from_file�max�socket�gaierrorrA �strerror�format�abs�close� Exception� traceback� print_exc)
rQ rn �connrE rc �fdrK �ex�endr# )
�_uploadrp rO ra r^ � multipart_idro r. rN rL r r r� � s\
����
� ������ z(multipart_upload_worker.<locals>._uploadr )
rN r. rL rO r� r^ ro ra rQ r r ) r� rp rO ra r^ r� ro r. r rN rL r rT � s 'rT )"�
__future__r �futurer �install_aliases�builtinsr rB r3 r r} �sys� threadingrz r� � duplicityr r r �duplicity.errors�duplicity.filechunkior � _boto_singler r* r �BOTO_MIN_VERSION�platformr9 r r6 �Threadr rT r r r r �<module> s6
r