CasperSecurity
o
p̦isD � @ sb d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlZd dlZd dl Z d dl
Z
d dl m Z m
Z
mZmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlmZmZmZm Z m Z m Z m Z d dl m!Z! d d l"m#Z#m$Z$ d
Z%e%e ksxJ �d Z&e&e%ks�J �d
Z'e(ed �o�e(ed
�Z)dedede*fdd�Z+de,ee-f fdd�Z.de-dedej/fdd�Z0dede-de
fdd�Z1G dd � d �Z2G d d � d e �Z3e-Z4e,ed e3f e,ed e4f B Z5G d!d"� d"e6�Z7G d#d$� d$ej8�Z9d%e:e9 de:e9 fd&d'�Z;G d(d)� d)�Z<d*eddfd+d,�Z=G d-d.� d.ej>�Z?d/e@e-ef dee@e-ef B fd0d1�ZAdeBeCe fd2d3�ZDdS )4� N)�
NamedTuple�NoReturn�Literal�Any)�support)� os_helper)� Namespace)�Regrtest)�runtest� is_failed�
TestResult� Interrupted�Timeout�
ChildError�PROGRESS_MIN_TIME)� setup_tests)�format_duration�
print_warningg >@g �r@�setsid�killpg�result�ns�returnc C s& t | t�rdS |jrt| |�rdS dS )NTF)�
isinstancer
�failfastr )r r � r �2/usr/lib/python3.10/test/libregrtest/runtest_mp.py� must_stop) s
r c C s$ t �| �\}}tdi |��}||fS )Nr )�json�loadsr )� worker_args�ns_dict� test_namer r r r �parse_worker_args1 s r# �testnamec C sx t |�}|| f}t�|�}tjgt�� �d�d�d�d�|�}i }tr'd|d< tj |ftj
tj
dt j dkt
jd�|��S ) Nz-uz-mz
test.regrtestz
--worker-argsT�start_new_session�nt)�stdout�stderr�universal_newlines� close_fds�cwd)�varsr �dumps�sys�
executabler �args_from_interpreter_flags�USE_PROCESS_GROUP�
subprocess�Popen�PIPE�os�namer �SAVEDCWD)r$ r r! r �cmd�kwr r r �run_test_in_subprocess7 s4
�������r: r" c C s< t | � t| |�}t� ttj|td�dd� t�d� d S )N)�clsT��flushr )r r
�printr r- �EncodeTestResultr. �exit)r r" r r r r �run_tests_workerP s
rA c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
S ) �MultiprocessIteratorz8A thread-safe iterator over tests for multiprocess mode.c C s t �� | _|| _d S �N)� threading�Lock�lock�
tests_iter)�selfrG r r r �__init__a s
z MultiprocessIterator.__init__c C s | S rC r �rH r r r �__iter__e s z MultiprocessIterator.__iter__c C sB | j � | jd u r t�t| j�W d � S 1 sw Y d S rC )rF rG �
StopIteration�nextrJ r r r �__next__h s
$�z MultiprocessIterator.__next__c C s2 | j � d | _W d � d S 1 sw Y d S rC )rF rG rJ r r r �stopn s "�zMultiprocessIterator.stopN)�__name__�
__module__� __qualname__�__doc__rI rK rN rO r r r r rB ] s rB c @ s. e Zd ZU eed< eed< eed< eed< dS )�MultiprocessResultr r' r( � error_msgN)rP rQ rR r �__annotations__�strr r r r rT s s
rT FTc @ s e Zd ZdS )�
ExitThreadN)rP rQ rR r r r r rX ~ s rX c s� e Zd Zdeddddf� fdd� Zdefdd �Zd d
d �Zd d d
�Z d!de dedede
fdd�Z dede eeef fdd�Z
dede
fdd�Zd dd�Zd dd �Zd eddfd d �Z� ZS )"�TestWorkerProcess� worker_id�runner�MultiprocessTestRunnerr Nc sZ t � �� || _|j| _|j| _|j| _|j| _|j| _d | _ d | _
d | _ d| _ d| _
d S )NF)�superrI rZ �pending�outputr �worker_timeout�timeout�regrtest�current_test_name�
start_time�_popen�_killed�_stopped)rH rZ r[ �� __class__r r rI � s
zTestWorkerProcess.__init__c C s� d| j � �g}| �� r|�d� n|�d� | j}|r#|�d|� �� | j}|d urBt�� | j }|�d| jj � �dt
|�� �f� dd� |� S ) NzTestWorkerProcess #�running�stoppedztest=zpid=ztime=z<%s>� ) rZ �is_alive�appendrc re �time� monotonicrd �extend�pidr �join)rH �info�test�popen�dtr r r �__repr__� s
�zTestWorkerProcess.__repr__c
C s� | j }|d u r d S | jrd S d| _tr| � d�}n| � }td|� �tjdd� ztr5t�|jt j
� W d S |� � W d S t yE Y d S t
ya } ztd|� d|��� W Y d }~d S d }~ww )NTz process groupzKill ��filer= zFailed to kill z: )re rf r1 r> r. r( r5 r rr �signal�SIGKILL�kill�ProcessLookupError�OSErrorr )rH rv �what�excr r r �_kill� s( "��zTestWorkerProcess._killc C s d| _ | �� d S )NT)rg r� rJ r r r rO � s zTestWorkerProcess.stop� � test_resultr' r( c C s t �� | j |_t||||�S rC )ro rp rd � duration_secrT )rH r� r' r( �err_msgr r r �mp_result_error� s z!TestWorkerProcess.mp_result_errorr" c C s t �� | _|| _zt|| j�}d| _|| _W n d | _� zez\| jr)| � � t
�z|j | j d�\}}|j
}|d us<J �W n% tjyV | jrJt
�| � � d }d }}Y n tyb | jrat
�� w |�� }|�� }|||fW W | �� d | _d | _S | � � � | �� d | _d | _w )NF�ra r� )ro rp rd rc r: r rf re rg r� rX � communicatera �
returncoder2 �TimeoutExpiredr �strip�rstrip�_wait_completed)rH r" rv r' r( �retcoder r r � _run_process� sP
� �z TestWorkerProcess._run_processc
C s� | � |�\}}}|d u r| �t|�||�S d }|dkr d| }n1|�d�\}}}|�� }|s1d}n z tj|td�}W n tyP } z
d| }W Y d }~nd }~ww |d ur_| �t |�|||�S t
||||�S )Nr z Exit code %s�
z Failed to parse worker stdout)� object_hookz Failed to parse worker JSON: %s) r� r� r �
rpartitionr� r r �decode_test_result� Exceptionr rT ) rH r" r� r' r( r� �_r r� r r r �_runtest s*
��
�zTestWorkerProcess._runtestc C s� | j sRz,zt| j�}W n ty Y W d S w | �|�}| j�d|f� t|j| j �r.W d S W n t
y9 Y d S t yL | j�dt �
� f� Y d S w | j rd S d S )NFT)rg rM r^ rL r� r_ �putr r r rX �
BaseException� traceback�
format_exc)rH r" � mp_resultr r r �run" s& �
� ��zTestWorkerProcess.runc
C sv | j }|j�� |j�� z|�t� W d S tjtfy: } zt d| � dt
t�� d|��� W Y d }~d S d }~ww )NzFailed to wait for z completion (timeout=z): ) re r' �closer( �wait� JOIN_TIMEOUTr2 r� r r r )rH rv r� r r r r� 5 s
����z!TestWorkerProcess._wait_completedrd c C sf | � d� | �� s d S t�� | }| j�d| � dt|�� �� |tkr2td| � dt|�� �� d S q)NTg �?z Waiting for z thread for zFailed to join z in ) rs rm ro rp rb �logr r� r )rH rd rw r r r � wait_stoppedD s
��z TestWorkerProcess.wait_stopped�r N)r� r� N)rP rQ rR �intrI rW rx r� rO r rT r� �tupler� r� r� r� �floatr� �
__classcell__r r rh r rY � s,
����
�
;
rY �workersc C sN g }| D ] }|j }|s qt�� |j }|tkr$d|t|�f }|�|� q|S )Nz%s (%s))rc ro rp rd r r rn )r� rj �workerrc rw �textr r r � get_running\ s
�r� c @ sr e Zd Zdeddfdd�Zddd�Zddd �ZdedB fd
d �Zd e ddfd
d�Z
dede fdd�Z ddd�Z
dS )r\ rb r Nc C sj || _ | j j| _|j| _t�� | _t| j j�| _| jj d ur-t
| jj d | jj d �| _ nd | _ d | _ d S )Ng �?i, )
rb r� r �queue�Queuer_ rB �testsr^ ra �minr` r� )rH rb r r r rI j s
�
z MultiprocessTestRunner.__init__c s| � fdd�t d� jjd �D �� _dt� j�� d�}� jjr-|dt� jj�t� j�f 7 }� �|� � jD ]}|� � q5d S )Nc s g | ]}t |� ��qS r )rY )�.0�indexrJ r r �
<listcomp>{ s �z8MultiprocessTestRunner.start_workers.<locals>.<listcomp>� z Run tests in parallel using z child processesz" (timeout: %s, worker timeout: %s))
�ranger �use_mpr� �lenra r r` r� �start)rH �msgr� r rJ r �
start_workersz s
�
��
�z$MultiprocessTestRunner.start_workersc C s6 t �� }| jD ]}|�� q| jD ]}|�|� qd S rC )ro rp r� rO r� )rH rd r� r r r � stop_workers� s
�z#MultiprocessTestRunner.stop_workersc C s� | j jd u}t}tdd� | jD ��rM|rtjtdd� z| jj |d�W S t
j y- Y nw t | j�}|rC| j j
sC| �dd�|� � tdd� | jD ��sz| jj dd�W S t
j y` Y d S w ) Nc s s � | ]}|� � V qd S rC )rm )r� r� r r r � <genexpr>� s � z5MultiprocessTestRunner._get_result.<locals>.<genexpr>T)r@ r� z running: %s�, r )r ra �PROGRESS_UPDATE�anyr� � faulthandler�dump_traceback_later�MAIN_PROCESS_TIMEOUTr_ �getr� �Emptyr� �pgor� rs )rH �use_faulthandlerra rj r r r � _get_result� s* ��
��z"MultiprocessTestRunner._get_resultr� c C s� |j }t|�}|jd ur|d|j 7 }n|jtkr&| jjs&|dt|j� 7 }t| j �}|r:| jjs:|dd�
|� 7 }| j � | j
|� d S )Nz (%s)z -- running: %sr� )r rW rU r� r r r� r r� r� rs rb �display_progress�
test_index)rH r� r r� rj r r r �display_result� s
z%MultiprocessTestRunner.display_result�itemc C s� |d r|d }t d|� �� dS | jd7 _|d }| j�|j� | �|� |jr2t|jdd� |jrB| j j
sBt|jt jdd� t |j| j �rKdS dS )z&Returns True if test runner must stop.r r� z regrtest worker thread failed: Tr< ry F)
r r� rb �accumulate_resultr r� r' r> r( r r� r. r )rH r� r� r� r r r �_process_result� s
z&MultiprocessTestRunner._process_resultc C s� | � � d| _z<z | �� }|d u rn | �|�}|rnq
W n ty- t� d| j_Y nw W | jj d ur9t
� � | j �
� | �� d S | jj d urNt
� � | j �
� | �� w )Nr T)r� r� r� r� �KeyboardInterruptr> rb � interruptedr ra r� �cancel_dump_traceback_laterr^ rO r� )rH r� rO r r r � run_tests� s2
�� ��
�
z MultiprocessTestRunner.run_testsr� )rP rQ rR r rI r� r� � QueueOutputr� rT r� �boolr� r� r r r r r\ i s
r\ rb c C s t | ��� d S rC )r\ r� )rb r r r �run_tests_multiprocess� s r� c s2 e Zd ZdZdedeeef f� fdd� Z� ZS )r? z7Encode a TestResult (sub)class object into a JSON dict.�or c s. t |t�rt|�}|jj|d< |S t� �|�S )N�__test_result__)r r r, ri rP r] �default)rH r� r rh r r r� � s
zEncodeTestResult.default) rP rQ rR rS r �dictrW r� r� r r rh r r? � s &r? �dc C sB d| vr| S | � d�}t� D ]}|j|kr |di | �� S qdS )z7Decode a TestResult (sub)class object from a JSON dict.r� Nr )�pop�get_all_test_result_classesrP )r� �cls_namer; r r r r� � s
��r� c C sT d} t h}t|�| kr(t|�} g }|D ] }|�|�� � q|�|� t|�| ks |S )Nr )r r� rq �__subclasses__�update)�
prev_count�classes�to_addr; r r r r� s
�r� )E� collectionsr� r r5 r� r{ r2 r. rD ro r� �typingr r r r ru r � test.supportr �test.libregrtest.cmdliner �test.libregrtest.mainr �test.libregrtest.runtestr
r r r
r r r �test.libregrtest.setupr �test.libregrtest.utilsr r r� r� r� �hasattrr1 r� r r� rW r# r3 r: rA rB rT �ExcStrr� r� rX �ThreadrY �listr� r\ r� � JSONEncoderr? r� r� �set�typer� r r r r �<module> sT $
[
&