o e$@sddlZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZdd lmZmZdd lmZmZmZmZmZded jd ddZeZGd ddZGddde ZdS)N) MessageToJson) service_pb2service_pb2_grpc)InferenceServerClientBase)Request) InferResult) _InferStream_RequestIterator)_get_inference_request_grpc_compression_typeget_error_grpc raise_errorraise_error_grpcic@s"eZdZdZedddfddZdS)KeepAliveOptionsaA KeepAliveOptions object is used to encapsulate GRPC KeepAlive related parameters for initiating an InferenceServerclient object. See the https://github.com/grpc/grpc/blob/master/doc/keepalive.md documentation for more information. Parameters ---------- keepalive_time_ms: int The period (in milliseconds) after which a keepalive ping is sent on the transport. Default is INT32_MAX. keepalive_timeout_ms: int The period (in milliseconds) the sender of the keepalive ping waits for an acknowledgement. If it does not receive an acknowledgment within this time, it will close the connection. Default is 20000 (20 seconds). keepalive_permit_without_calls: bool Allows keepalive pings to be sent even if there are no calls in flight. Default is False. http2_max_pings_without_data: int The maximum number of pings that can be sent when there is no data/header frame to be sent. gRPC Core will not continue sending pings if we run over the limit. Setting it to 0 allows sending pings without such a restriction. Default is 2. i NFrcCs||_||_||_||_dSN)keepalive_time_mskeepalive_timeout_mskeepalive_permit_without_callshttp2_max_pings_without_data)selfrrrrrID:\Projects\ConvertPro\env\Lib\site-packages\tritonclient/grpc/_client.py__init__Ws zKeepAliveOptions.__init__N)__name__ __module__ __qualname____doc__ INT32_MAXrrrrrr8s rcseZdZdZ        dDfdd ZddZdd Zd d Zd d ZddZ dEddZ dEddZ dFddZ dGddZ dHddZ dHddZdGddZdIdd ZdGd!d"Z dJd#d$Zdiddfd%d&ZdKd'd(ZdGd)d*ZdGd+d,Z dHd-d.Z dLd0d1ZdFd2d3Z dHd4d5Z dEd6d7ZdFd8d9Z    /   /     dMd:d;Z    /   /     dMdd?Z d@dAZ!    /    /  dNdBdCZ"Z#S)OInferenceServerClienta An InferenceServerClient object is used to perform any kind of communication with the InferenceServer using gRPC protocol. Most of the methods are thread-safe except start_stream, stop_stream and async_stream_infer. Accessing a client stream with different threads will cause undefined behavior. Parameters ---------- url : str The inference server URL, e.g. 'localhost:8001'. verbose : bool If True generate verbose output. Default value is False. ssl : bool If True use SSL encrypted secure channel. Default is False. root_certificates : str File holding the PEM-encoded root certificates as a byte string, or None to retrieve them from a default location chosen by gRPC runtime. The option is ignored if `ssl` is False. Default is None. private_key : str File holding the PEM-encoded private key as a byte string, or None if no private key should be used. The option is ignored if `ssl` is False. Default is None. certificate_chain : str File holding PEM-encoded certificate chain as a byte string to use or None if no certificate chain should be used. The option is ignored if `ssl` is False. Default is None. creds: grpc.ChannelCredentials A grpc.ChannelCredentials object to use for the connection. The ssl, root_certificates, private_key and certificate_chain options will be ignored when using this option. Default is None. keepalive_options: KeepAliveOptions Object encapsulating various GRPC KeepAlive options. See the class definition for more information. Default is None. channel_args: List[Tuple] List of Tuple pairs ("key", value) to be passed directly to the GRPC channel as the channel_arguments. If this argument is provided, it is expected the channel arguments are correct and complete, and the keepalive_options parameter will be ignored since the corresponding keepalive channel arguments can be set directly in this parameter. See https://grpc.github.io/grpc/python/glossary.html#term-channel_arguments for more details. Default is None. Raises ------ Exception If unable to create a client. FNc szt| dur | } n|st}dtfdtfd|jfd|jfd|jfd|jfg} |r5tj ||| d|_ ny|rd} } } |durZt |d }| } Wdn1sUwY|durwt |d }| } Wdn1srwY|durt |d }| } Wdn1swYtj | | | d }tj ||| d|_ ntj|| d|_ t|j |_||_d|_dS) Nzgrpc.max_send_message_lengthzgrpc.max_receive_message_lengthzgrpc.keepalive_time_mszgrpc.keepalive_timeout_msz#grpc.keepalive_permit_without_callsz!grpc.http2.max_pings_without_data)optionsrb)root_certificates private_keycertificate_chain)superrrMAX_GRPC_MESSAGE_SIZErrrrgrpcZsecure_channel_channelopenreadZssl_channel_credentialsZinsecure_channelrZGRPCInferenceServiceStub _client_stub_verbose_stream)rurlverbosesslr%r&r'credsZkeepalive_optionsZ channel_argsZ channel_optZrc_bytesZpk_bytesZcc_bytesZrc_fsZpk_fsZcc_fs __class__rrrsT         zInferenceServerClient.__init__cCs2t|}|||jdur|j}|Sd}|S)Nr)rZ _call_pluginheadersitems)rr7requestZrequest_metadatarrr _get_metadatas z#InferenceServerClient._get_metadatacCs|Srrrrrr __enter__szInferenceServerClient.__enter__cC |dSrclose)rtypevalue tracebackrrr__exit__ zInferenceServerClient.__exit__cCr=rr>r;rrr__del__rDzInferenceServerClient.__del__cCs||jdS)zWClose the client. Any future calls to server will result in an Error. N) stop_streamr+r?r;rrrr?szInferenceServerClient.closec C||}z"t}|jrtd|||jj||d}|jr$t||jWSt j y?}z t |WYd}~dSd}~ww)aContact the inference server and get liveness. Parameters ---------- headers: dict Optional dictionary specifying additional HTTP headers to include in the request. Returns ------- bool True if server is live, False if server is not live. Raises ------ InferenceServerException If unable to get liveness. zis_server_live, metadata {} {}r9metadataN) r:rZServerLiveRequestr/printformatr.Z ServerLiveliver*RpcErrorrrr7rIr9response rpc_errorrrris_server_live z$InferenceServerClient.is_server_livec CrG)aContact the inference server and get readiness. Parameters ---------- headers: dict Optional dictionary specifying additional HTTP headers to include in the request. Returns ------- bool True if server is ready, False if server is not ready. Raises ------ InferenceServerException If unable to get readiness. zis_server_ready, metadata {} {}rHN) r:rZServerReadyRequestr/rJrKr.Z ServerReadyreadyr*rMrrNrrris_server_readyrRz%InferenceServerClient.is_server_readyc Cs||}z/t|tkrtdtj||d}|jr"td|||j j ||d}|jr1t||j WSt j yL}z t|WYd}~dSd}~ww)aContact the inference server and get the readiness of specified model. Parameters ---------- model_name: str The name of the model to check for readiness. model_version: str The version of the model to check for readiness. The default value is an empty string which means then the server will choose a version based on the model and internal policy. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. Returns ------- bool True if the model is ready, False if not ready. Raises ------ InferenceServerException If unable to get model readiness. model version must be a stringnameversionzis_model_ready, metadata {} {}rHN)r:r@strrrZModelReadyRequestr/rJrKr.Z ModelReadyrSr*rMr)r model_name model_versionr7rIr9rOrPrrris_model_ready8s"  z$InferenceServerClient.is_model_readyc C||}z-t}|jrtd|||jj||d}|jr$t||r0t t |ddWS|WSt j yJ}z t |WYd}~dSd}~ww)aContact the inference server and get its metadata. Parameters ---------- headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns server metadata as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or ServerMetadataResponse message holding the metadata. Raises ------ InferenceServerException If unable to get server metadata. z#get_server_metadata, metadata {} {}rHTZpreserving_proto_field_nameN)r:rZServerMetadataRequestr/rJrKr.ZServerMetadatajsonloadsrr*rMrrr7as_jsonrIr9rOrPrrrget_server_metadatabs&  z)InferenceServerClient.get_server_metadatac C||}z:t|tkrtdtj||d}|jr"td|||j j ||d}|jr1t||r=t t |ddWS|WStjyW}z t|WYd}~dSd}~ww)aContact the inference server and get the metadata for specified model. Parameters ---------- model_name: str The name of the model model_version: str The version of the model to get metadata. The default value is an empty string which means then the server will choose a version based on the model and internal policy. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns model metadata as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or ModelMetadataResponse message holding the metadata. Raises ------ InferenceServerException If unable to get model metadata. rVrWz"get_model_metadata, metadata {} {}rHTr_N)r:r@rZrrZModelMetadataRequestr/rJrKr.Z ModelMetadatar`rarr*rMr rr[r\r7rcrIr9rOrPrrrget_model_metadatas. %  z(InferenceServerClient.get_model_metadatac Cre)aContact the inference server and get the configuration for specified model. Parameters ---------- model_name: str The name of the model model_version: str The version of the model to get configuration. The default value is an empty string which means then the server will choose a version based on the model and internal policy. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns configuration as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or ModelConfigResponse message holding the metadata. Raises ------ InferenceServerException If unable to get model configuration. rVrWz get_model_config, metadata {} {}rHTr_N)r:r@rZrrZModelConfigRequestr/rJrKr.Z ModelConfigr`rarr*rMrrfrrrget_model_configs* %  z&InferenceServerClient.get_model_configc Cr^)aYGet the index of model repository contents Parameters ---------- headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns model repository index as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or RepositoryIndexResponse message holding the model repository index. z*get_model_repository_index, metadata {} {}rHTr_N)r:rZRepositoryIndexRequestr/rJrKr.ZRepositoryIndexr`rarr*rMrrbrrrget_model_repository_indexs.  z0InferenceServerClient.get_model_repository_indexc Cs||}zFtj|d}|dur||jd_|jr!td|||dur4|D] \}}||j|_ q)|j j ||d|jrItd|WdSWdSt j yc} z t| WYd} ~ dSd} ~ ww)aRequest the inference server to load or reload specified model. Parameters ---------- model_name : str The name of the model to be loaded. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. config: str Optional JSON representation of a model config provided for the load request, if provided, this config will be used for loading the model. files: dict Optional dictionary specifying file path (with "file:" prefix) in the override model directory to the file content as bytes. The files will form the model directory that the model will be loaded from. If specified, 'config' must be provided to be the model configuration of the override model directory. Raises ------ InferenceServerException If unable to load the model. r[Nconfigz2load_model, metadata {} override files omitted: {}rHzLoaded model '{}')r:rZRepositoryModelLoadRequest parameters string_paramr/rJrKr8Z bytes_paramr.ZRepositoryModelLoadr*rMr) rr[r7rkfilesrIr9pathcontentrPrrr load_model8s,   z InferenceServerClient.load_modelc Cs||}z/tj|d}||jd_|jrtd|||jj ||d|jr2td|WdSWdSt j yL}z t |WYd}~dSd}~ww)aRequest the inference server to unload specified model. Parameters ---------- model_name : str The name of the model to be unloaded. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. unload_dependents : bool Whether the dependents of the model should also be unloaded. Raises ------ InferenceServerException If unable to unload the model. rjunload_dependentszunload_model, metadata {} {}rHzUnloaded model '{}'N) r:rZRepositoryModelUnloadRequestrl bool_paramr/rJrKr.ZRepositoryModelUnloadr*rMr)rr[r7rrrIr9rPrrr unload_modelhs   z"InferenceServerClient.unload_modelc Cre)aGet the inference statistics for the specified model name and version. Parameters ---------- model_name : str The name of the model to get statistics. The default value is an empty string, which means statistics of all models will be returned. model_version: str The version of the model to get inference statistics. The default value is an empty string which means then the server will return the statistics of all available model versions. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns inference statistics as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Raises ------ InferenceServerException If unable to get the model inference statistics. rVrWz(get_inference_statistics, metadata {} {}rHTr_N)r:r@rZrrZModelStatisticsRequestr/rJrKr.ZModelStatisticsr`rarr*rMrrfrrrget_inference_statisticss6 #  z.InferenceServerClient.get_inference_statisticsc Cs||}z\t}|dur|dkr||_|D]\}}|dur'|j|q|j|jt|t r4|n|gq|j rDt d |||j j||d} |j rSt | |r_tt| ddWS| WStjyy} z t| WYd} ~ dSd} ~ ww)aUpdate the trace settings for the specified model name, or global trace settings if model name is not given. Returns the trace settings after the update. Parameters ---------- model_name : str The name of the model to update trace settings. Specifying None or empty string will update the global trace settings. The default value is None. settings: dict The new trace setting values. Only the settings listed will be updated. If a trace setting is listed in the dictionary with a value of 'None', that setting will be cleared. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns trace settings as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or TraceSettingResponse message holding the updated trace settings. Raises ------ InferenceServerException If unable to update the trace settings. NrUz%update_trace_settings, metadata {} {}rHTr_)r:rTraceSettingRequestr[r8settingsrAextend isinstancelistr/rJrKr. TraceSettingr`rarr*rMr) rr[rwr7rcrIr9keyrArOrPrrrupdate_trace_settingss: *    z+InferenceServerClient.update_trace_settingsc Cs||}z8t}|dur|dkr||_|jr td|||jj||d}|jr/t||r;t t |ddWS|WSt j yU}z t|WYd}~dSd}~ww)aGet the trace settings for the specified model name, or global trace settings if model name is not given Parameters ---------- model_name : str The name of the model to get trace settings. Specifying None or empty string will return the global trace settings. The default value is None. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns trace settings as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or TraceSettingResponse message holding the trace settings. Raises ------ InferenceServerException If unable to get the trace settings. NrUz"get_trace_settings, metadata {} {}rHTr_)r:rrvr[r/rJrKr.r{r`rarr*rMr)rr[r7rcrIr9rOrPrrrget_trace_settingss* # z(InferenceServerClient.get_trace_settingsc Cs||}z`t}|D].\}}|dur|j|q|dks$|dkr+||j|_q|dkr6||j|_q||j|_q|jrHt d |||j j ||d}|jrWt ||rct t|ddWS|WStjy}} z t| WYd} ~ dSd} ~ ww) aVUpdate the global log settings. Returns the log settings after the update. Parameters ---------- settings: dict The new log setting values. Only the settings listed will be updated. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns trace settings as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or LogSettingsResponse message holding the updated log settings. Raises ------ InferenceServerException If unable to update the log settings. NZlog_fileZ log_formatZlog_verbose_levelz#update_log_settings, metadata {} {}rHTr_)r:rLogSettingsRequestr8rwrmZ uint32_paramrsr/rJrKr. LogSettingsr`rarr*rMr) rrwr7rcrIr9r|rArOrPrrrupdate_log_settingsFs2   z)InferenceServerClient.update_log_settingsc Cr^)aGet the global log settings. Parameters ---------- headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns log settings as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or LogSettingsResponse message holding the log settings. Raises ------ InferenceServerException If unable to get the log settings. z get_log_settings, metadata {} {}rHTr_N)r:rrr/rJrKr.rr`rarr*rMrrbrrrget_log_settingss"  z&InferenceServerClient.get_log_settingsc C||}z/tj|d}|jrtd|||jj||d}|jr&t||r2t t |ddWS|WSt j yL}z t |WYd}~dSd}~ww)aRequest system shared memory status from the server. Parameters ---------- region_name : str The name of the region to query status. The default value is an empty string, which means that the status of all active system shared memory will be returned. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns system shared memory status as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or SystemSharedMemoryStatusResponse message holding the system shared memory status. Raises ------ InferenceServerException If unable to get the status of specified shared memory. rXz/get_system_shared_memory_status, metadata {} {}rHTr_N)r:rZSystemSharedMemoryStatusRequestr/rJrKr.ZSystemSharedMemoryStatusr`rarr*rMrrZ region_namer7rcrIr9rOrPrrrget_system_shared_memory_statuss. #  z5InferenceServerClient.get_system_shared_memory_statusrc Cs||}z,tj||||d}|jrtd|||jj||d|jr/td|WdSWdStj yI}z t |WYd}~dSd}~ww)apRequest the server to register a system shared memory with the following specification. Parameters ---------- name : str The name of the region to register. key : str The key of the underlying memory object that contains the system shared memory region. byte_size : int The size of the system shared memory region, in bytes. offset : int Offset, in bytes, within the underlying memory object to the start of the system shared memory region. The default value is zero. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. Raises ------ InferenceServerException If unable to register the specified system shared memory. )rXr|offset byte_sizez-register_system_shared_memory, metadata {} {}rHz.Registered system shared memory with name '{}'N) r:rZ!SystemSharedMemoryRegisterRequestr/rJrKr.ZSystemSharedMemoryRegisterr*rMr) rrXr|rrr7rIr9rPrrrregister_system_shared_memorys* z3InferenceServerClient.register_system_shared_memoryc C||}z4tj|d}|jrtd|||jj||d|jr7|dkr0td|WdStdWdSWdStj yQ}z t |WYd}~dSd}~ww)a^Request the server to unregister a system shared memory with the specified name. Parameters ---------- name : str The name of the region to unregister. The default value is empty string which means all the system shared memory regions will be unregistered. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. Raises ------ InferenceServerException If unable to unregister the specified system shared memory region. rz/unregister_system_shared_memory, metadata {} {}rHrUz0Unregistered system shared memory with name '{}'z-Unregistered all system shared memory regionsN) r:rZ#SystemSharedMemoryUnregisterRequestr/rJrKr.ZSystemSharedMemoryUnregisterr*rMrrrXr7rIr9rPrrrunregister_system_shared_memorys.   z5InferenceServerClient.unregister_system_shared_memoryc Cr)aRequest cuda shared memory status from the server. Parameters ---------- region_name : str The name of the region to query status. The default value is an empty string, which means that the status of all active cuda shared memory will be returned. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. as_json : bool If True then returns cuda shared memory status as a json dict, otherwise as a protobuf message. Default value is False. The returned json is generated from the protobuf message using MessageToJson and as a result int64 values are represented as string. It is the caller's responsibility to convert these strings back to int64 values as necessary. Returns ------- dict or protobuf message The JSON dict or CudaSharedMemoryStatusResponse message holding the cuda shared memory status. Raises ------ InferenceServerException If unable to get the status of specified shared memory. rz-get_cuda_shared_memory_status, metadata {} {}rHTr_N)r:rZCudaSharedMemoryStatusRequestr/rJrKr.ZCudaSharedMemoryStatusr`rarr*rMrrrrrget_cuda_shared_memory_status?s. $  z3InferenceServerClient.get_cuda_shared_memory_statusc Cs||}z/tj|t|||d}|jrtd|||jj ||d|jr2td|WdSWdSt j yL}z t |WYd}~dSd}~ww)aRequest the server to register a system shared memory with the following specification. Parameters ---------- name : str The name of the region to register. raw_handle : bytes The raw serialized cudaIPC handle in base64 encoding. device_id : int The GPU device ID on which the cudaIPC handle was created. byte_size : int The size of the cuda shared memory region, in bytes. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. Raises ------ InferenceServerException If unable to register the specified cuda shared memory. )rX raw_handle device_idrz+register_cuda_shared_memory, metadata {} {}rHz,Registered cuda shared memory with name '{}'N) r:rZCudaSharedMemoryRegisterRequestbase64 b64decoder/rJrKr.ZCudaSharedMemoryRegisterr*rMr) rrXrrrr7rIr9rPrrrregister_cuda_shared_memoryzs0 z1InferenceServerClient.register_cuda_shared_memoryc Cr)aXRequest the server to unregister a cuda shared memory with the specified name. Parameters ---------- name : str The name of the region to unregister. The default value is empty string which means all the cuda shared memory regions will be unregistered. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. Raises ------ InferenceServerException If unable to unregister the specified cuda shared memory region. rz-unregister_cuda_shared_memory, metadata {} {}rHrUz.Unregistered cuda shared memory with name '{}'z+Unregistered all cuda shared memory regionsN) r:rZ!CudaSharedMemoryUnregisterRequestr/rJrKr.ZCudaSharedMemoryUnregisterr*rMrrrrrunregister_cuda_shared_memorys*  z3InferenceServerClient.unregister_cuda_shared_memoryc Cs|| }t|tkrtdt||||||||| | |d }|jr)td||z|jj ||| t | d}|jr=t|t |}|WSt j y[}z t|WYd}~dSd}~ww)aKRun synchronous inference using the supplied 'inputs' requesting the outputs specified by 'outputs'. Parameters ---------- model_name: str The name of the model to run inference. inputs : list A list of InferInput objects, each describing data for a input tensor required by the model. model_version : str The version of the model to run inference. The default value is an empty string which means then the server will choose a version based on the model and internal policy. outputs : list A list of InferRequestedOutput objects, each describing how the output data must be returned. If not specified all outputs produced by the model will be returned using default settings. request_id : str Optional identifier for the request. If specified will be returned in the response. Default value is an empty string which means no request_id will be used. sequence_id : int The unique identifier for the sequence being represented by the object. Default value is 0 which means that the request does not belong to a sequence. sequence_start : bool Indicates whether the request being added marks the start of the sequence. Default value is False. This argument is ignored if 'sequence_id' is 0. sequence_end : bool Indicates whether the request being added marks the end of the sequence. Default value is False. This argument is ignored if 'sequence_id' is 0. priority : int Indicates the priority of the request. Priority value zero indicates that the default priority level should be used (i.e. same behavior as not specifying the priority parameter). Lower value priorities indicate higher priority levels. Thus the highest priority level is indicated by setting the parameter to 1, the next highest is 2, etc. If not provided, the server will handle the request using default setting for the model. timeout : int The timeout value for the request, in microseconds. If the request cannot be completed within the time the server can take a model-specific action such as terminating the request. If not provided, the server will handle the request using default setting for the model. This option is only respected by the model that is configured with dynamic batching. See here for more details: https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher client_timeout : float The maximum end-to-end time, in seconds, the request is allowed to take. The client will abort request and raise InferenceServerExeption with message "Deadline Exceeded" when the specified time elapses. The default value is None which means client will wait for the response from the server. headers : dict Optional dictionary specifying additional HTTP headers to include in the request. compression_algorithm : str Optional grpc compression algorithm to be used on client side. Currently supports "deflate", "gzip" and None. By default, no compression is used. parameters : dict Optional custom parameters to be included in the inference request. Returns ------- InferResult The object holding the result of the inference. Raises ------ InferenceServerException If server fails to perform inference. rV r[inputsr\ request_idoutputs sequence_idsequence_start sequence_endprioritytimeoutrlzinfer, metadata {} {}r9rIr compressionN)r:r@rZrr r/rJrKr. ModelInferr r r*rMr)rr[rr\rrrrrrrclient_timeoutr7compression_algorithmrlrIr9rOresultrPrrrinfersB ^  zInferenceServerClient.inferc sfdd}| }t|tkrtdt|||||||| | | |d }jr0td||z.jj j ||| t |d_ j |jr\d}|dkrU|d |}t|Wd SWd Stjyv}z t|WYd }~d Sd }~ww) aRun asynchronous inference using the supplied 'inputs' requesting the outputs specified by 'outputs'. Parameters ---------- model_name: str The name of the model to run inference. inputs : list A list of InferInput objects, each describing data for a input tensor required by the model. callback : function Python function that is invoked once the request is completed. The function must reserve the last two arguments (result, error) to hold InferResult and InferenceServerException objects respectively which will be provided to the function when executing the callback. The ownership of these objects will be given to the user. The 'error' would be None for a successful inference. model_version: str The version of the model to run inference. The default value is an empty string which means then the server will choose a version based on the model and internal policy. outputs : list A list of InferRequestedOutput objects, each describing how the output data must be returned. If not specified all outputs produced by the model will be returned using default settings. request_id : str Optional identifier for the request. If specified will be returned in the response. Default value is an empty string which means no request_id will be used. sequence_id : int The unique identifier for the sequence being represented by the object. Default value is 0 which means that the request does not belong to a sequence. sequence_start: bool Indicates whether the request being added marks the start of the sequence. Default value is False. This argument is ignored if 'sequence_id' is 0. sequence_end: bool Indicates whether the request being added marks the end of the sequence. Default value is False. This argument is ignored if 'sequence_id' is 0. priority : int Indicates the priority of the request. Priority value zero indicates that the default priority level should be used (i.e. same behavior as not specifying the priority parameter). Lower value priorities indicate higher priority levels. Thus the highest priority level is indicated by setting the parameter to 1, the next highest is 2, etc. If not provided, the server will handle the request using default setting for the model. timeout : int The timeout value for the request, in microseconds. If the request cannot be completed within the time the server can take a model-specific action such as terminating the request. If not provided, the server will handle the request using default setting for the model. This option is only respected by the model that is configured with dynamic batching. See here for more details: https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher The maximum end-to-end time, in seconds, the request is allowed to take. The client will abort request and provide error with message "Deadline Exceeded" in the callback when the specified time elapses. The default value is None which means client will wait for the response from the server. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. compression_algorithm : str Optional grpc compression algorithm to be used on client side. Currently supports "deflate", "gzip" and None. By default, no compression is used. parameters : dict Optional custom parameters to be included in the inference request. Raises ------ InferenceServerException If server fails to issue inference. c sjd}}z|}jrt|t|}Wntjy,}z t|}WYd}~nd}~ww||ddS)N)rerror)rr/rJr r*rMr)Z call_futurerrrOrPcallbackrrrwrapped_callbacks z;InferenceServerClient.async_infer..wrapped_callbackrVrzasync_infer, metadata {} {}rz Sent requestrUz '{}'N)r:r@rZrr r/rJrKr.rfuturer Z _call_futureadd_done_callbackr*rMr)rr[rrr\rrrrrrrrr7rrlrrIr9Zverbose_messagerPrrr async_inferSsJa   z!InferenceServerClient.async_inferc Cs|jdur td||}t||j|_z|jjt|j||t|d}|j |WdSt j yE}z t |WYd}~dSd}~ww)a|Starts a grpc bi-directional stream to send streaming inferences. Note: When using stream, user must ensure the InferenceServerClient.close() gets called at exit. Parameters ---------- callback : function Python function that is invoked upon receiving response from the underlying stream. The function must reserve the last two arguments (result, error) to hold InferResult and InferenceServerException objects respectively which will be provided to the function when executing the callback. The ownership of these objects will be given to the user. The 'error' would be None for a successful inference. stream_timeout : float Optional stream timeout (in seconds). The stream will be closed once the specified timeout expires. headers: dict Optional dictionary specifying additional HTTP headers to include in the request. compression_algorithm : str Optional grpc compression algorithm to be used on client side. Currently supports "deflate", "gzip" and None. By default, no compression is used. Raises ------ InferenceServerException If unable to start a stream or a stream was already running for this client. Nzcannot start another stream with one already running. 'InferenceServerClient' supports only a single active stream at a given time.)rIrr) r0rr:r r/r.ZModelStreamInferr r Z _init_handlerr*rMr)rrZstream_timeoutr7rrIZresponse_iteratorrPrrr start_streams$ # z"InferenceServerClient.start_streamcCs|jdur |jd|_dS)z Stops a stream if one available.N)r0r?r;rrrrFs   z!InferenceServerClient.stop_streamc  Cs|jdur tdt|tkrtdt||||||||| | | d } | r*d| jd_|jr4td | |j | |jrFtd |dSdS) aYRuns an asynchronous inference over gRPC bi-directional streaming API. A stream must be established with a call to start_stream() before calling this function. All the results will be provided to the callback function associated with the stream. Parameters ---------- model_name: str The name of the model to run inference. inputs : list A list of InferInput objects, each describing data for a input tensor required by the model. model_version: str The version of the model to run inference. The default value is an empty string which means then the server will choose a version based on the model and internal policy. outputs : list A list of InferRequestedOutput objects, each describing how the output data must be returned. If not specified all outputs produced by the model will be returned using default settings. request_id : str Optional identifier for the request. If specified will be returned in the response. Default value is an empty string which means no request_id will be used. sequence_id : int or str The unique identifier for the sequence being represented by the object. A value of 0 or "" means that the request does not belong to a sequence. Default is 0. sequence_start: bool Indicates whether the request being added marks the start of the sequence. Default value is False. This argument is ignored if 'sequence_id' is 0 or "". sequence_end: bool Indicates whether the request being added marks the end of the sequence. Default value is False. This argument is ignored if 'sequence_id' is 0 or "". enable_empty_final_response: bool Indicates whether "empty" responses should be generated and sent back to the client from the server during streaming inference when they contain the TRITONSERVER_RESPONSE_COMPLETE_FINAL flag. This strictly relates to the case of models/backends that send flags-only responses (use TRITONBACKEND_ResponseFactorySendFlags(TRITONSERVER_RESPONSE_COMPLETE_FINAL) or InferenceResponseSender.send(flags=TRITONSERVER_RESPONSE_COMPLETE_FINAL)) Currently, this only occurs for decoupled models, and can be used to communicate to the client when a request has received its final response from the model. If the backend sends the final flag along with a non-empty response, this arg is not needed. Default value is False. priority : int Indicates the priority of the request. Priority value zero indicates that the default priority level should be used (i.e. same behavior as not specifying the priority parameter). Lower value priorities indicate higher priority levels. Thus the highest priority level is indicated by setting the parameter to 1, the next highest is 2, etc. If not provided, the server will handle the request using default setting for the model. timeout : int The timeout value for the request, in microseconds. If the request cannot be completed within the time the server can take a model-specific action such as terminating the request. If not provided, the server will handle the request using default setting for the model. This does not stop the grpc stream itself and is only respected by the model that is configured with dynamic batching. See here for more details: https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher parameters : dict Optional custom parameters to be included in the inference request. Raises ------ InferenceServerException If server fails to issue inference. Nz?stream not available, use start_stream() to make one available.rVrTZ"triton_enable_empty_final_responsezasync_stream_infer {}z enqueued request {} to stream...) r0rr@rZr rlrsr/rJrKZ_enqueue_request)rr[rr\rrrrrZenable_empty_final_responserrrlr9rrrasync_stream_infer"s6 Y   z(InferenceServerClient.async_stream_infer)FFNNNNNNr)rUN)NF)rUNF)NNN)rUrUNF)NNF)rN) rUNrUrFFrNNNNN) rUNrUrFFFrNN)$rrrr rr:r<rCrEr?rQrTr]rdrgrhrirqrtrur}r~rrrrrrrrrrrrFr __classcell__rrr5rr"ds<@   *1 = : 0 0 ? I 8 :, ; 0+ < 0,   8 r")rstructr*Z rapidjsonr`Zgoogle.protobuf.json_formatrZtritonclient.grpcrrZ_clientr_requestrZ _infer_resultr Z _infer_streamr r _utilsr r rrrStructsizer!r)rr"rrrrs     ,