Hello!
I have faced an unexpected behavior in the inference performance while testing classification models targeting the IPU device.
Problem: FPS performance for the INT8-quantized model by vai_q_onnx (with IPU as target) is worse than for the original FP32 model without quantization.
Ryzen series: 7940HS
RyzenAI version: 0.8
Description:
There are 2 models: Resnet50 and Mobilenet_V2 from the Torchvision have been tested in the Ryzen-AI project.
Firstly, pre-trained models were converted to the ONNX format, and then the main development flow from the docs was applied.
The VitisAI-ONNX quantizer was used for INT8 quantization. The inference tests were as with IPU targeting, as with CPU targeting.
To reproduce the issue,
The quantization parameters were as follows:
- For IPU (the recommended config from the docs):
vai_q_onnx.quantize_static(
model_input,
model_output,
calibration_data_reader,
quant_format=QuantFormat.QDQ,
calibrate_method=vai_q_onnx.PowerOfTwoMethod.MinMSE,
activation_type=QuantType.QInt8,
weight_type=QuantType.QInt8,
enable_dpu=True,
extra_options={'ActivationSymmetric': True}
)- For CPU:
vai_q_onnx.quantize_static(
input_model_path,
output_model_path,
data_reader,
calibrate_method=vai_q_onnx.PowerOfTwoMethod.NonOverflow,
quant_format=QuantFormat.QDQ,
activation_type=QuantType.QInt8,
weight_type=QuantType.QInt8,
)
The inference was evaluated on the stratified ImageNet data (1000 images, 1 image per class).
The results are as follows:
The original model in ONNX format performs faster than the quantized versions which should not be the case.
Do you have any suggestions on how to adjust the flow to resolve the issue?
Looking forward to the collaboration. Thank you!
