AI Discussions

[EnlinJiang]

Hi,

My environment is AMD Ryzen 7 7840U w/ Radeon 780M Graphics 3.30 GHz on Windows 11 64bits

python quicktest.py --ep ipu

 can get result :

I believe the IPU is working fine.

I want to inference a face detector model on Ryzen AI , the following step is what I done:

1. download a public model: ultra-lightweight-face-detection-rfb-320
2. to remove initialize warning message:

python remove_initializer_from_input.py --input ultra-lightweight-face-detection-rfb-320.onnx --output ultra-lightweight-face-detection-rfb-320_optimize.onnx​

• to remove not used node by following code:

model = onnx.load('ultra-lightweight-face-detection-rfb-320_optimized.onnx'')
so = ort.SessionOptions()
so.optimized_model_filepath = '.\\ultra-lightweight-face-detection-rfb-320_optimized2.onnx'
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
session = ort.InferenceSession(model.SerializeToString(), providers=providers,provider_options=provider_options,sess_options=so)​

• load model "ultra-lightweight-face-detection-rfb-320_optimized2.onnx" for inference:

ryzen_path = 'C:\\AMD\\ryzen-ai-sw-1.0\\voe-4.0-win_amd64'
providers = ['VitisAIExecutionProvider']
cache_dir = pathlib.Path(__file__).parent.resolve()
provider_options = [{
  'config_file': os.path.join(ryzen_path, 'vaip_config.json'),
  'cacheDir': str(cache_dir),
  'cacheKey': 'modelcachekey'}]
model = onnx.load('ultra-lightweight-face-detection-rfb-320_optimized2.onnx')
session = ort.InferenceSession(model.SerializeToString(), providers=providers,provider_options=provider_options)​

• the session constructor message is:

providers=VitisAIExecutionProvider
2024-03-23 11:49:12.2275300 [W:onnxruntime:Default, vitisai_provider_factory.cc:48 onnxruntime::VitisAIProviderFactory::CreateProvider] Construting a FlexML EP instance in Vitis AI EP
2024-03-23 11:49:12.2296755 [W:onnxruntime:Default, vitisai_execution_provider.cc:117 onnxruntime::VitisAIExecutionProvider::SetFlexMLEPPtr] Assigning the FlexML EP pointer in Vitis AI EP
2024-03-23 11:49:12.2431745 [W:onnxruntime:Default, vitisai_execution_provider.cc:137 onnxruntime::VitisAIExecutionProvider::GetCapability] Trying FlexML EP GetCapability
2024-03-23 11:49:12.2456771 [W:onnxruntime:Default, flexml_execution_provider.cc:180 onnxruntime::FlexMLExecutionProvider::GetCapability] FlexMLExecutionProvider::GetCapability, C:\amd\voe\binary-modules\ResNet.flexml\flexml_bm.signature can't not be found!
2024-03-23 11:49:12.2482746 [W:onnxruntime:Default, vitisai_execution_provider.cc:153 onnxruntime::VitisAIExecutionProvider::GetCapability] FlexML EP ignoring a non-ResNet50 graph
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240323 11:49:12.250612  6312 vitisai_compile_model.cpp:336] Vitis AI EP Load ONNX Model Success   
I20240323 11:49:12.250612  6312 vitisai_compile_model.cpp:337] Graph Input Node Name/Shape (1)       
I20240323 11:49:12.250612  6312 vitisai_compile_model.cpp:341]   input : [1x3x240x320]
I20240323 11:49:12.250612  6312 vitisai_compile_model.cpp:347] Graph Output Node Name/Shape (2)      
I20240323 11:49:12.250612  6312 vitisai_compile_model.cpp:351]   scores : [1x4420x2]
I20240323 11:49:12.250612  6312 vitisai_compile_model.cpp:351]   boxes : [1x4420x4]
I20240323 11:49:12.251765  6312 vitisai_compile_model.cpp:226] use cache key modelcachekey
[Vitis AI EP] No. of Operators :   CPU   160 
[Vitis AI EP] No. of Subgraphs :   CPU     1​

• the inference result is correct :outputs = session.run(None, {'input': input_tensor})
• but no any IPU utilization , the FPS is same as inference on 
  providers = ['CPUExecutionProvider']

  Thanks

  Enlin Jiang

[Uday_Das]

Hi Enlin Jiang, 

I am unsure which model you are trying, but I don't see you mentioned the Quantization step. Did you (if yes, how) quantize the model and then pick the model for the VitisAI Execution Provider inference session? For quantization you can check Vitis AI ONNX flow, recommended configuration for CNN's on NPU:  https://ryzenai.docs.amd.com/en/latest/vai_quant/vai_q_onnx.html#cnns-on-npu 

[EnlinJiang]

Hi,

I didn't quantize the model, is it necessary step? I suppose FP32 can run in IPU?

Enlin

[Uday_Das · Accepted solution]

Hi @EnlinJiang 

Yes quantization is a necessary step. Please check the documentation 

https://ryzenai.docs.amd.com/en/latest/devflow.html

For Vision model quantization you can take example from "Getting started Resnet" for Quantization

[Uday_Das]

Hi @EnlinJiang ,

Also, the other reply is talking about flexml and all is not correct. It is obvious from the answer someone used AI to generate the answer. Unfortunately, that answer is not pointing the actual issue. Your actual issue is you did not quantize the model. 

Thank you

[DipakJ27]

Understanding the Problem:

• You're trying to run a face detection model (presumably ultra-lightweight-face-detection-rfb-320) on your AMD Ryzen 7 7840U with Radeon 780M graphics using Ryzen AI and the Vitis AI Execution Provider.
• The model loads and infers correctly, but the IPU (Integrated Processing Unit) isn't being utilized.
• This means the inference is happening on the CPU, resulting in no performance improvement over CPU-only execution.

Potential Causes and Solutions:

1) FlexML EP Model Compatibility:

The warning message FlexMLExecutionProvider::GetCapability, C:\amd\voe\binary-modules\ResNet.flexml\flexml_bm.signature can't not be found! suggests a compatibility issue.
The FlexML Execution Provider (EP) might be looking for a specific signature file (ResNet.flexml) associated with ResNet50 models. Your face detection model likely has a different architecture.

Solution:

Explore Ryzen AI documentation or community forums to see if there are pre-built FlexML models compatible with your face detection model's architecture.

If not, consider using a different execution provider that doesn't rely on FlexML, such as the CPUExecutionProvider (which you've already tried) or potentially the DnnlExecutionProvider (if supported by Ryzen AI).

2) IPU Configuration:

Ryzen AI might require specific configuration steps to enable IPU inference by default.

Solution:

Refer to Ryzen AI documentation or community resources to find instructions on enabling IPU inference for your specific hardware and software setup. There might be settings in BIOS or Ryzen AI configuration tools that need to be adjusted.

3) Model Optimization:

While you've removed initializer warnings, consider further model optimization for IPU execution.

Solution:

If Ryzen AI provides tools or documentation for model optimization specifically for the IPU, utilize them to potentially improve performance.

Troubleshooting and Debugging:

• Check Ryzen AI logs or developer tools for more detailed error messages that might provide clues about the root cause.
• If possible, try running a simple, known-compatible model with Ryzen AI to verify its basic functionality and eliminate setup issues as a cause.
• Search online forums or communities for similar problems and solutions related to Ryzen AI and face detection models.

Additional Tips:

1. Make sure you're using the latest Ryzen AI software version with any available updates.
2. If you're still facing issues after trying these suggestions, consider contacting AMD Ryzen AI support for further assistance.

[EnlinJiang]

Hi Dipakjadhav,

thanks for your reply.

for the FlexML EP Model Compatibility issue, it also happen on quicktest.py, is it normal ? ( C:\amd\voe\binary-modules\ResNet.flexml\flexml_bm.signature can't not be found!)BTW, IPU 400 99.50% did occurs in quicktest.py.

(ryzen) C:\AMD\ryzen-ai-sw-1.0\quicktest>python quicktest.py --ep ipu
2024-03-26 16:27:42.8915774 [W:onnxruntime:Default, vitisai_provider_factory.cc:48 onnxruntime::VitisAIProviderFactory::CreateProvider] Construting a FlexML EP instance in Vitis AI EP
2024-03-26 16:27:42.8965006 [W:onnxruntime:Default, vitisai_execution_provider.cc:117 onnxruntime::VitisAIExecutionProvider::SetFlexMLEPPtr] Assigning the FlexML EP pointer in Vitis AI EP
2024-03-26 16:27:42.9720050 [W:onnxruntime:Default, vitisai_execution_provider.cc:137 onnxruntime::VitisAIExecutionProvider::GetCapability] Trying FlexML EP GetCapability
2024-03-26 16:27:42.9761170 [W:onnxruntime:Default, flexml_execution_provider.cc:180 onnxruntime::FlexMLExecutionProvider::GetCapability] FlexMLExecutionProvider::GetCapability, C:\amd\voe\binary-modules\ResNet.flexml\flexml_bm.signature can't not be found!
2024-03-26 16:27:42.9797490 [W:onnxruntime:Default, vitisai_execution_provider.cc:153 onnxruntime::VitisAIExecutionProvider::GetCapability] FlexML EP ignoring a non-ResNet50 graph
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240326 16:27:42.983819 33260 vitisai_compile_model.cpp:336] Vitis AI EP Load ONNX Model Success
I20240326 16:27:42.984239 33260 vitisai_compile_model.cpp:337] Graph Input Node Name/Shape (1)
I20240326 16:27:42.984452 33260 vitisai_compile_model.cpp:341] input : [-1x3x32x32]
I20240326 16:27:42.984452 33260 vitisai_compile_model.cpp:347] Graph Output Node Name/Shape (1)
I20240326 16:27:42.984452 33260 vitisai_compile_model.cpp:351] output : [-1x10]
I20240326 16:27:42.989856 33260 vitisai_compile_model.cpp:226] use cache key quickstart_modelcachekey
[Vitis AI EP] No. of Operators : CPU 2 IPU 400 99.50%
[Vitis AI EP] No. of Subgraphs : CPU 1 IPU 1 Actually running on IPU 1
2024-03-26 16:27:43.0690893 [W:onnxruntime:, session_state.cc:1169 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2024-03-26 16:27:43.0731762 [W:onnxruntime:, session_state.cc:1171 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
Image 0: Actual Label cat, Predicted Label cat
2024-03-26 16:27:45.6689147 [W:onnxruntime:Default, vitisai_execution_provider.cc:74 onnxruntime::VitisAIExecutionProvider::~VitisAIExecutionProvider] Releasing the FlexML EP pointer in Vitis AI EP

for the pre-built FlexML models , I have lots of models that need to ported to Ryzen AI, it's not realistic to explorer document or community forums for each model,is there easy way?

thanks for your replay again. an NB OEM ask me to inference some models on both AMD and Intel platform. the Intel part had been done , please help me on the AMD part.

Regards

Enlin Jiang