cancel
Showing results for 
Search instead for 
Did you mean: 

Server Gurus Discussions

Adept I
Adept I

AMDuProf crashes at report generation

I successfully profiled MPI application (8 MPI ranks) with  AMDuProf_Linux_x64_3.3.462

> srun -n 8 AMDuProfCLI collect --config assess --mpi --output-dir amdprof_out_jusuf bt-mz.C.8

But summarization crashed

> AMDuProfCLI-bin report --detail --verbose 2 -i amdprof_out_jusuf/*

> Translation started ...
> [TRANSLATION PROGRESS] 100% Translation done...
> Translation done...
> Report generation started ...
> Generating report file...
>
> terminate called after throwing an instance of 'std::system_error'
> what(): Resource deadlock avoided
> Aborted

Do you know how to solve this issue?

0 Kudos
15 Replies
Esteemed Contributor III

Re: AMDuProf crashes at report generation

Users using uProf needs to go here : https://community.amd.com/t5/newcomers-start-here/bd-p/newcomer-forum to get access to AMD Server Gurus where this specific program/software is moderated: https://community.amd.com/t5/server-gurus/ct-p/amd-server-gurus

 

Staff
Staff

Re: AMDuProf crashes at report generation

Yes, for AMD uProf related support,  AMD Server Gurus community is the best place to post any query/issue.  I'm moving this post there.

Just to clarify one point, AMD Server Gurus is not part of Devgurus community and it is independently moderated. 

0 Kudos
Adept I
Adept I

Re: AMDuProf crashes at report generation


@dipak wrote:

Yes, for AMD uProf related support,  AMD Server Gurus community is the best place to post any query/issue.  I'm moving this post there.

Just to clarify one point, AMD Server Gurus is not part of Devgurus community and it is independently moderated. 


Thanks for clarification. I do not see my original question here. Should I post it again or copy/paste here? Sorry for stupid question Smiley Happy

0 Kudos
Esteemed Contributor III

Re: AMDuProf crashes at report generation

Thank you for that information. I wasn't sure if the OP needed to get whitelisted to go to Server Gurus since once another User mentioned he couldn't get access a while back.

0 Kudos
Staff
Staff

Re: AMDuProf crashes at report generation

Hi @izhukov,

Please use the following command to generate the report for MPI application profiling and let us know if it resolves the issue.

 AMDuProfCLI report --detail --input-dir amdprof_out_jusuf

0 Kudos
Adept I
Adept I

Re: AMDuProf crashes at report generation


@swarup wrote:

Hi @izhukov,

Please use the following command to generate the report for MPI application profiling and let us know if it resolves the issue.

 AMDuProfCLI report --detail --input-dir amdprof_out_jusuf


I ran application again

> srun -n 8 AMDuProfCLI collect --config assess --mpi --output-dir amdprof_out_jusuf bt-mz.C.8

It was successful with following additional output from the profiler

> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc177-Dec-11-2020_08-24-36-30958.caperf
> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc177-Dec-11-2020_08-24-36-30961.caperf
> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc177-Dec-11-2020_08-24-36-30959.caperf
> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc176-Dec-11-2020_08-24-36-30570.caperf
> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc176-Dec-11-2020_08-24-36-30573.caperf
> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc176-Dec-11-2020_08-24-36-30572.caperf
> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc177-Dec-11-2020_08-24-36-30960.caperf
> Profile started ...
> Profile completed ...
> Generated raw file : amdprof_out_jusuf/AMDuProf-jsfc176-Dec-11-2020_08-24-36-30571.caperf

Unfortunately suggested command failed too

> AMDuProfCLI report --detail ./amdprof_out_jusuf
> ./AMDuProf_Linux_x64_3.3.462/bin/AMDuProfCLI
> Report generation started ...
>
> ERROR: Report Generation Failed...

I do measurements on compute node, but report generation is on login node.

 

 
0 Kudos
Staff
Staff

Re: AMDuProf crashes at report generation

Hi @izhukov ,

Looks like "--input-dir" option was missing. Please try with "--input-dir" option. If you are using different node for compute and login, you may need to use "--host" option.

Example-

$ AMDuProfCLI report --detail --host all --input-dir ./amdprof_out_jusuf 

For more details on "--host" option for report generation, please refer "User Guide", section 7.3.2

https://developer.amd.com/wordpress/media/files/AMDuprof_Resources/User_Guide_AMD_uProf_v3.3_GA.pdf

 

0 Kudos
Adept I
Adept I

Re: AMDuProf crashes at report generation


@swarup wrote:

Hi @izhukov ,

Looks like "--input-dir" option was missing. Please try with "--input-dir" option. If you are using different node for compute and login, you may need to use "--host" option.

Example-

$ AMDuProfCLI report --detail --host all --input-dir ./amdprof_out_jusuf 

For more details on "--host" option for report generation, please refer "User Guide", section 7.3.2

https://developer.amd.com/wordpress/media/files/AMDuprof_Resources/User_Guide_AMD_uProf_v3.3_GA.pdf

 


+ srun -n 8 ./AMDuProf_Linux_x64_3.3.462/bin/AMDuProfCLI collect --config assess --mpi --output-dir./amdprof_jureca_1n ./bt-mz.C.8
+ ./AMDuProf_Linux_x64_3.3.462/bin/AMDuProfCLI report --detail --input-dir ./amdprof_jureca_1n
Generating report file...

terminate called after throwing an instance of 'std::system_error'
what(): Resource deadlock avoided

Still the same error on login and on compute nodes. "host " option didn't help. There is a core file which is useless as AMDuProf was not build with source code information.

0 Kudos
Staff
Staff

Re: AMDuProf crashes at report generation

Hi @izhukov,

It would be helpful of you can provide the following information.

  1. Which Linux based OS is used here? Also the share the version details.
  2. Which compiler and the version is used here? 
  3. We are also trying to create a local setup to generate the issue. We assume you are using NPB BT-MZ application. Are you using the 3.4.1-MZ version? Any specific compiler flags/options used?
0 Kudos