Server Processors

[dkokron]

I used uProf (version 4.0.341) to profile an application running on 720 MPI ranks. The raw profile data is 210GB. I used "AMDuProfCLI translate" to get it down to a 27GB database file (cpu.db) which is still too large to transfer to my local machine. The machine I'm working on does not support GUIs so I need to interrogate the database using the CLI.  Is there a way to generate a text report from the cpu.db using the CLI?

[santosh_zanjurne]

Hi Dan,

There is no direct way of using .db file with CLI command for report generation.  But there are multiple ways in which you can optimize the uProf data collection and report generation overhead.   The first and foremost being reduce the runtime of the application or input data size of the application.  Other points being:

(1) profile all the ranks but translate per host data which will generate smaller DB
(--host option in translate command),
(2) profile a smaller subset of 720 ranks to collect lesser data (1 rank should suffice),
(3) profile only selective custom events if it is EBP profiling,
(4) do not enable CSS if callstack analysis not needed,
(5) avoid IBS profiling, instead try with EBP profiling,
(6) profile for a limited/shorter duration.

[dkokron]

Translating per host does work for some of the hosts.  However, I could not get reports for many hosts because AMDuProfCLI hung.  Here is a stack trace from one of the hung jobs at 62 minutes into.  The runs that worked finished in about 45 minutes.

(gdb) where
#0 0x000014cbe43ded20 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
#1 0x000014cbe3e77cc7 in std::thread::join() () from /usr/lib64/libstdc++.so.6
#2 0x000014cbe5595f0e in std::deque<std::unique_ptr<std::thread, void (*)(std::thread*)>, std::allocator<std::unique_ptr<std::thread, void (*)(std::thread*)> > >::_M_destroy_data_aux(std::_Deque_iterator<std::unique_ptr<std::thread, void (*)(std::thread*)>, std::unique_ptr<std::thread, void (*)(std::thread*)>&, std::unique_ptr<std::thread, void (*)(std::thread*)>*>, std::_Deque_iterator<std::unique_ptr<std::thread, void (*)(std::thread*)>, std::unique_ptr<std::thread, void (*)(std::thread*)>&, std::unique_ptr<std::thread, void (*)(std::thread*)>*>) () from /u/daniel.kokron/play/AMDuProf/AMDuProf_Linux_x64_4.0.341/bin/libAMDProfileCommon.so
#3 0x000014cbe558ea9a in AMDTProfileManager::JoinAllCollectionThreads() () from /u/daniel.kokron/play/AMDuProf/AMDuProf_Linux_x64_4.0.341/bin/libAMDProfileCommon.so
#4 0x000014cbe558eb19 in AMDTProfileManager::WaitForCompletion() () from /u/daniel.kokron/play/AMDuProf/AMDuProf_Linux_x64_4.0.341/bin/libAMDProfileCommon.so
#5 0x0000000000414616 in HandleTranslateCommand(AMDTProcessArgs&) ()
#6 0x000000000041496b in HandleReportCommand(AMDTProcessArgs&) ()
#7 0x000000000040f949 in main ()

[santosh_zanjurne]

Hi Dan,

I have reproduced this hang issue and have filed a bug report. I shall keep you posted on the progress of the same.