Server Processors

[wmuriel]

I am compiling aocl-scalapack from source code, the compilation finishes without problems but, when I link the library to the program that I need, I always get an additional output

1 LU

1 LU

1 LU

.

.

.

1 LU

And I don't understand why. The compilation of aocl-scalapack was done according to the user guide. This does not happen when I use the prebuild binaries, but I need to compile from source code.

Additionally when I run the test (./scalapack_test.sh) it always pauses on "xsllt".

I hope someone can help me

[Dankomaister]

Hi,

I have the same problem did you solve this?

/Daniel

[Dankomaister]

Hi,

I have the same problem, did you solve this?

/Daniel

[santosh_zanjurne]

Hello Daniel,

What is the AOCC/AOCL version you are using?

Regards,

Santosh

[Dankomaister]

Hi Santosh,

I'm using version 4.0 downloaded from here:
https://github.com/amd/aocl-scalapack/archive/refs/tags/4.0.tar.gz

/Daniel

[shrjoshi]

Hi Daniel,

Can you please share the below details for scalapack build? 
1.  AOCC version used
2.  Make command 

Also can you confirm if you are facing both reported issues?

[Dankomaister]

Hi,

I used easybuild to compile aocl-scalapack on a Cray machine using gcc 11.2, the easybuild logfile is available here: https://drive.google.com/file/d/1UkZosTYn6kIB48e4xwg2e0Flgj2RXCS8 

The following cmake command was used:

cmake -DCMAKE_INSTALL_PREFIX=/cfs/klemming/projects/snic/ltu-fy/eb/software/ScaLAPACK/4.0-CrayGNU-21.11 -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DENABLE_AOCL_PROGRESS=OFF -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=ON -DLAPACK_INCLUDE_DIRS="${EBROOTLIBFLAME}/include" -DLAPACK_LIBRARIES= -DBLAS_LIBRARIES= /dev/shm/ScaLAPACK/4.0/CrayGNU-21.11/aocl-scalapack-4.0/

I did not run the tests so I cannot confirm the other error, but I definitely have the problem of additional output.

[Dankomaister]

@shrjoshi any way to solve this? I guess there are some precompiler flags that are missing?

[shrjoshi]

I would like to suggest you to use AMD Zen Software Studio  - AMD Zen Software Studio | AMD – It has optimized compiler/libraries and profiler, which works best on AMD systems.

The AOCC compiler and AOCL library – Scalapack is part of it – generated optimized code should help you.

[Dankomaister]

Hi,

No that wont work, I need to build it myself to add it to my EasyBuild environment.
Clearly there is some bug in the code that causes this strange output.

Where you guys able to reproduce this problem?

[shrjoshi]

Hi Daniel,

We have not been able to reproduce the issue so far.
Can you please provide details regarding the example tried?
Also which library function are you using?

[Dankomaister]

Sure,

I compiled BLIS, libFLAME and ScaLAPACK version 4.0 with gcc 11.2 and used these to compile VASP 6.3.2. After running VASP I get these strange outputs which overwhelms the standard out.

[Dankomaister]

Sure I compiled BLIS, libFLAME and ScaLAPACK version 4.0 using gcc 11.2 and used these to build VASP version 6.3.2. When I then run VASP I get many strange outputs to the standard out, not sure exactly which library function VASP uses but I guess its many of them since it doesn't only output "1 LU"

[shrjoshi]

Hello Daniel,

We will try to reproduce the error with VASP and the configuration provided by you.
Will keep you updated on the progress.

[shrjoshi]

Hi Daniel,

Checked for VASP-6.3.2 with GCC-11.2.0 and scalapack(4.0) built from source.
Built below examples to see if additional ouput occurs :
langevin_nve
andersen_nve_constrain_fixed_RPR
Tl_z
Was still not able to reproduce the issue.

Can you please provide the below details?
- Example tried in VASP (Reference link if any)
- MPI used and its version.
- cat /etc/*release
- uname -a
- lscpu

[Dankomaister]

Hi,

I see this strange output on every calculation I run.
For example, the testsuite that comes with VASP see here: https://drive.google.com/file/d/1Pbq0dU_D8VpujF79XhtgP2on6teo7stw

Let me give some more information on the system I’m using.
I built the AMD version of BLIS, libFLAME and ScaLAPACK version 4.0 on the Dardel supercomputer (https://www.pdc.kth.se/hpc-services/computing-systems/dardel-1.1043529)

> cat /etc/*release
NAME="SLES"
VERSION="15-SP1"
VERSION_ID="15.1"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP1"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp1"

> uname -a
Linux uan01 4.12.14-197.78_9.1.60-cray_shasta_c #1 SMP Wed Jul 28 20:06:44 UTC 2021 (9b81641) x86_64 x86_64 x86_64 GNU/Linux

> lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 256
On-line CPU(s) list: 0-255
Thread(s) per core: 2
Core(s) per socket: 64
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7742 64-Core Processor
Stepping: 0
CPU MHz: 2250.000
CPU max MHz: 2250.0000
CPU min MHz: 1500.0000
BogoMIPS: 4491.50
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 16384K
NUMA node0 CPU(s): 0-63,128-191
NUMA node1 CPU(s): 64-127,192-255

Using the CrayGNU/21.11 easybuild toolchain which uses the following cray modules:
gcc/11.2.0
craype-x86-rome
libfabric/1.11.0.4.67
atp/3.14.7
cray-pmi/6.0.15
CrayGNU/21.11
craype/2.7.12
PrgEnv-gnu/8.2.0
craype-network-ofi
xpmem/2.2.40-7.0.1.0_2.4__g1d7a24d.shasta
cray-dsmml/0.2.2
cray-mpich/8.1.11

You can find the easyconfig files that I used to build BLIS, libFLAME and ScaLAPACK here: https://drive.google.com/drive/folders/1npMm2FaizjzmaRf3Os8tnIU9eMofGPBl

Hope this helps to reproduce this strange error.

[shrjoshi]

Hi Daniel,

I built Scalapack from source referring the cmake command shared previously by you.
My cmake command is as below :
cmake .. -DCMAKE_INSTALL_PREFIX=/home/Scalapack_issue/aocl-scalapack-4.0/build/install -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DENABLE_AOCL_PROGRESS=OFF -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=ON -DLAPACK_LIBRARIES=/home/Scalapack_issue/blis_flame_scripts/amd_flame_4.0_GCC_11.2.0/lib/libflame.a -DBLAS_LIBRARIES=/home/Scalapack_issue/blis_flame_scripts/install/amd_blis_4.0_GCC_11.2.0/lib/libblis.a

But this cmake command exited with error :
make[2]: *** No rule to make target 'lib/liblapack.a', needed by 'lib/libscalapack.so'.  Stop.
and CMakeError.log file shows the below error:
"bli_gemv_zen_int_4.c:(.text+0x2089): undefined reference to `GOMP_parallel'
collect2: error: ld returned 1 exit status"

Can you please confirm if you have used "-fopenmp" flag in flame and scalapack library build?
If not used, the scalapack build should fail. So we need to confirm if the proper library is getting linked to vasp.
Can you test  by adding openmp flag to your scalapack build? You can refer the below command :

cmake .. -DCMAKE_INSTALL_PREFIX=/home/Scalapack_issue/aocl-scalapack-4.0/build/install -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DENABLE_AOCL_PROGRESS=OFF -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=ON -DLAPACK_LIBRARIES=/home/Scalapack_issue/blis_flame_scripts/amd_flame_4.0_GCC_11.2.0/lib/libflame.a -DBLAS_LIBRARIES=/home/Scalapack_issue/blis_flame_scripts/install/amd_blis_4.0_GCC_11.2.0/lib/libblis.a -DCMAKE_C_FLAGS="-g -fopenmp" -DCMAKE_Fortran_FLAGS="-g -fopenmp"

[Dankomaister]

Hi!

No I did not use OpenMP when compiling, make sure that you compiled blis and flame without openmp threading.

Here you can see the configure/cmake outputs from my builds. As you can see threading is disabled for all three.

BLIS

configure: detected Linux kernel version 4.12.14-197.78_9.1.60-cray_shasta_c.
configure: python interpeter search list is: python python3 python2.
configure: using 'python' python interpreter.
configure: found python version 3.10.9 (maj: 3, min: 10, rev: 9).
configure: python 3.10.9 appears to be supported.
configure: C compiler search list is: gcc clang cc.
configure: using 'cc' C compiler.
configure: C++ compiler search list is: g++ clang++ c++.
configure: using 'CC' C++ compiler (for sandbox only).
configure: found gcc version 11.2.0 (maj: 11, min: 2, rev: 0).
configure: checking for blacklisted configurations due to cc 11.2.0.
configure: checking cc 11.2.0 against known consequential version ranges.
configure: found assembler ('as') version 2.35.1 (maj: 2, min: 35, rev: 1).
configure: checking for blacklisted configurations due to as 2.35.1.
configure: reading configuration registry...done.
configure: determining default version string.
configure: starting configuration of BLIS AOCL-BLIS 4.0.0 Build 20230201.
configure: configuring with official version string.
configure: found shared library .so version '4.0.0'.
configure:   .so major version: 4
configure:   .so minor.build version: 0.0
configure: manual configuration requested; configuring with 'zen2'.
configure: checking configuration against contents of 'config_registry'.
configure: configuration 'zen2' is registered.
configure: 'zen2' is defined as having the following sub-configurations:
configure:    zen2
configure: which collectively require the following kernels:
configure:    zen2 zen haswell
configure: checking sub-configurations:
configure:   'zen2' is registered...and exists.
configure: checking sub-configurations' requisite kernels:
configure:   'zen2' kernels...exist.
configure:   'zen' kernels...exist.
configure:   'haswell' kernels...exist.
configure: detected --prefix='/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11'.
configure: no install exec_prefix option given; defaulting to PREFIX.
configure: no install libdir option given; defaulting to EXECPREFIX/lib.
configure: no install includedir option given; defaulting to PREFIX/include.
configure: no install sharedir option given; defaulting to PREFIX/share.
configure: final installation directories:
configure:   prefix:      /cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11
configure:   exec_prefix: ${prefix}
configure:   libdir:      ${exec_prefix}/lib
configure:   includedir:  ${prefix}/include
configure:   sharedir:    ${prefix}/share
configure: NOTE: the variables above can be overridden when running make.
configure: detected preset CFLAGS; prepending:
configure:   -O2 -fno-math-errno -craype-verbose
configure: detected preset LDFLAGS; prepending:
configure:   -L/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib
configure: debug symbols disabled.
configure: disabling verbose make output. (enable with 'make V=1'.)
configure: disabling ARG_MAX hack.
configure: building BLIS as both static and shared libraries.
configure: exporting only public symbols within shared library.
configure: enabling operating system support.
configure: threading is disabled.
configure: requesting slab threading in jr and ir loops.
configure: internal memory pools for packing blocks are enabled.
configure: internal memory pools for small blocks are enabled.
configure: memory tracing output is disabled.
configure: libmemkind not found; disabling.
configure: compiler appears to support #pragma omp simd.
configure: the BLAS compatibility layer is enabled.
configure: the CBLAS compatibility layer is enabled.
configure: mixed datatype support is enabled.
configure: mixed datatype optimizations requiring extra memory are enabled.
configure: small matrix handling is enabled.
configure: trsm diagonal element pre-inversion is enabled.
configure: dynamic threading is disabled as multithreading is disabled
configure: the BLIS API integer size is automatically determined.
configure: the BLAS/CBLAS API integer size is 32-bit.
configure: configuring with no addons.
configure: configuring for conventional gemm implementation.
configure: configuring complex return type as "gnu".
configure: creating ./config.mk from ./build/config.mk.in
configure: creating ./bli_config.h from ./build/bli_config.h.in
configure: creating ./bli_addon.h from ./build/bli_addon.h.in
configure: creating ./obj/zen2
configure: creating ./obj/zen2/config/zen2
configure: creating ./obj/zen2/kernels/zen2
configure: creating ./obj/zen2/kernels/zen
configure: creating ./obj/zen2/kernels/haswell
configure: creating ./obj/zen2/ref_kernels/zen2
configure: creating ./obj/zen2/aocl_dtl
configure: creating ./obj/zen2/frame
configure: creating ./obj/zen2/blastest
configure: creating ./obj/zen2/testsuite
configure: creating ./lib/zen2
configure: creating ./include/zen2
configure: mirroring ./config/zen2 to ./obj/zen2/config/zen2
configure: mirroring ./kernels/zen2 to ./obj/zen2/kernels/zen2
configure: mirroring ./kernels/zen to ./obj/zen2/kernels/zen
configure: mirroring ./kernels/haswell to ./obj/zen2/kernels/haswell
configure: mirroring ./ref_kernels to ./obj/zen2/ref_kernels
configure: mirroring ./ref_kernels to ./obj/zen2/ref_kernels/zen2
configure: mirroring ./frame to ./obj/zen2/frame
configure: mirroring ./aocl_dtl to ./obj/zen2/aocl_dtl
configure: creating makefile fragments in ./obj/zen2/config/zen2
configure: creating makefile fragments in ./obj/zen2/kernels/zen2
configure: creating makefile fragments in ./obj/zen2/kernels/zen
configure: creating makefile fragments in ./obj/zen2/kernels/haswell
configure: creating makefile fragments in ./obj/zen2/ref_kernels
configure: creating makefile fragments in ./obj/zen2/aocl_dtl
configure: creating makefile fragments in ./obj/zen2/frame
configure: configured to build within top-level directory of source distribution.

libFLAME

checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for GNU make... make
checking for GNU bash... bash
checking whether user requested a specific C compiler... no
configure: CC environment variable is set to cc, which will override --with-cc option and default search list for C compiler.
checking for x86_64-pc-linux-gnu-gcc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether cc accepts -g... yes
checking for cc option to accept ISO C89... none needed
configure: setting the CC_VENDOR environment variable to gcc.
checking whether user requested a specific library archiver... no
checking for ar... ar
checking whether user requested a specific library archive indexer... no
checking for ranlib... ranlib
checking for a BSD-compatible install... /usr/bin/install -c
checking for find... find
checking for xargs... xargs
checking whether ln -s works... yes
checking whether user requested auto-detection of Fortran linker flags... yes
checking whether user requested auto-detection of Fortran name-mangling... yes
checking for x86_64-pc-linux-gnu-g77... ftn
checking whether we are using the GNU Fortran 77 compiler... yes
checking whether ftn accepts -g... yes
checking how to get verbose linking output from ftn... -v
checking for Fortran 77 libraries of ftn...  -L/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/lib64 -L/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/lib -L/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib -L/opt/cray/pe/mpich/8.1.11/ofi/gnu/9.1/lib -L/opt/cray/pe/dsmml/0.2.2/dsmml//lib -L/opt/cray/pe/atp/3.14.7/lib -L/opt/cray/xpmem/2.2.40-7.0.1.0_2.4__g1d7a24d.shasta/lib64 -L/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/lib/../lib64 -L/opt/cray/pe/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0 -L/opt/cray/pe/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/cray/pe/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/../../.. -lblis -lAtpSigHandler -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml -lxpmem -lgfortran -lquadmath -lpthread -lm
checking for dummy main to link with Fortran 77 libraries... none
checking for Fortran 77 name-mangling scheme... lower case, underscore, no extra underscore
checking how to run the C preprocessor... cc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking ia64intrin.h usability... no
checking ia64intrin.h presence... no
checking for ia64intrin.h... no
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking signal.h usability... yes
checking signal.h presence... yes
checking for signal.h... yes
checking assert.h usability... yes
checking assert.h presence... yes
checking for assert.h... yes
checking math.h usability... yes
checking math.h presence... yes
checking for math.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking whether time.h and sys/time.h may both be included... yes
checking whether cc understands -c and -o together... yes
checking for size_t... yes
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking for function prototypes... yes
checking whether user requested enabling verbose make output... no
checking whether user requested building a static library... yes
checking whether user requested building a dynamic library... yes
checking whether user requested enabling ILP64... no
checking whether user requested hack to circumvent small ARG_MAX... yes
checking for (guessing) appropriate gcc language flags... -std=c99 -Wall -Wno-unused-function -Wno-parentheses -Wfatal-errors
checking whether user requested enabling non-critical code... yes
checking whether user requested a built-in BLAS implementation... no
checking whether user requested the lapack2flame compatibility layer... yes
checking whether user requested enabling external LAPACK for small subproblems... no
checking whether user requested enabling interfaces to external LAPACK routines... no
checking whether user requested enabling F2C calling format for complex dotc/dotu function... no
checking whether user requested enabling AMD optimized path... yes
checking whether user requested setting libFLAME version... no
checking whether user requested enabling void return for functions that by default return complex type as per LAPACK standard... no
checking whether user requested enabling level-3 BLAS front end control trees... no
checking whether user requested GPU extensions... no
no
no
checking whether user requested support for multithreading... no
checking whether user requested SuperMatrix functionality... yes
checking whether user requested enabling gemmt based optimizations... yes
checking whether user requested use of vector intrinsics... no
checking whether user requested support for memory alignment... no
checking whether user requested enabling additional leading dimension alignment... no
checking whether to enable compiler optimizations... yes
checking for (guessing) appropriate gcc optimization flags... -march=znver2 -mtune=znver2 -O3
checking whether to enable compiler warnings... yes
checking for (guessing) appropriate gcc warning flags... -Wall -Wno-comment
checking whether to enable compiler debugging symbols... no
checking for (guessing) appropriate gcc debug flags... -g0
checking whether to enable compiler profiling symbols... no
checking for (guessing) appropriate gcc profiling flags...
checking whether user requested support for internal error checking... yes
checking user-requested internal error checking level... full
checking whether user requested enabling FLA_malloc()/FLA_free() memory leak counter... no
checking whether user requested enabling code that defines bl1_malloc() in terms of FLA_malloc()... yes
checking whether user requested enabling interfaces to internal/low-level libgoto functionality... no
checking whether user requested using the CBLAS interfaces... no
checking whether user requested specific default blocksize in m dimension... no
checking whether user requested specific default blocksize in k dimension... no
checking whether user requested specific default blocksize in n dimension... no
checking whether user requested a portable FLA_Clock() timer... yes
checking for clock_gettime... yes
checking for gettimeofday... yes
checking whether user requested modifying libflame compilating/linking on Windows... no
checking whether user requested SCC extensions... no
checking whether user requested support for Texas Instruments' DSP... no
checking for error_at_line... yes
checking for working memcmp... yes
checking for sqrt in -lm... yes
checking for fabs in -lm... yes
checking for pow in -lm... yes
configure: creating config/x86_64-pc-linux-gnu
configure: creating obj/x86_64-pc-linux-gnu
configure: creating lib/x86_64-pc-linux-gnu
configure: creating include/x86_64-pc-linux-gnu
configure: creating config.sys_type
configure: creating config.dist_path
configure: determining default version string.
configure: could not find '.git' directory; using unmodified version file.
configure: so major version: 4
configure: so minor.build version: 0
configure: creating ./config.status
config.status: creating build/config.mk
config.status: creating build/post-configure.sh
config.status: creating build/FLA_config.h
config.status: executing config.status commands
config.status: retreiving host string from config.sys_type
config.status: retreiving dist_path from config.dist_path
config.status: moving build/config.mk to config/x86_64-pc-linux-gnu
config.status: moving build/post-configure.sh to config/x86_64-pc-linux-gnu
config.status: moving build/FLA_config.h to config/x86_64-pc-linux-gnu
config.status: mirroring ./src into obj/x86_64-pc-linux-gnu/src
config.status: generating makefile fragments in obj/x86_64-pc-linux-gnu/src
config.status: running post-configure.sh

libflame configuration summary

distribution path............................... : .

library version................................. : 5.2.0

shared object (so) version (major).............. : 4
shared object (so) version (minor.build)........ : 0

build system type............................... : x86_64-pc-linux-gnu
host system type................................ : x86_64-pc-linux-gnu

OS name......................................... : Linux
OS version...................................... : 4.12.14-197.78_9.1.60-cray_shasta_c

Enable verbose make output...................... : no

Enable maximum argument list hack............... : yes

C compiler...................................... : cc
C compiler vendor............................... : gcc
Library archiver................................ : ar
Library archive indexer......................... : ranlib

Enable Windows build (experimental)............. : no

Enable ILP64 data types for libflame............. : no

Create static library........................... : yes
Create shared (dynamically-linked) library...... : yes

Enable non-critical code........................ : yes

Enable built-in BLAS implementation............. : no

Enable lapack2flame............................. : yes
Enable external LAPACK for subproblems.......... : no
Enable external LAPACK interfaces............... : no

Enable multithreading support................... : no

Enable SuperMatrix support...................... : yes

Enable GPU support.............................. : no

Enable SCC support.............................. : no

Enable support for Texas Instruments' DSP....... : no

Enable vector intrinsics........................ : no

Enable memory alignment......................... : no

C compiler language flags....................... : -std=c99 -Wall -Wno-unused-function -Wno-parentheses -Wfatal-errors

Enable compiler optimizations................... : yes
   C compiler optimization flags................ : -march=znver2 -mtune=znver2 -O3
Enable compiler warnings........................ : yes
   C compiler warning flags..................... : -Wall -Wno-comment
Enable compiler debugging symbols............... : no
   C compiler debug flags....................... : -g0
Enable compiler profiling symbols............... : no
   C compiler profiling flags................... :

User-specified CFLAGS (prepended)............... : -O2 -fno-math-errno -craype-verbose -I/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/include/blis

Enable internal error checking.................. : yes
   Internal error checking level................ : full

Enable memory leak counter...................... : no

Enable level-3 BLAS front-end control trees..... : no

Enable BLIS use of FLA_malloc()................. : yes

Enable interfaces to internal libgoto symbols... : no

Enable interfaces to CBLAS...................... : no

Enable user-defined default m blocksize......... : no
Enable user-defined default k blocksize......... : no
Enable user-defined default n blocksize......... : no

Enable portable timer........................... : yes
   Portable timer function...................... : clock_gettime()

Autodetect Fortran linker flags................. : yes
   Fortran linker flags......................... :  -L/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/lib64 -L/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/lib -L/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib -L/opt/cray/pe/mpich/8.1.11/ofi/gnu/9.1/lib -L/opt/cray/pe/dsmml/0.2.2/dsmml//lib -L/opt/cray/pe/atp/3.14.7/lib -L/opt/cray/xpmem/2.2.40-7.0.1.0_2.4__g1d7a24d.shasta/lib64 -L/cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/lib/../lib64 -L/opt/cray/pe/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0 -L/opt/cray/pe/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/cray/pe/gcc/11.2.0/snos/lib/gcc/x86_64-suse-linux/11.2.0/../../.. -lblis -lAtpSigHandler -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml -lxpmem -lgfortran -lquadmath -lpthread -lm
Autodetect Fortran name-mangling................ : yes
   Unmangled name............................... : foobar
   Mangled name................................. : foobar_

Installation directories
   prefix....................................... : /cfs/klemming/projects/snic/ltu-fy/eb/software/libFLAME/4.0-CrayGNU-21.11
   exec_prefix.................................. : ${prefix}

   libraries will be installed in............... : ${exec_prefix}/lib
   header files will be installed in............ : ${prefix}/include

Configuration complete!

NOTE: Autodetection of Fortran linker flags was enabled. The configure
script thinks that the flags listed above are necessary to successfully
link a program to Fortran object code. If your program uses any Fortran
libraries, you will probably need to link with these flags.

You may now run 'make' to build all libflame libraries and then 'make install'
to install the libraries.

ScaLAPACK

-- The C compiler identification is GNU 11.2.0
-- The Fortran compiler identification is GNU 11.2.0
cc1: warning: command-line option '-fno-backtrace' is valid for Fortran but not for C
-- Check for working C compiler: /opt/cray/pe/craype/2.7.12/bin/cc
-- Check for working C compiler: /opt/cray/pe/craype/2.7.12/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.7.12/bin/ftn
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.7.12/bin/ftn  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/cray/pe/craype/2.7.12/bin/ftn supports Fortran 90
-- Checking whether /opt/cray/pe/craype/2.7.12/bin/ftn supports Fortran 90 -- yes
-- Found MPI_C: /opt/cray/pe/craype/2.7.12/bin/cc (found version "3.1")
-- Found MPI_Fortran: /opt/cray/pe/craype/2.7.12/bin/ftn (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found MPI_LIBRARY : TRUE
-- --> MPI C Compiler : /opt/cray/pe/craype/2.7.12/bin/cc
-- --> C Compiler : /opt/cray/pe/craype/2.7.12/bin/cc
-- --> MPI Fortran Compiler : /opt/cray/pe/craype/2.7.12/bin/ftn
-- --> Fortran Compiler : /opt/cray/pe/craype/2.7.12/bin/ftn
-- Reducing RELEASE optimization level to O2
-- =========
-- Compiling and Building BLACS INSTALL Testing to set correct variables
-- Configure in the INSTALL directory successful
-- Build in the BLACS INSTALL directory successful
-- =========
-- Testing FORTRAN_MANGLING
-- CDEFS set to Add_
-- =========
-- ENABLE_SET_LIB_VERSION : OFF
-- --> BLACS supplied by user is NOT WORKING, will use BLACS source code for building aocl-scalapack
-- CHECKING BLAS AND LAPACK LIBRARIES
-- --> LAPACK supplied by user is /cfs/klemming/projects/snic/ltu-fy/eb/software/libFLAME/4.0-CrayGNU-21.11/lib/libflame.so.
-- Looking for Fortran dgesv
-- Looking for Fortran dgesv - found
-- --> LAPACK routine dgesv is found: 1.
-- --> LAPACK supplied by user is WORKING, will use /cfs/klemming/projects/snic/ltu-fy/eb/software/libFLAME/4.0-CrayGNU-21.11/lib/libflame.so.
-- BLAS library: /cfs/klemming/projects/snic/ltu-fy/eb/software/BLIS/4.0-CrayGNU-21.11/lib/libblis.so
-- LAPACK library: /cfs/klemming/projects/snic/ltu-fy/eb/software/libFLAME/4.0-CrayGNU-21.11/lib/libflame.so
-- =========
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_FIND_USE_PACKAGE_REGISTRY
    LAPACK_INCLUDE_DIRS


-- Build files have been written to: /dev/shm/ScaLAPACK/4.0/CrayGNU-21.11/easybuild_obj

[shrjoshi]

Hi @Dankomaister 

We are able to reproduce the issue of additional o/p "1 LU" at our end.
Filed a bug report for the same.
Thank you for writing to us, will keep you updated about the progress.

[Dankomaister]

Great!

Looking forward to a fix.

/Daniel

[shrjoshi]

Hi @Dankomaister 

Further testing shows that the source for the issue of additional o/p of 1 LU is the libflame build method.
AOCL-4.0 libFlame does not support the configure option "--enable-supermatrix". This unsupported option is leading to additional prints. Also it is recommended to build libflame ( and other AOCL libraries) with the config options mentioned in the AOCL user guide to avoid the additional prints.
Can you please verify the  issue by building the libflame without the "--enable-supermatrix" config option.

[shrjoshi]

Hi wmuriel,

Regarding the scalapack_test.sh file execution issue, we were able to reproduce the issue at our end.
We have filed a bug report, will keep you updated on the progress of the same.

[shrjoshi]

Hi @wmuriel ,

The fix for the issue - "scalapack_test.sh file execution hangs at xsllt," will be included in the next release of AOCL (tentatively mid 2023).
Thank you for writing to us.

[shrjoshi]

Hi @wmuriel 

We have filed a bug report for the issue of  scalapack library link producing additional o/p "1 LU".
Thank you for writing to us, will keep you updated about the progress.

[wmuriel]

Hi shrjoshi,

I hope it can be solved soon.

Thank you.

[shrjoshi]

Hi @wmuriel 

Can you please confirm if you are building the libflame with the "--enable-supermatrix" config option?
If so, that could be the reason for additional o/p. AOCL-4.0 libFlame does not support the configure option "--enable-supermatrix". This unsupported option is leading to additional prints. Also it is recommended to build libflame ( and other AOCL libraries) with the config options mentioned in the AOCL user guide to avoid the additional prints.

[wmuriel]

Hi shrjoshi

I did not use the "--enable-supermatrix" option. I want to specify that the additional outputs and the hangs at xsllt only occur when I compile with aocc 4.0, when I compile with aocc 3.2.0 these issues do not occur.

[ayyaz0698]

I am compiling aocl-scalapack from source code, the compilation finishes without problems but, when I link the library to the program that I need, I always get an additional output.

[shrjoshi]

Hi @wmuriel , @ayyaz0698 

Can you please share the below details so as to be able to reproduce the issue at my end :

1. application name and version
2. Compiler versions/flags
3. blis/flame/scalapack version/configuration command
4. os/glibc version
5. lscpu

[shrjoshi]

Hi @wmuriel @ayyaz0698 

AOCL-4.0 works best when the sources built as per the AOCL-user guide build commands.
Can you confirm if all the options tried are in accordance with the user guide?
Also as requested in previous message, configuration command for all the libraries will be helpful.

[wmuriel]

Hello

Below is the configuration I used with both the aocc 4.0 compiler and the aocc 3.2.0 compiler. With the aocc 3.2.0 compiler everything works fine, and with the aocc 4.0 compiler the problems I reported in my original post appear.

../configure --prefix=/softw/libs/aocl/4.0.0-serial-aocc-3.2.0 --enable-threading=no --enable-system \
--enable-blas --enable-cblas --int-size=32 --complex-return=intel --disable-aocl-dynamic \
CC=clang CXX=clang++ FC=flang CFLAGS="-fPIC" zen2

./configure --prefix=/softw/libs/aocl/4.0.0-serial-aocc-3.2.0 --enable-dynamic-build \
--enable-static-build --enable-multithreading=no --enable-max-arg-list-hack \
--enable-vector-intrinsics=sse --enable-blas-ext-gemmt \
--enable-f2c-dotc --enable-void-return-complex --enable-optimizations="-O3 -march=znver2 -mfma" \
--enable-lapack2flame \
--enable-amd-opt --enable-external-lapack-interfaces=no \
F77=flang CC=clang FC=flang CXX=clang++ FLIBS="-lflang" CFLAGS="-fPIC" \
FFLAGS="-fPIC" FCFLAGS="-fPIC" CXXFLAGS="-fPIC"


cmake .. -DCMAKE_INSTALL_PREFIX=/softw/libs/aocl/4.0.0-serial-aocc-3.2.0 \
-DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=OFF -DUSE_F2C=ON -DENABLE_AOCL_PROGRESS=OFF\
-DBLAS_LIBRARIES="/softw/libs/aocl/4.0.0-serial-aocc-3.2.0/lib/libblis.so" \
-DLAPACK_LIBRARIES="/softw/libs/aocl/4.0.0-serial-aocc-3.2.0/lib/libflame.so" \
-DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF \
-DCMAKE_Fortran_FLAGS=" -march=znver2 -fPIC " \
-DCMAKE_C_FLAGS=" -march=znver2 -fPIC "

[shrjoshi]

Hi @wmuriel 

We checked the configure options provided by you with VASP application and could not reproduce the issue at our end.
Can you please refer the configure options of the AOCC section as mentioned in the AOCL-user guide and see if the issue is still reproducible.
Also add CC, CXX , FC flags wherever required.
Below is the sample code for reference:
Blis : Section 4.1.1 (Page - 20)
AOCC :
$ ./configure --enable-cblas --enable-threading=[Mode] --prefix=<your-install-dir> --complexreturn=intel CC=clang CXX=clang++ auto
[Mode] values can be openmp and no. "no" will disable multi-threading.

Flame : Section 5.1.1 (Page - 50)
Using 32-bit Integer (LP64)
$ ./configure --enable-amd-aocc-flags --prefix=<your-install-dir> --enable-multithreading=no
Using 64-bit Integer (ILP64)
$ ./configure --enable-amd-aocc-flags --enable-ilp64 --prefix=<your-install-dir> --enable-multithreading=no

Scalapack : Section 8.1.1 (Page - 74)
$ cmake .. -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLIS library>/libblis.so" -DLAPACK_LIBRARIES="<path to libflame library>/libflame.so" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-D DENABLE_ILP64=ON]

If the issue is still reproducible with the options provided in the user guide, please let us know the application details and version with which scalapack library is linked.