6 Replies Latest reply on Dec 19, 2011 7:58 PM by klin938

    Build HPL 2.0 with Open64 which ACML 4.4.0 to use?

    klin938
      /opt/acml4.4.0/gfortran64 or /opt/acml4.4.0/open64_64?

      Hi guys,

      On a Centos 5.7 machine, I have successfully built HPL 2.0 with the pre-compiled OpenMPI and ACML 4.4.0 for gfortran (http://developer.amd.com/documentation/articles/pages/hpchighperformancelinpack.aspx).

      Later I moved forward and tried to do the same thing with Open64 4.2.5.2. I compiled OpenMPI 1.4.4 from source with Open64, thus both mpicc and mpif90 use Open64 linker. Then I compiled the HPL again with the following key configurations:

      Qucikly the compilation failed with some errors look like:

       

      /opt/acml4.4.0/gfortran64/lib/libacml.a(dgemv.o): In function `dgemv_':

      dgemv.F:(.text+0x2b0): undefined reference to `_gfortran_string_index'

      /opt/acml4.4.0/gfortran64/lib/libacml.a(xerbla.o): In function `xerbla_':

      xerbla.f:(.text+0x76): undefined reference to `_gfortran_compare_string'

      Although I fixed this by changing LAdir to use /opt/acml4.4.0/open64_64, compilation successed, I am not sure why it fixed the issue.
      I am appriciated if anyone can explain the reason.
      Regards,
      Derrick

       

      # A Open64 version of MPI MPdir = /opt/openmpi/1.4-open64 MPinc = -I$(MPdir)/include MPlib = # LAdir = /opt/acml4.4.0/gfortran64 <-- changed to /opt/acml4.4.0/open64_64 fixed the errors. LAinc = -I$(LAdir)/include LAlib = $(LAdir)/lib/libacml.a CC = /opt/openmpi/1.4-open64/bin/mpicc CCNOOPT = $(HPL_DEFS) CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall # LINKER = /opt/openmpi/1.4-open64/bin/mpif90 LINKFLAGS = $(CCFLAGS)

        • Build HPL 2.0 with Open64 which ACML 4.4.0 to use?
          chipf

          The gfortran version of ACML has dependencies on Gfortran runtime libraries.  You specified the gfortran ACML in the offending build.

          Your LINKER macro is using mpif90, which will will be using the Open64 fortran front end for the linker step.  But the open64 link command will not include the required gfortran runtimes needed by the gfortran ACML.

          When you changed the ACML version to open64, you removed any gfortran dependencies.

          Basically, you need to use the same ACML as the Fortran linker you are using.

            • Build HPL 2.0 with Open64 which ACML 4.4.0 to use?
              klin938

              Thanks chipf,

              If I understand correctly, since my mpi90 uses Open64, I am correct to provide Open64's ACML?

              In an earlier thread http://forums.amd.com/forum/messageview.cfm?catid=373&threadid=154868

              The user did the similar thing (except he used gfortran ACML 5.0.0), the helper suggested him to try linking libgfortran, I am wondering if he should use Open64's ACML as well?

              Cheers,

              Derrick

                • Build HPL 2.0 with Open64 which ACML 4.4.0 to use?
                  chipf

                  The appnote in the other thread uses an OpenMPI that was compiled with gcc/gfortran.  So that mpif90 command is using gfortran.  That example needs to use the gfortran ACML.

                  In your case, your mpi path implies you used the open64 compiler to build OpenMPI.  For your case, the mpif90 command is using openf90 as the compiler.  So you would need to use the open64 ACML.

                    • Build HPL 2.0 with Open64 which ACML 4.4.0 to use?
                      klin938

                       

                      Originally posted by: chipf The appnote in the other thread uses an OpenMPI that was compiled with gcc/gfortran.


                      In the script provided by that user, there is MPdir        = /usr/local/openmpi-1.4.3-open64, so I guess he used his own open64 built openmpi as well.

                      Anyway, now I learn for all gcc/gfortran compiling I should use gfortran ACML. open64 ACML is for open64 compiling.

                      I still have issue with the open64 built HPL 2.0 though. When I tried to run, it gave:

                      ./xhpl: error while loading shared libraries: libacml_mv.so: cannot open shared object file: No such file or directory

                      I guess I may need to add appropriate entry in LD_LIBRARY_PATH etc. But for the gcc built HPL (with gfortran ACML), I don't have such error.

                      Why libacml_mv.so is needed in open64 ACML's case and how can I static link it?

                      Cheers,

                      Derrick

                       

                      LAdir = /opt/acml4.4.0/open64_64 LAinc = -I$(LAdir)/include LAlib = $(LAdir)/lib/libacml.a <-- static link to libacml.a, can I additionally static link to libacml_mv.a as well?

                        • Build HPL 2.0 with Open64 which ACML 4.4.0 to use?
                          chipf

                          The acml_mv dependency comes from libacml, which uses a few of the vector trancendental functions in the Random Number Generators.

                          I'm not sure why gfortran is not requiring it, it should.  If you use ldd on your executable it may reveal that it is being picked up and where it is found.

                          You should be able to statically link libacml_mv.a on the command line as you show.

                          Open64 can also create dependencies on functions in libacml_mv.so when it finds opportunities to vectorize loops that have those functions.  A copy of libacml_mv is included in the Open64 library directories.  Appropriate use of LD_LIBRARY_PATH may be necessary.

                          For ACML 5.0, we supplied private copies of the necessary routines and removed the need  for libacml_mv.    We also removed libacml_mv from the ACML package since libmv is now a separate download.  Open64 will still have it's own dependencies separate from  ACML.

                            • Build HPL 2.0 with Open64 which ACML 4.4.0 to use?
                              klin938

                               

                              Originally posted by: chipf , it should.  If you use ldd on your executable it may reveal that it is being picked up and where it is found.


                              I did the ldd on the executables with gfortran and open64. gfortran doesn't seem to look for libacml_mv at all.

                              I couldn't tell if the gfortran's HPL even looks for ACML library, doesn't it?

                              Also there are few libraries such as libmpi* not found, do they matter? The HPLs run successfully though.

                              Cheers,

                              D

                              pwbcad@pwbclinuxlab:hpl-2.0$ ldd gcc_acml_4.4.0/bin/Linux_ATHLON_FBLAS/xhpl linux-vdso.so.1 => (0x00007fff08dff000) libmpi_f90.so.0 => not found libmpi_f77.so.0 => not found libmpi.so.0 => not found libopen-rte.so.0 => not found libopen-pal.so.0 => not found libdl.so.2 => /lib64/libdl.so.2 (0x00000039afe00000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00000039b1a00000) libutil.so.1 => /lib64/libutil.so.1 (0x00000039bae00000) libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00007f5746fc0000) libm.so.6 => /lib64/libm.so.6 (0x00000039b0600000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000039b4600000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000039b0200000) libc.so.6 => /lib64/libc.so.6 (0x00000039afa00000) /lib64/ld-linux-x86-64.so.2 (0x00000039af600000) pwbcad@pwbclinuxlab:hpl-2.0$ ldd open64_acml_4.4.0/bin/Linux_ATHLON_FBLAS/xhpl linux-vdso.so.1 => (0x00007fff46dff000) libmpi_f90.so.0 => not found libmpi_f77.so.0 => not found libmpi.so.0 => not found libopen-rte.so.0 => not found libopen-pal.so.0 => not found libdl.so.2 => /lib64/libdl.so.2 (0x00000039afe00000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00000039b1a00000) libutil.so.1 => /lib64/libutil.so.1 (0x00000039bae00000) libacml_mv.so => /home/pwbcad/opt/x86_open64-4.2.5.2//lib/gcc-lib/x86_64-open64-linux/4.2.5.2/libacml_mv.so (0x00007fc1a7d2d000) libmv.so.1 => /home/pwbcad/opt/x86_open64-4.2.5.2//lib/gcc-lib/x86_64-open64-linux/4.2.5.2/libmv.so.1 (0x00007fc1a7c24000) libm.so.6 => /lib64/libm.so.6 (0x00000039b0600000) libfortran.so => /home/pwbcad/opt/x86_open64-4.2.5.2//lib/gcc-lib/x86_64-open64-linux/4.2.5.2/libfortran.so (0x00007fc1a79b2000) libffio.so => /home/pwbcad/opt/x86_open64-4.2.5.2//lib/gcc-lib/x86_64-open64-linux/4.2.5.2/libffio.so (0x00007fc1a7826000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000039b0200000) libc.so.6 => /lib64/libc.so.6 (0x00000039afa00000) /lib64/ld-linux-x86-64.so.2 (0x00000039af600000)