Server Processors

[eshernan]

I'm trying to compile WRF 4.0 with aocc v 2.1.0, but I'm experimenting with several problems trying to compile .  My Current compile.wrf is: 

DESCRIPTION     =       CLANG ($SFC/$SCC) :aocc 2.1
DMPARALLEL      =       1
OMPCPP          =       -D_OPENMP
OMP             =       -fopenmp
OMPCC           =       -fopenmp
SFC             =       flang
SCC             =       clang
CCOMP           =       clang
DM_FC           =       mpif90 -f90=$(SFC)
DM_CC           =       mpicc -cc=$(SCC)
FC              =       time $(DM_FC)
CC              =       $(DM_CC) -DFSEEKO64_OK
LD              =       $(FC)
RWORDSIZE       =       $(NATIVE_RWORDSIZE)
PROMOTION       =       #-fdefault-real-8
ARCH_LOCAL      =       -DNONSTANDARD_SYSTEM_SUBR  -DWRF_USE_CLM $(NETCDF4_IO_OPTS) -DRPC_TYPES=1
CFLAGS_LOCAL    =       -O3 -march=znver2 -mavx2 # -DRSL0_ONLY
LDFLAGS_LOCAL   =
CPLUSPLUSLIB    =
ESMF_LDFLAG     =       $(CPLUSPLUSLIB)
FCOPTIM         =       -O3 -march=znver2 -mavx2
FCREDUCEDOPT    =       $(FCOPTIM)
FCNOOPT         =       -O0
FCDEBUG         =       # -g $(FCNOOPT)  # -fbacktrace -ggdb -fcheck=bounds,do,mem,pointer -ffpe-trap=invalid,zero,overflow
FORMAT_FIXED    =       -ffixed-form
FORMAT_FREE     =       -ffree-form -ffree-line-length-none
FCSUFFIX        =
BYTESWAPIO      =       -fconvert=big-endian -frecord-marker=4
FCBASEOPTS_NO_G =       -w $(FORMAT_FREE) $(BYTESWAPIO)
FCBASEOPTS      =       $(FCBASEOPTS_NO_G) $(FCDEBUG)
MODULE_SRCH_FLAG =
TRADFLAG        =      -traditional-cpp $(NETCDF4_IO_OPTS)
CPP             =      /lib/cpp -P -nostdinc
AR              =      ar
ARFLAGS         =      ru
M4              =      m4 -B 14000
RANLIB          =      ranlib
RLFLAGS         =
CC_TOOLS        =      $(SCC)

Some advices in the flags ? 


Thanks, 

[santosh_zanjurne]

Can post the details of the exact issue you are facing? Try setting FCREDUCEDOPT=01.

Between, using vectorized math library will result in better performance.  Vectorized math library comes with glibc-2.22 and later.  You can use below flags to use the same.  You need to use these flags carefully to avoid compilation error that results when these options are used multiple times.

-mllvm -vector-library=LIBMVEC  -fno-finite-math-only

Regards,

Santosh

[eshernan]

In the compilation process appear this problems several times 

make[3]: [module_mp_SBM_polar_radar.o] Error 1 (ignored)

make[3]: Leaving directory `/global/home/users/ejhernandezb/installer/amd/WRF/phys'

ar: module_mp_SBM_polar_radar.o: No such file or directory 

and 

ld.lld: error: undefined symbol: module_diag_zld_zld_

>>> referenced by start_em.f90:2217

>>>               start_em.o:(start_domain_em_) in archive libwrflib.a

this error about undefined symbol appear several times, 

[eshernan]

I Configure the following env variables, before of compile 

export SHARED=some filesystems. 

export CC=clang

export FC=flang

export CXX=clang++

export MPI_CC=clang

export NETCDF=$SHARED/libs/amd/netcdf

export PNETCDF=$SHARED/libs/pnetcdf

export HDF5=$SHARED/libs/hdf5

export PHDF5=$SHARED/libs/hdf5

export JASPERLIB=$SHARED/libs/jasper/lib

export JASPERINC=$SHARED/libs/jasper/include

export ZLIB=$SHARED/libs/zlib

export MPICH=$SHARED/libs/mpich

export PAPI_DIR=$SHARED/libs/papi

export PATH=$PATH:$NETCDF/bin:$MPICH/bin:$PAPI_DIR/bin

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MPICH/lib:$NETCDF/lib:$HDF5/lib:$ZLIB/lib:$PNETCDF/lib:$PAPI_DIR/lib

export WRFIO_NCD_LARGE_FILE_SUPPORT=1

[eshernan]

The process compiles file until a specific point. Then start to show this problem 

if [ 0 -eq 1 ]   ; then make -i -r MODULE_DIRS="-I../dyn_em -I../dyn_nmm  -I/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90  -I/global/home/users/ejhernandezb/installer/amd/WRF/main -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_int -I/global/home/users/ejhernandezb/installer/amd/WRF/frame -I/global/home/users/ejhernandezb/installer/amd/WRF/share -I/global/home/users/ejhernandezb/installer/amd/WRF/phys -I/global/home/users/ejhernandezb/installer/amd/WRF/wrftladj -I/global/home/users/ejhernandezb/installer/amd/WRF/chem -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -I/global/scratch/users/ejhernandezb/libs/amd/netcdf/include " wrf_hydro ; fi

( cd main ; make -i -r RLFLAGS="" MODULE_DIRS="-I../dyn_em -I../dyn_nmm  -I/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90  -I/global/home/users/ejhernandezb/installer/amd/WRF/main -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_int -I/global/home/users/ejhernandezb/installer/amd/WRF/frame -I/global/home/users/ejhernandezb/installer/amd/WRF/share -I/global/home/users/ejhernandezb/installer/amd/WRF/phys -I/global/home/users/ejhernandezb/installer/amd/WRF/wrftladj -I/global/home/users/ejhernandezb/installer/amd/WRF/chem -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -I/global/scratch/users/ejhernandezb/libs/amd/netcdf/include " SOLVER=em em_wrf )

make[1]: Entering directory `/global/home/users/ejhernandezb/installer/amd/WRF/main'

ranlib  libwrflib.a

time mpif90 -f90=flang -o wrf.exe -fopenmp -O3 -march=znver1 -mavx2   -w -ffree-form -ffree-line-length-none -fconvert=big-endian -frecord-marker=4    wrf.o ../main/module_wrf_top.o libwrflib.a /global/home/users/ejhernandezb/installer/amd/WRF/external/fftpack/fftpack5/libfftpack.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib1/libio_grib1.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib_share/libio_grib_share.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_int/libwrfio_int.a -L/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90 -lesmf_time /global/home/users/ejhernandezb/installer/amd/WRF/external/RSL_LITE/librsl_lite.a /global/home/users/ejhernandezb/installer/amd/WRF/frame/module_internal_header_util.o /global/home/users/ejhernandezb/installer/amd/WRF/frame/pack_utils.o -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -lwrfio_nf -L/global/scratch/users/ejhernandezb/libs/amd/netcdf/lib -lnetcdff -lnetcdf  -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_pnetcdf -lwrfio_pnf -L/global/scratch/users/ejhernandezb//libs/pnetcdf/lib -lpnetcdf   -L/global/scratch/users/ejhernandezb//libs/hdf5/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lm -lz   -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -L/global/scratch/users/ejhernandezb//libs/hdf5/lib/ -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -lhdf5_hl -lhdf5 -lz   

ld.lld: error: undefined symbol: bep_ndm_

>>> referenced by module_check_a_mundo.f90:375

>>>               module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a

ld.lld: error: undefined symbol: bep_nz_um_

>>> referenced by module_check_a_mundo.f90:376

>>>               module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a

ld.lld: error: undefined symbol: bep_ng_u_

>>> referenced by module_check_a_mundo.f90:377

>>>               module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a

ld.lld: error: undefined symbol: bep_nwr_u_

>>> referenced by module_check_a_mundo.f90:378

>>>               module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a

ld.lld: error: undefined symbol: bep_bem_ndm_

>>> referenced by module_check_a_mundo.f90:381

>>>               module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a

ld.lld: error: undefined symbol: bep_bem_nz_um_

>>> referenced by module_check_a_mundo.f90:382

>>>               module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a

[santosh_zanjurne]

Are you compiling with "-j 1" ?  That would help you locate the first compilation error. Some times the 'make' will continue in parallel compilation and show different errors, overlapping the original/first one.  Also you may want to check if the undefined symbols have extra underscore -- refer -fno-underscoring flag. And for ar/ranlib, you should use llvm-ar llvm-ranlib etc...  Hope this helps.

Regards,

Santosh

[eshernan]

compiling with -j1, the first error that appear is:

time mpif90 -f90=flang -o wrf.exe -fopenmp -O3 -march=znver1 -mavx2   -w -ffree-form -ffree-line-length-none -fconvert=big-endian -frecord-marker=4    wrf.o ../main/module_wrf_top.o libwrflib.a /global/home/users/ejhernandezb/installer/amd/WRF/external/fftpack/fftpack5/libfftpack.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib1/libio_grib1.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib_share/libio_grib_share.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_int/libwrfio_int.a -L/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90 -lesmf_time /global/home/users/ejhernandezb/installer/amd/WRF/external/RSL_LITE/librsl_lite.a /global/home/users/ejhernandezb/installer/amd/WRF/frame/module_internal_header_util.o /global/home/users/ejhernandezb/installer/amd/WRF/frame/pack_utils.o -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -lwrfio_nf -L/global/scratch/users/ejhernandezb/libs/amd/netcdf/lib -lnetcdff -lnetcdf  -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_pnetcdf -lwrfio_pnf -L/global/scratch/users/ejhernandezb//libs/pnetcdf/lib -lpnetcdf   -L/global/scratch/users/ejhernandezb//libs/hdf5/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lm -lz   -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -L/global/scratch/users/ejhernandezb//libs/hdf5/lib/ -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -lhdf5_hl -lhdf5 -lz   

ld.lld: error: undefined symbol: bep_ndm_

>>> referenced by module_check_a_mundo.f90:375

>>>               module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a

[eshernan]

I change the  ar to llvm-ar and libtool to llvm-ranllib, and compile with -j1, the problem now is: 

make -i -r -j 1 non_nmm      ; \

                  llvm-ar  ru ../main/libwrflib.a module_cam_shr_kind_mod.o module_cam_support.o module_cam_shr_const_mod.o module_cam_physconst.o module_cam_cldwat.o module_cam_wv_saturation.o module_cam_esinti.o module_cam_gffgch.o module_cam_error_function.o module_cam_constituents.o module_cam_trb_mtn_stress.o module_cam_molec_diff.o module_cam_upper_bc.o module_cam_bl_diffusion_solver.o module_cam_bl_eddy_diff.o module_cam_mp_microp_aero.o module_cam_mp_cldwat2m_micro.o module_cam_mp_ndrop.o module_cam_mp_modal_aero_initialize_data_phys.o module_cam_mp_radconstants.o module_cam_mp_conv_water.o module_cam_mp_qneg3.o module_data_cam_mam_aero.o module_data_cam_mam_asect.o module_cam_infnan.o  module_bl_ysu.o module_bl_shinhong.o module_bl_mrf.o module_bl_gfs.o module_bl_gfsedmf.o module_bl_myjpbl.o module_bl_qnsepbl.o module_bl_acm.o module_bl_mynn.o module_bl_fogdes.o module_bl_gwdo.o module_bl_myjurb.o module_bl_boulac.o module_checkerror.o module_bl_camuwpbl_driver.o module_bl_mfshconvpbl.o module_shcu_camuwshcu_driver.o module_shcu_camuwshcu.o module_shcu_deng.o module_shcu_grims.o module_shcu_nscv.o module_cu_camzm_driver.o module_cu_camzm.o module_bl_temf.o module_bl_gbmpbl.o module_cu_g3.o module_cu_kf.o module_cu_bmj.o module_cu_kfeta.o module_cu_mskf.o module_cu_tiedtke.o module_cu_ntiedtke.o module_cu_gd.o module_cu_gf_wrfdrv.o module_cu_gf_deep.o module_cu_gf_sh.o module_cu_ksas.o module_cu_nsas.o module_cu_sas.o module_gocart_coupling.o module_cu_scalesas.o module_cu_osas.o module_cu_kfcup.o module_mp_radar.o module_mp_kessler.o module_mp_lin.o module_mp_sbu_ylin.o module_mp_wsm3.o module_mp_wsm5.o module_mp_wsm6.o module_mp_wsm7.o module_mp_etanew.o module_mp_fer_hires.o module_mp_HWRF.o module_mp_thompson.o module_mp_SBM_polar_radar.o module_mp_full_sbm.o module_mp_fast_sbm.o module_ltng_lpi.o module_mp_gsfcgce.o module_mp_gsfcgce_4ice_nuwrf.o module_mp_morr_two_moment.o module_mp_p3.o module_mp_jensen_ishmael.o module_mp_morr_two_moment_aero.o module_mp_milbrandt2mom.o module_mp_nssl_2mom.o module_mp_wdm5.o module_mp_wdm6.o module_mp_wdm7.o module_mp_cammgmp_driver.o module_ra_sw.o module_ra_clWRF_support.o module_ra_gsfcsw.o module_ra_goddard.o module_ra_effective_radius.o module_ra_rrtm.o module_ra_rrtmg_lw.o module_ra_rrtmg_sw.o module_ra_rrtmg_lwf.o module_ra_rrtmg_swf.o module_ra_rrtmg_lwk.o module_ra_rrtmg_swk.o module_ra_cam_support.o module_ra_cam.o module_ra_gfdleta.o module_ra_flg.o module_ra_HWRF.o module_ra_hs.o module_ra_aerosol.o module_ra_farms.o module_sf_sfclay.o module_sf_sfclayrev.o module_sf_gfs.o module_sf_exchcoef.o module_sf_gfdl.o module_sf_slab.o module_sf_noahdrv.o module_sf_noahlsm.o module_sf_clm.o module_sf_ssib.o module_sf_noah_seaice.o module_sf_noah_seaice_drv.o module_sf_noahlsm_glacial_only.o module_sf_noahmp_groundwater.o module_sf_gecros.o module_sf_noahmpdrv.o module_sf_noahmplsm.o module_sf_noahmp_glacier.o module_sf_urban.o module_sf_bep.o module_sf_bep_bem.o module_sf_bem.o module_sf_pxlsm.o module_sf_pxlsm_data.o module_sf_ruclsm.o module_sf_sfcdiags.o module_sf_sfcdiags_ruclsm.o module_sf_sstskin.o module_sf_tmnupdate.o module_sf_ocean_driver.o module_sf_oml.o module_sf_3dpwp.o module_sf_myjsfc.o module_sf_qnsesfc.o module_sf_mynn.o module_sf_fogdes.o module_sf_pxsfclay.o module_sf_temfsfclay.o module_sf_idealscmsfclay.o module_sf_scmflux.o module_sf_scmskintemp.o module_physics_addtendc.o module_physics_init.o module_gfs_machine.o module_gfs_funcphys.o module_gfs_physcons.o module_progtm.o module_pbl_driver.o module_data_gocart_dust.o module_dust_emis.o module_cumulus_driver.o module_shallowcu_driver.o module_microphysics_driver.o module_microphysics_zero_out.o module_mixactivate.o module_radiation_driver.o module_surface_driver.o module_lightning_driver.o module_ltng_cpmpr92z.o module_ltng_crmpr92.o module_ltng_iccg.o module_fdda_psufddagd.o module_fdda_spnudging.o module_fddagd_driver.o module_fddaobs_rtfdda.o module_fddaobs_driver.o module_wind_fitch.o module_sf_lake.o module_diagnostics_driver.o module_irrigation.o  \

                                   module_fr_fire_driver.o module_fr_fire_driver_wrf.o module_fr_fire_atm.o module_fr_fire_model.o module_fr_fire_core.o module_fr_fire_phys.o module_fr_fire_util.o \

                                   module_diag_afwa.o module_diag_cl.o module_diag_functions.o module_diag_hailcast.o module_diag_misc.o module_diag_nwp.o module_diag_rasm.o module_diag_pld.o module_diag_zld.o module_diag_trad_fields.o module_diag_solar.o  ; \

                fi

make[3]: Entering directory `/global/home/users/ejhernandezb/installer/amd/WRF/phys'

rm -f module_mp_SBM_polar_radar.o

sed -e "s/^\!.*'.*//" -e "s/^ *\!.*'.*//" module_mp_SBM_polar_radar.F > module_mp_SBM_polar_radar.G

/lib/cpp -P -nostdinc -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600 -DDA_CORE=0 -DWRFPLUS=0 -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4 -DLWORDSIZE=4 -DNONSTANDARD_SYSTEM_SUBR  -DWRF_USE_CLM -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT -DRPC_TYPES=1  -DDM_PARALLEL -DNETCDF -DPNETCDF -DHDF5 -DLANDREAD_STUB=1 -DUSE_ALLOCATABLES -Dwrfmodel -DGRIB1 -DINTIO -DKEEP_INT_AROUND -DLIMIT_ARGS -DBUILD_RRTMG_FAST=0 -DBUILD_RRTMK=0 -DBUILD_SBM_FAST=1 -DSHOW_ALL_VARS_USED=0 -DCONFIG_BUF_LEN=65536 -DMAX_DOMAINS_F=21 -DMAX_HISTORY=25 -DNMM_NEST=0  -I. -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT  -D_OPENMP module_mp_SBM_polar_radar.G  > module_mp_SBM_polar_radar.bb

/global/home/users/ejhernandezb/installer/amd/WRF/tools/standard.exe module_mp_SBM_polar_radar.bb | /lib/cpp -P -nostdinc -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT > module_mp_SBM_polar_radar.f90

rm -f module_mp_SBM_polar_radar.G module_mp_SBM_polar_radar.bb

time mpif90 -f90=flang -o module_mp_SBM_polar_radar.o -c -O3 -march=znver1 -mavx2   -w -ffree-form -ffree-line-length-none -fconvert=big-endian -frecord-marker=4  -fopenmp -I../dyn_em -I../dyn_nmm  -I/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90  -I/global/home/users/ejhernandezb/installer/amd/WRF/main -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_int -I/global/home/users/ejhernandezb/installer/amd/WRF/frame -I/global/home/users/ejhernandezb/installer/amd/WRF/share -I/global/home/users/ejhernandezb/installer/amd/WRF/phys -I/global/home/users/ejhernandezb/installer/amd/WRF/wrftladj -I/global/home/users/ejhernandezb/installer/amd/WRF/chem -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -I/global/scratch/users/ejhernandezb/libs/amd/netcdf/include    module_mp_SBM_polar_radar.f90

F90-S-0038-Symbol, derf, has not been explicitly declared (module_mp_SBM_polar_radar.f90)

  0 inform,   0 warnings,   1 severes, 0 fatal for calc_dc_wet_snow

0.17user 0.05system 0:00.29elapsed 77%CPU (0avgtext+0avgdata 22720maxresident)k

32inputs+5800outputs (0major+22359minor)pagefaults 0swaps

make[3]: [module_mp_SBM_polar_radar.o] Error 1 (ignored)

make[3]: Leaving directory `/global/home/users/ejhernandezb/installer/amd/WRF/phys'

llvm-ar: error: module_mp_SBM_polar_radar.o: No such file or directory.

[santosh_zanjurne]

Hello esharnan,

I have reproduced the problem, i shall get back to you on this soon.

Regards,

Santosh

[santosh_zanjurne]

Hello esharnan,

You will need to make few changes in the sources :

1. Open module_mp_SBM_polar_radar.F and add below statement, may be after polar_hucm defination.

external :: derf

2. If you are using parallel netcdf, make below changes in the configure.wrf file.

a. Add below statement in LIB_EXTERNAL defination.

    -L$(WRF_SRC_ROOT_DIR)/external/io_pnetcdf -lwrfio_pnf

b. Add below statement in the io_only target.

    wrfio_pnf

Hope this helps.

Regards,

Santosh

[melahi1994]

Hello, have you achieved compile WRF with AOCC compilers?

I am trying to do these since some days ago, but I have not having much success.

I would thank you very much if you could help me.

My best regards,
Hiram.