I'm trying to compile WRF 4.0 with aocc v 2.1.0, but I'm experimenting with several problems trying to compile . My Current compile.wrf is:
DESCRIPTION = CLANG ($SFC/$SCC) :aocc 2.1
DMPARALLEL = 1
OMPCPP = -D_OPENMP
OMP = -fopenmp
OMPCC = -fopenmp
SFC = flang
SCC = clang
CCOMP = clang
DM_FC = mpif90 -f90=$(SFC)
DM_CC = mpicc -cc=$(SCC)
FC = time $(DM_FC)
CC = $(DM_CC) -DFSEEKO64_OK
LD = $(FC)
RWORDSIZE = $(NATIVE_RWORDSIZE)
PROMOTION = #-fdefault-real-8
ARCH_LOCAL = -DNONSTANDARD_SYSTEM_SUBR -DWRF_USE_CLM $(NETCDF4_IO_OPTS) -DRPC_TYPES=1
CFLAGS_LOCAL = -O3 -march=znver2 -mavx2 # -DRSL0_ONLY
LDFLAGS_LOCAL =
CPLUSPLUSLIB =
ESMF_LDFLAG = $(CPLUSPLUSLIB)
FCOPTIM = -O3 -march=znver2 -mavx2
FCREDUCEDOPT = $(FCOPTIM)
FCNOOPT = -O0
FCDEBUG = # -g $(FCNOOPT) # -fbacktrace -ggdb -fcheck=bounds,do,mem,pointer -ffpe-trap=invalid,zero,overflow
FORMAT_FIXED = -ffixed-form
FORMAT_FREE = -ffree-form -ffree-line-length-none
FCSUFFIX =
BYTESWAPIO = -fconvert=big-endian -frecord-marker=4
FCBASEOPTS_NO_G = -w $(FORMAT_FREE) $(BYTESWAPIO)
FCBASEOPTS = $(FCBASEOPTS_NO_G) $(FCDEBUG)
MODULE_SRCH_FLAG =
TRADFLAG = -traditional-cpp $(NETCDF4_IO_OPTS)
CPP = /lib/cpp -P -nostdinc
AR = ar
ARFLAGS = ru
M4 = m4 -B 14000
RANLIB = ranlib
RLFLAGS =
CC_TOOLS = $(SCC)
Some advices in the flags ?
Thanks,
Can post the details of the exact issue you are facing? Try setting FCREDUCEDOPT=01.
Between, using vectorized math library will result in better performance. Vectorized math library comes with glibc-2.22 and later. You can use below flags to use the same. You need to use these flags carefully to avoid compilation error that results when these options are used multiple times.
-mllvm -vector-library=LIBMVEC -fno-finite-math-only
Regards,
Santosh
In the compilation process appear this problems several times
make[3]: [module_mp_SBM_polar_radar.o] Error 1 (ignored)
make[3]: Leaving directory `/global/home/users/ejhernandezb/installer/amd/WRF/phys'
ar: module_mp_SBM_polar_radar.o: No such file or directory
and
ld.lld: error: undefined symbol: module_diag_zld_zld_
>>> referenced by start_em.f90:2217
>>> start_em.o:(start_domain_em_) in archive libwrflib.a
this error about undefined symbol appear several times,
I Configure the following env variables, before of compile
export SHARED=some filesystems.
export CC=clang
export FC=flang
export CXX=clang++
export MPI_CC=clang
export NETCDF=$SHARED/libs/amd/netcdf
export PNETCDF=$SHARED/libs/pnetcdf
export HDF5=$SHARED/libs/hdf5
export PHDF5=$SHARED/libs/hdf5
export JASPERLIB=$SHARED/libs/jasper/lib
export JASPERINC=$SHARED/libs/jasper/include
export ZLIB=$SHARED/libs/zlib
export MPICH=$SHARED/libs/mpich
export PAPI_DIR=$SHARED/libs/papi
export PATH=$PATH:$NETCDF/bin:$MPICH/bin:$PAPI_DIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MPICH/lib:$NETCDF/lib:$HDF5/lib:$ZLIB/lib:$PNETCDF/lib:$PAPI_DIR/lib
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
The process compiles file until a specific point. Then start to show this problem
if [ 0 -eq 1 ] ; then make -i -r MODULE_DIRS="-I../dyn_em -I../dyn_nmm -I/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90 -I/global/home/users/ejhernandezb/installer/amd/WRF/main -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_int -I/global/home/users/ejhernandezb/installer/amd/WRF/frame -I/global/home/users/ejhernandezb/installer/amd/WRF/share -I/global/home/users/ejhernandezb/installer/amd/WRF/phys -I/global/home/users/ejhernandezb/installer/amd/WRF/wrftladj -I/global/home/users/ejhernandezb/installer/amd/WRF/chem -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -I/global/scratch/users/ejhernandezb/libs/amd/netcdf/include " wrf_hydro ; fi
( cd main ; make -i -r RLFLAGS="" MODULE_DIRS="-I../dyn_em -I../dyn_nmm -I/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90 -I/global/home/users/ejhernandezb/installer/amd/WRF/main -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_int -I/global/home/users/ejhernandezb/installer/amd/WRF/frame -I/global/home/users/ejhernandezb/installer/amd/WRF/share -I/global/home/users/ejhernandezb/installer/amd/WRF/phys -I/global/home/users/ejhernandezb/installer/amd/WRF/wrftladj -I/global/home/users/ejhernandezb/installer/amd/WRF/chem -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -I/global/scratch/users/ejhernandezb/libs/amd/netcdf/include " SOLVER=em em_wrf )
make[1]: Entering directory `/global/home/users/ejhernandezb/installer/amd/WRF/main'
ranlib libwrflib.a
time mpif90 -f90=flang -o wrf.exe -fopenmp -O3 -march=znver1 -mavx2 -w -ffree-form -ffree-line-length-none -fconvert=big-endian -frecord-marker=4 wrf.o ../main/module_wrf_top.o libwrflib.a /global/home/users/ejhernandezb/installer/amd/WRF/external/fftpack/fftpack5/libfftpack.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib1/libio_grib1.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib_share/libio_grib_share.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_int/libwrfio_int.a -L/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90 -lesmf_time /global/home/users/ejhernandezb/installer/amd/WRF/external/RSL_LITE/librsl_lite.a /global/home/users/ejhernandezb/installer/amd/WRF/frame/module_internal_header_util.o /global/home/users/ejhernandezb/installer/amd/WRF/frame/pack_utils.o -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -lwrfio_nf -L/global/scratch/users/ejhernandezb/libs/amd/netcdf/lib -lnetcdff -lnetcdf -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_pnetcdf -lwrfio_pnf -L/global/scratch/users/ejhernandezb//libs/pnetcdf/lib -lpnetcdf -L/global/scratch/users/ejhernandezb//libs/hdf5/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lm -lz -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -L/global/scratch/users/ejhernandezb//libs/hdf5/lib/ -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -lhdf5_hl -lhdf5 -lz
ld.lld: error: undefined symbol: bep_ndm_
>>> referenced by module_check_a_mundo.f90:375
>>> module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a
ld.lld: error: undefined symbol: bep_nz_um_
>>> referenced by module_check_a_mundo.f90:376
>>> module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a
ld.lld: error: undefined symbol: bep_ng_u_
>>> referenced by module_check_a_mundo.f90:377
>>> module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a
ld.lld: error: undefined symbol: bep_nwr_u_
>>> referenced by module_check_a_mundo.f90:378
>>> module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a
ld.lld: error: undefined symbol: bep_bem_ndm_
>>> referenced by module_check_a_mundo.f90:381
>>> module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a
ld.lld: error: undefined symbol: bep_bem_nz_um_
>>> referenced by module_check_a_mundo.f90:382
>>> module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a
Are you compiling with "-j 1" ? That would help you locate the first compilation error. Some times the 'make' will continue in parallel compilation and show different errors, overlapping the original/first one. Also you may want to check if the undefined symbols have extra underscore -- refer -fno-underscoring flag. And for ar/ranlib, you should use llvm-ar llvm-ranlib etc... Hope this helps.
Regards,
Santosh
compiling with -j1, the first error that appear is:
time mpif90 -f90=flang -o wrf.exe -fopenmp -O3 -march=znver1 -mavx2 -w -ffree-form -ffree-line-length-none -fconvert=big-endian -frecord-marker=4 wrf.o ../main/module_wrf_top.o libwrflib.a /global/home/users/ejhernandezb/installer/amd/WRF/external/fftpack/fftpack5/libfftpack.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib1/libio_grib1.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_grib_share/libio_grib_share.a /global/home/users/ejhernandezb/installer/amd/WRF/external/io_int/libwrfio_int.a -L/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90 -lesmf_time /global/home/users/ejhernandezb/installer/amd/WRF/external/RSL_LITE/librsl_lite.a /global/home/users/ejhernandezb/installer/amd/WRF/frame/module_internal_header_util.o /global/home/users/ejhernandezb/installer/amd/WRF/frame/pack_utils.o -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -lwrfio_nf -L/global/scratch/users/ejhernandezb/libs/amd/netcdf/lib -lnetcdff -lnetcdf -L/global/home/users/ejhernandezb/installer/amd/WRF/external/io_pnetcdf -lwrfio_pnf -L/global/scratch/users/ejhernandezb//libs/pnetcdf/lib -lpnetcdf -L/global/scratch/users/ejhernandezb//libs/hdf5/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lm -lz -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -L/global/scratch/users/ejhernandezb//libs/hdf5/lib/ -L/global/scratch/users/ejhernandezb//libs/amd/netcdf/lib -lhdf5_hl -lhdf5 -lz
ld.lld: error: undefined symbol: bep_ndm_
>>> referenced by module_check_a_mundo.f90:375
>>> module_check_a_mundo.o:(module_check_a_mundo_check_nml_consistency_) in archive libwrflib.a
I change the ar to llvm-ar and libtool to llvm-ranllib, and compile with -j1, the problem now is:
make -i -r -j 1 non_nmm ; \
llvm-ar ru ../main/libwrflib.a module_cam_shr_kind_mod.o module_cam_support.o module_cam_shr_const_mod.o module_cam_physconst.o module_cam_cldwat.o module_cam_wv_saturation.o module_cam_esinti.o module_cam_gffgch.o module_cam_error_function.o module_cam_constituents.o module_cam_trb_mtn_stress.o module_cam_molec_diff.o module_cam_upper_bc.o module_cam_bl_diffusion_solver.o module_cam_bl_eddy_diff.o module_cam_mp_microp_aero.o module_cam_mp_cldwat2m_micro.o module_cam_mp_ndrop.o module_cam_mp_modal_aero_initialize_data_phys.o module_cam_mp_radconstants.o module_cam_mp_conv_water.o module_cam_mp_qneg3.o module_data_cam_mam_aero.o module_data_cam_mam_asect.o module_cam_infnan.o module_bl_ysu.o module_bl_shinhong.o module_bl_mrf.o module_bl_gfs.o module_bl_gfsedmf.o module_bl_myjpbl.o module_bl_qnsepbl.o module_bl_acm.o module_bl_mynn.o module_bl_fogdes.o module_bl_gwdo.o module_bl_myjurb.o module_bl_boulac.o module_checkerror.o module_bl_camuwpbl_driver.o module_bl_mfshconvpbl.o module_shcu_camuwshcu_driver.o module_shcu_camuwshcu.o module_shcu_deng.o module_shcu_grims.o module_shcu_nscv.o module_cu_camzm_driver.o module_cu_camzm.o module_bl_temf.o module_bl_gbmpbl.o module_cu_g3.o module_cu_kf.o module_cu_bmj.o module_cu_kfeta.o module_cu_mskf.o module_cu_tiedtke.o module_cu_ntiedtke.o module_cu_gd.o module_cu_gf_wrfdrv.o module_cu_gf_deep.o module_cu_gf_sh.o module_cu_ksas.o module_cu_nsas.o module_cu_sas.o module_gocart_coupling.o module_cu_scalesas.o module_cu_osas.o module_cu_kfcup.o module_mp_radar.o module_mp_kessler.o module_mp_lin.o module_mp_sbu_ylin.o module_mp_wsm3.o module_mp_wsm5.o module_mp_wsm6.o module_mp_wsm7.o module_mp_etanew.o module_mp_fer_hires.o module_mp_HWRF.o module_mp_thompson.o module_mp_SBM_polar_radar.o module_mp_full_sbm.o module_mp_fast_sbm.o module_ltng_lpi.o module_mp_gsfcgce.o module_mp_gsfcgce_4ice_nuwrf.o module_mp_morr_two_moment.o module_mp_p3.o module_mp_jensen_ishmael.o module_mp_morr_two_moment_aero.o module_mp_milbrandt2mom.o module_mp_nssl_2mom.o module_mp_wdm5.o module_mp_wdm6.o module_mp_wdm7.o module_mp_cammgmp_driver.o module_ra_sw.o module_ra_clWRF_support.o module_ra_gsfcsw.o module_ra_goddard.o module_ra_effective_radius.o module_ra_rrtm.o module_ra_rrtmg_lw.o module_ra_rrtmg_sw.o module_ra_rrtmg_lwf.o module_ra_rrtmg_swf.o module_ra_rrtmg_lwk.o module_ra_rrtmg_swk.o module_ra_cam_support.o module_ra_cam.o module_ra_gfdleta.o module_ra_flg.o module_ra_HWRF.o module_ra_hs.o module_ra_aerosol.o module_ra_farms.o module_sf_sfclay.o module_sf_sfclayrev.o module_sf_gfs.o module_sf_exchcoef.o module_sf_gfdl.o module_sf_slab.o module_sf_noahdrv.o module_sf_noahlsm.o module_sf_clm.o module_sf_ssib.o module_sf_noah_seaice.o module_sf_noah_seaice_drv.o module_sf_noahlsm_glacial_only.o module_sf_noahmp_groundwater.o module_sf_gecros.o module_sf_noahmpdrv.o module_sf_noahmplsm.o module_sf_noahmp_glacier.o module_sf_urban.o module_sf_bep.o module_sf_bep_bem.o module_sf_bem.o module_sf_pxlsm.o module_sf_pxlsm_data.o module_sf_ruclsm.o module_sf_sfcdiags.o module_sf_sfcdiags_ruclsm.o module_sf_sstskin.o module_sf_tmnupdate.o module_sf_ocean_driver.o module_sf_oml.o module_sf_3dpwp.o module_sf_myjsfc.o module_sf_qnsesfc.o module_sf_mynn.o module_sf_fogdes.o module_sf_pxsfclay.o module_sf_temfsfclay.o module_sf_idealscmsfclay.o module_sf_scmflux.o module_sf_scmskintemp.o module_physics_addtendc.o module_physics_init.o module_gfs_machine.o module_gfs_funcphys.o module_gfs_physcons.o module_progtm.o module_pbl_driver.o module_data_gocart_dust.o module_dust_emis.o module_cumulus_driver.o module_shallowcu_driver.o module_microphysics_driver.o module_microphysics_zero_out.o module_mixactivate.o module_radiation_driver.o module_surface_driver.o module_lightning_driver.o module_ltng_cpmpr92z.o module_ltng_crmpr92.o module_ltng_iccg.o module_fdda_psufddagd.o module_fdda_spnudging.o module_fddagd_driver.o module_fddaobs_rtfdda.o module_fddaobs_driver.o module_wind_fitch.o module_sf_lake.o module_diagnostics_driver.o module_irrigation.o \
module_fr_fire_driver.o module_fr_fire_driver_wrf.o module_fr_fire_atm.o module_fr_fire_model.o module_fr_fire_core.o module_fr_fire_phys.o module_fr_fire_util.o \
module_diag_afwa.o module_diag_cl.o module_diag_functions.o module_diag_hailcast.o module_diag_misc.o module_diag_nwp.o module_diag_rasm.o module_diag_pld.o module_diag_zld.o module_diag_trad_fields.o module_diag_solar.o ; \
fi
make[3]: Entering directory `/global/home/users/ejhernandezb/installer/amd/WRF/phys'
rm -f module_mp_SBM_polar_radar.o
sed -e "s/^\!.*'.*//" -e "s/^ *\!.*'.*//" module_mp_SBM_polar_radar.F > module_mp_SBM_polar_radar.G
/lib/cpp -P -nostdinc -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600 -DDA_CORE=0 -DWRFPLUS=0 -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4 -DLWORDSIZE=4 -DNONSTANDARD_SYSTEM_SUBR -DWRF_USE_CLM -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT -DRPC_TYPES=1 -DDM_PARALLEL -DNETCDF -DPNETCDF -DHDF5 -DLANDREAD_STUB=1 -DUSE_ALLOCATABLES -Dwrfmodel -DGRIB1 -DINTIO -DKEEP_INT_AROUND -DLIMIT_ARGS -DBUILD_RRTMG_FAST=0 -DBUILD_RRTMK=0 -DBUILD_SBM_FAST=1 -DSHOW_ALL_VARS_USED=0 -DCONFIG_BUF_LEN=65536 -DMAX_DOMAINS_F=21 -DMAX_HISTORY=25 -DNMM_NEST=0 -I. -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT -D_OPENMP module_mp_SBM_polar_radar.G > module_mp_SBM_polar_radar.bb
/global/home/users/ejhernandezb/installer/amd/WRF/tools/standard.exe module_mp_SBM_polar_radar.bb | /lib/cpp -P -nostdinc -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT > module_mp_SBM_polar_radar.f90
rm -f module_mp_SBM_polar_radar.G module_mp_SBM_polar_radar.bb
time mpif90 -f90=flang -o module_mp_SBM_polar_radar.o -c -O3 -march=znver1 -mavx2 -w -ffree-form -ffree-line-length-none -fconvert=big-endian -frecord-marker=4 -fopenmp -I../dyn_em -I../dyn_nmm -I/global/home/users/ejhernandezb/installer/amd/WRF/external/esmf_time_f90 -I/global/home/users/ejhernandezb/installer/amd/WRF/main -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_netcdf -I/global/home/users/ejhernandezb/installer/amd/WRF/external/io_int -I/global/home/users/ejhernandezb/installer/amd/WRF/frame -I/global/home/users/ejhernandezb/installer/amd/WRF/share -I/global/home/users/ejhernandezb/installer/amd/WRF/phys -I/global/home/users/ejhernandezb/installer/amd/WRF/wrftladj -I/global/home/users/ejhernandezb/installer/amd/WRF/chem -I/global/home/users/ejhernandezb/installer/amd/WRF/inc -I/global/scratch/users/ejhernandezb/libs/amd/netcdf/include module_mp_SBM_polar_radar.f90
F90-S-0038-Symbol, derf, has not been explicitly declared (module_mp_SBM_polar_radar.f90)
0 inform, 0 warnings, 1 severes, 0 fatal for calc_dc_wet_snow
0.17user 0.05system 0:00.29elapsed 77%CPU (0avgtext+0avgdata 22720maxresident)k
32inputs+5800outputs (0major+22359minor)pagefaults 0swaps
make[3]: [module_mp_SBM_polar_radar.o] Error 1 (ignored)
make[3]: Leaving directory `/global/home/users/ejhernandezb/installer/amd/WRF/phys'
llvm-ar: error: module_mp_SBM_polar_radar.o: No such file or directory.
Hello esharnan,
I have reproduced the problem, i shall get back to you on this soon.
Regards,
Santosh
Hello esharnan,
You will need to make few changes in the sources :
1. Open module_mp_SBM_polar_radar.F and add below statement, may be after polar_hucm defination.
external :: derf
2. If you are using parallel netcdf, make below changes in the configure.wrf file.
a. Add below statement in LIB_EXTERNAL defination.
-L$(WRF_SRC_ROOT_DIR)/external/io_pnetcdf -lwrfio_pnf
b. Add below statement in the io_only target.
wrfio_pnf
Hope this helps.
Regards,
Santosh
Hello, have you achieved compile WRF with AOCC compilers?
I am trying to do these since some days ago, but I have not having much success.
I would thank you very much if you could help me.
My best regards,
Hiram.