Land Diagnostics: Available Tools and Future Needs

[adagj]

At the moment, there are no functioning land diagnostics for NorESM2.5 simulations. Since it's crucial that we can analyze the land component of NorESM, we need to discuss the best path forward. Below are some options and considerations:

NCL Diagnostics Package for NorESM2.5:
One approach is to adapt the older NCL diagnostics package for use with NorESM2.5. This package may not work out-of-the-box due to differences in file naming and variable formats, but Yanchun is currently working on updates. We should assess how much work is needed to get it fully functional for the latest model version.

CUPiD (CESM Unified Postprocessing and Diagnostics):
CESM is developing CUPiD, a new diagnostics tool designed for all model components, including land. Once available, CUPiD should offer a more integrated solution than NCL. However, it may be too early to consider integrating CUPiD with NorESM at this stage, as it's still in development.

ILAMB (International Land Model Benchmarking):
ILAMB is mainly a benchmarking tool for comparing land model performance against observations. We need to confirm whether ILAMB is installed on NIRD and available for all NorESM users, or if we need to set it up. ILAMB should serve as a complementary tool for model evaluation.

ADF for Land Diagnostics:
Could we potentially adapt the Atmospheric Diagnostics Framework (ADF) to handle basic land variables as well? It might provide a straightforward solution for plotting key land variables, depending on how challenging the adaptation would be.

@rosiealice @kjetilaas @mvertens @maritsandstad

[rosiealice]

Thanks Ada. This is very useful. A couple of points to add...

1. On Cupid and ADF, I believe that one of the plans NCAR have is indeed to integrate some land diagnostics using the same code as the atmosphere diagnostics package as the means of providing straightforward plotting of land output variables.

2. ILAMB is probably necessary to configure on NIRD. I've run ILAMB quite a lot but it is sensitive to python environment settings so would be useful to figure out whether it can work here.

3. The names of output variables in FATES are in many cases different from CLM (they have 'FATES_' prefixes) which would impact running the NCL package, for example.

I'd love to think about how to integrate the land variables into ADF as that seems like a sustainable approach. Do you have a link to the code so I can think about what it might entail?

Thanks!!

[adagj]

@rosiealice
ADF: https://github.com/NCAR/ADF
And the recipe: #17

@YanchunHe: Do we have a common ADF to use, or do we still need a personal version? I'm usually running mine, using the recipe I wrote in #17

[YanchunHe]

Thanks Ada and Rosie!

I just made some updates for the CAM diagnostics so that it will adapt to the new CAM file tags, and will manage to produce all the CAM diagnostics except for those on the pressure levels (e.g., contour plots on horizontal levels and vertical profile plots).

You will find a sample output here:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/

I just created a draft pull request for such updates. So it is not implemented yet in the installed copy on NIRD. I will merge it soon after that I confirm it looks OK.

It should have some relevant diagnostics for the land components.

Regarding the land diagnostics, I may need some more information to figure out how will the old NCL-based diagnostic tool can work for FATES. I make need to talk to you later on this.

For the new tool apart from the NCL-based tool, as I can tell now, I would prefer using iLAMB instead of expanding ADF (,except that NCAR will have initiative to do so), as this may make it difficult to merge future ADF updates from NCAR.

[mvertens]

@rosiealice @adagj @YanchunHe @maritsandstad - a few clarifications below.

• CTSM (unlike CAM) does not have the capability to output history variables on a lat/lon grid when it is running on a non lat/lon grid such as a spectral element grid. All of our current coupled simulations have CTSM running on the ne30pg3 grid like CAM. So unless I am missing something - preprocessing will need to be done before the current CTSM output from our coupled runs can work with the updated diagnostics that @YanchunHe has created.
• It's not clear if iLAMB can regrid data or if you need to regrid it before hand.
• The ADF does not regrid data to lat/lon output before analyzing it. CAM currently outputs all of its data on a lat/lon grid.
• That said - @maritsandstad has put in place a prototype utility to look at CTSM diagnostics when they are output on the SE grid and that was actually key in realizing the CTSM surface dataset had problems with soil properties. I'm not sure where that functionality is at this point.

[rosiealice]

Good point @mvertens. I guess ADF is written in python? (is it?) I wonder how hard it would be to wrap in the ESMF regridding. Should we try and discuss this a bit on friday?

[YanchunHe]

Hi @adagj @mvertens

I installed ADF under:

1. (log on the ipcc.nird.sigma2.no) /diagnostics/ADF/envs/adf_v0.11
2. (log on login.nird.sigma2.no) /projects/NS9560K/diagnostics/ADF/envs/adf_v0.11/

One should be able to load them directly:
e.g., `condo activate /diagnostics/ADF/envs/adf_v0.11'.

However, I have not test it further with more customised recipes.

First of all, I have very little idea on the grids of the land models. What I can see is that:

1. the CLM use the 'ne30np4' model grid,
2. the FATES use 'ne30pg3' as @mvertens points out.

The NCL-based clm diagnostic also remap the CLM output to a regular lat/lon grid with the ESMF library: https://www.ncl.ucar.edu/Applications/ESMF.shtml

If the NCL-based ESMF library can also handle the ne30pg3 grid, then it will straightforward to apply the diagnostic tool on the new FATES output, as the diagnostic are mostly handling the regridded files.

[rosiealice]

Sorry. I think I might have got something wrong. I thought ADF was the atmospheric diagnostics across both NorESM and CESM. I only think this because I have never thought about atmospheric diagnostics :) Didn't mean to send you down any rabbit holes.

[mvertens]

@rosiealice - ADF is just the atmosphere diagnostic package.

[mvertens]

@YanchunHe - thanks for the clarifications. My understanding has been that we want to move away from relying on NCL moving forwards since it is no longer supported. @maritsandstad was using XESMF to do the mapping - and that is in python. I didn't think we want to use the older CLM diagnostic package - but maybe that is all we have right now if we can do the remapping first? @maritsandstad - could you please clarify what you did for looking at CTSM diagnostic output when it was running on an ne30pg3 grid?

[maritsandstad]

However, I think @adagj was able to just plot directly without regridding using uarray, so maybe regridding is less essential than one would think, at least for an array of diagnostics such as generating maps etc...

[maritsandstad]

(But if you think the xesmf-stuff is valueable, I'm happy to clean up a bit and upload in a somewhat more useful format as part of a diagnostic. It does take a bit of setup to build a suitable environment with ESMF module working on Nird or other machines, though, so if we want it to be part of an integrated part of a diagnostic package that environment or at least the specs for setting it up to jive with other things needed for the diagnostics will probably also be something we'd like to provide so not everyone has to do that setup for themselves.)

[YanchunHe]

Hi @maritsandstad , Thanks!

The XESMF should be quite useful. But right now, I want to keep it within the old NCL framework, as I have a bit limited time to work on this now.

I did find a weight file from CESM, from ne30pg3 to 0.5x0.5:
on betzy: /cluster/shared/noresm/inputdata/lnd/clm2/mappingdata/maps/ne30pg3/map_ne30pg3_to_0.5x0.5_nomask_aave_da_c180515.nc

which is from the CESM inputdata repository.

In a standalone mode, the regridding with the weights works with the new CTSM output. However, I should do more work so that it work within the tool.

I used the 'landmask' data directly from the land model output, and some 'surface data' also from /cluster/shared/noresm/inputdata/lnd/clm2/surfdata.

Please let me know if you have better knowledge which version of grid data that I should use.

[rosiealice]

Hi all,

It would be great to have any solution that works in the shortish term, -if- you think getting the NCL version to run is feasible without too much investment @YanchunHe? I guess we won't know until it's actually running...

I imagine there is a variable list somewhere we can swop out the fares variables for the CLM-BGC ones? If you have a link to the code I can try and take a look.

It'll perhaps be slightly more effort (but maybe better in the long term) in terms of setup and regridding to add land output to the atmosphere diagnostics code. Maybe we should split that into a parallel thread??

[YanchunHe]

Thank you, @rosiealice !

I have made the calculation of the climatologies and time series work for the new CTSM output, and with proper remapping to lat/lon, we should make it work and produce a considerable amount of diagnostics as before.

Currently, the following variables are looked for:

for climatology : TSA,RAIN,SNOW,FSR,FSDS,FSA,FIRA,FCTR,FCEV,FGEV,QOVER,QDRAI,QRGWL,SNOWDP,FPSN,FSH,FSH_V,FSH_G,TV,TG,TSNOW,SABV,SABG,FIRE,FGR,FSM,TAUX,TAUY,ELAI,ESAI,TLAI,TSAI,LAISUN,LAISHA,BTRAN2,H2OSNO,H2OCAN,SNOWLIQ,SNOWICE,QINFL,QINTR,QDRIP,QSNOMELT,QSOIL,QVEGE,QVEGT,ERRSOI,ERRSEB,FSNO,ERRSOL,ERRH2O,TBOT,TLAKE,WIND,THBOT,QBOT,ZBOT,FLDS,FSDSNDLN,FSDSNI,FSDSVD,FSDSVDLN,FSDSVI,FSDSND,FSRND,FSRNDLN,FSRNI,FSRVD,FSRVDLN,FSRVI,Q2M,TREFMNAV,TREFMXAV,SOILLIQ,SOILICE,H2OSOI,TSOI,WA,WT,ZWT,QCHARGE,FCOV,PCO2,NEE,GPP,NPP,AR,HR,NEP,ER,SUPPLEMENT_TO_SMINN,SMINN_LEACHED,COL_FIRE_CLOSS,COL_FIRE_NLOSS,PFT_FIRE_CLOSS,PFT_FIRE_NLOSS,FIRESEASONL,FIRE_PROB,ANN_FAREA_BURNED,MEAN_FIRE_PROB,PBOT,SNOBCMCL,SNOBCMSL,SNODSTMCL,SNODSTMSL,SNOOCMCL,SNOOCMSL,BCDEP,DSTDEP,OCDEP

For timeseries:
TSA,RAIN,SNOW,TSOI,FPSN,ELAI,ESAI,TLAI,TSAI,LAISUN,LAISHA,BTRAN2,QINFL,QOVER,QRGWL,QDRAI,QINTR,QSOIL,QVEGT,SOILLIQ,SOILICE,SOILPSI,SNOWLIQ,SNOWICE,WA,ZWT,QCHARGE,FCOV,PCO2,NEE,landfrac,area,FSR,PBOT,SNOWDP,FSDS,FSA,FLDS,FIRE,FIRA,FSH,FCTR,FCEV,FGEV,FGR,WT

It does not hurt much if some of them are missing. The diagnostic will just what it finds missing.

To swap the names of the same field in the new version, it is more straightforward than add new field.

[mvertens]

@YanchunHe - thanks so much for the clarification. That sounds like an excellent plan! Thanks for working on this.

[YanchunHe]

Now I manage to produce some test diagnostics for the CLM climatologies.

It is now works for set_2, set_3, and set_5

• set 2 Horizontal contour plots of DJF, MAM, JJA, SON, and ANN means
• set 3 Line plots of monthly climatology: regional air temperature, precipitation, runoff, snow depth, radiative fluxes, and turbulent fluxes
• set 5 Tables of annual means

The test output is here, please have a look:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid_fatessp.202401007/CLM_DIAG/yrs1to10-obs/

I am working at time series (trend) diagnostics, i.e., set 1 and set 6. And hopefully be ready soon.

set 4 will not work in the foreseeing future, as it requires vertical atmospheric output, which can not be handled right now.

Please let me know if anything more is needed (but I will only be able to work more on this again in the next year).

[rosiealice]

@maritsandstad is going to look at whether her interpolation method produces the same strange numbers.

[YanchunHe]

Thanks for the comments. Now both the climatologies and time series should in general work for the CLM with SE.

set_4 for atmospheric profiles will of course not work.

See the example here:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid.20240829/CLM_DIAG/

set 1: line plots	set 2: contour	set 3: monthly plot

set 5: regional plots	set 6: regional plots

[rosiealice]

Hi @YanchunHe

This is great! Thanks for doing this :)

If I remember the diagnostics has a capability to run two different runs side by side, I think? If that remains true, could we look at the CLM-SP and FATES-SP runs together?

We should have a mini tutorial on how to do this so we can distribute the effort...

Cheers,
Rosie

[YanchunHe]

Yes, I am keep in mind of this. And it should work for model-model comparison. I will try that.

Could you give the case names of these two, and path if they are not under the ns9560k/noresm3/cases.

After I finish all necessary changes, it should be the same as before, with only one command line, but maybe with some differences in the specified keyword. I will make it clear how to make use of it.

[YanchunHe]

Another test with two model-model comparision:

https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid.20240829/CLM_DIAG/yrs1to14-n1850.ne30_tn14.hybrid_fatessp.202401007/setsIndex.html

However, this works for both simulations are new CLM versions. Some of the previous CMIP6 versions of CLM change the file frequency of output as annually for monthly data, e.g., case.clm2.h0.0001-02-01.nc, but the diagnostic tool would expect the old convention (and currently new CLM convention as, caseid.clm2.h0a.0001-01.nc).

So it will not work for most of the previous CMIP6 experiments, as I can understand.

update: a comparison between CLM-FATES with previous CMIP6 version (with *.clm2.h0.yyyy-mm.nc stream).:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid.20240829/CLM_DIAG/yrs1to14-N1850_f19_tn14_20190621/

[YanchunHe]

Now I commit these changes, and will be ready to implement these changes.

As I wrote in the pull request, there is small explanation/guide for these changes:

For the new spectral element version of CAM.

Call the diag_run as before, and the tool should automatically detect it the SE version of CAM.

For example:

diag_run -m all -c casename -s 1 -e 10 -i /path/to/case

or

diag_run -m cam -c casename -s 1 -e 10 -i /path/to/case

Sample output:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid.20240829/CAM_DIAG/

For the new CLM diagnostics:

1. for comparing new CLM with observations (model-obs comparision)
   One should use -m ctsm to indicate it is processing the new CLM.

./diag_run -m ctsm -c n1850.ne30_tn14.hybrid.20240829 -s 1 -e 14 -i /projects/NS9560K-datalake/noresm3/case

Sample output:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid.20240829/CLM_DIAG/yrs1to14-obs/

2. For comparing two new CLM simulations (model1-model2 comparison)

/diag_run -m ctsm -c1 n1850.ne30_tn14.hybrid.20240829 -s1 1 -e1 14 -c2 n1850.ne30_tn14.hybrid_fatessp.202401007 -s2 1 -e2 14 -i1 /projects/NS9560K-datalake/noresm3/cases -i2 /projects /NS9560K-datalake/noresm3/cases

Sample output:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid.20240829/CLM_DIAG/yrs1to14-n1850.ne30_tn14.hybrid_fatessp.202401007/

3. For comparing new CLM simulation with old CLM (NorESM2 version):
   One has to go to the passive mode of diag_run to tweak a little bit.

./diag_run -m ctsm -c1 n1850.ne30_tn14.hybrid.20240829 -s1 1 -e1 14 -c2 N1850_f19_tn14_20190621 -s2 1601 -e2 1614 -i1 /projects/NS9560K-datalake/noresm3/cases -i2 /projects/NS9560K/noresm/cases -p

Then

cd /scratch/$USER/diagnostics/noresm/out/CLM_DIAG

and change in the lnd_template.csh
setenv regrid_2 1 to setenv regrid_2 0

then execute the script:

./lnd_template.csh

Sample output:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid.20240829/CLM_DIAG/yrs1to14-N1850_f19_tn14_20190621/

Note, for the CMIP6 version of CLM, the diagnostic only works for some simulations with the monthly averaged field stored in files of each individual months. For those with 12-monthly files stored in one single file of a year, will not work. There is difficulty for NCO to deal with these name conventions.

[YanchunHe]

I have merged the updates, so that one can invoke diag_run:
/projects/NS9560K/diagnostics/noresm/bin/diag_run
or
/diagnostics/noresm/bin/diag_run

(It does not for the installation on Betzy so far, as there is problem with the toolchain after the major upgrade).

Thanks to @gold2718 , the wiki page is updated to include the instructions: https://github.com/NorESMhub/noresm3_dev_simulations/wiki

[adagj]

@YanchunHe
Running diag_run with model = ctsm works when compared to observation, but not when I compare two models. When running this command, which works for the other components:

for model in 'ctsm' ; do diag_run -m ${model} -c1 n1850.ne30_tn14.hybrid_fatessp.20241219 -s1 85 -e1 105 -i1 /projects/NS9560K-datalake/noresm3/cases/ -c2 n1850.ne30_tn14.hybrid_fatessp.20241219 -s2 65 -e2 84 -i2 /projects/NS9560K-datalake/noresm3/cases/ --web-dir=/projects/NS2345K-datalake/www/diagnostics/noresm/adagj/ ; done

I get this error

Starting set1 ----------------------------------------
 Copyright (C) 1995-2019 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.6.2
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
(0)	=========================================
(0)	Starting:  set_1.ncl
(0)	Start Time: Tue Jan 21 12:51:31 CET 2025
(0)	=========================================
fatal:Could not create (/scratch/adagj/diagnostics/noresm/out/CLM_DIAG/n1850.ne30_tn14.hybrid_fatessp.20241219/n1850.ne30_tn14.hybrid_fatessp.20241219_ANN_globalClimo.nc)
fatal:["Execute.c":8637]:Execute: Error occurred at or near line 103 in file /projects/NS9560K/diagnostics/noresm/packages/CLM_DIAG/code/model1-model2/set_1.ncl

 
Premature Exit (set1)
 
 
 
EXIT lnd_diag-version:/projects/NS9560K/diagnostics/noresm/packages/CLM_DIAG Tue Jan 21 12:51:31 CET 2025
Standard output and errors are saved in /scratch/adagj/diagnostics/noresm/out/CLM_DIAG/config/n1850.ne30_tn14.hybrid_fatessp.20241219/logs/out_250121_125129.log
-------------------------------------------------
   
-------------------------------------------------
TOTAL diag_run RUNTIME: 0m2s
-CLM diagnostics: 0m2s
-------------------------------------------------
DONE: Tue Jan 21 12:51:31 CET 2025

[YanchunHe]

I see that the direct reason is the program is writing data to the same intermediate data. The tool meant to compare two real different 'models/cases', instead of the same simulation but with two different periods.

I will see if I can make it work for this case.

[YanchunHe]

Hi @adagj I made some updates, so now it should be able to compare two periods of the same case.

https://ns2345k.web.sigma2.no/datalake/diagnostics/noresm/yanchun/n1850.ne30_tn14.hybrid_fatessp.20241219/CLM_DIAG/yrs85to105-n1850.ne30_tn14.hybrid_fatessp.20241219/setsIndex.html

Can you check if it looks OK? They are quite similar, but should be different.

[rosiealice]

Hi all.

For the sake of a straw man, here is a list of diagnostics to be used with FATES runs.
The first part pertains to the variable that have benchmarks in set 1 of the NCL diagnostics. The latter to model only variables. I have grouped them by type, for the sake of everyones sanity, but possibly also for use in the 'grouped variables' that appear in set 3.

BENCHMARKS IN SET1

SOME OF THESE, IN BOLD, ARE POST-PROCESSED FROM >1 OUTPUT VARIABLE.

#DRIVERS
TSA
PREC

#RUNOFF
TOTRUNOFF = QOVER+QDRAI+QRGWL

#SNOW
SNOWDP
H2OSNO
FSNO

#ALBEDOS (will have to work out how the first four are calculated
VBSA
NBSA
VWSA
NWSA
ASA = All sky albedo = FSR/FSDS 
LHEAT = latent heat = FCTR+FCEV+FGEV 

#FATES variables..
GPP -> FATES_GPP
TLAI -> FATES_LAI

MODEL ONLY

#MISC
CO2_PPMV

#RADIATION FLUXES
FSDS
FSA
FSR
SABV
SABG
FIRA
FIRE

#HEAT FLUXES
FCTR
FCEV
FGR
FSH
EFLX_LH_TOT

#ENERGY STATE
TV
TG
TBOT

#WATER STATE
SOILPSI
FSNO
H20CAN
H2OSNO
TOTSOILLIQ
TOTSOILICE
BTRAN

#WATER FLUXES
QINFL
QOVER
QINTR
QSOIL
QVEGT
QVEGE

#PLANT STATE
FATES_LEAFC
FATES_FROOTC
FATES_NONSTRUCTC
FATES_SAPWOODC
FATES_STOREC
FATES_VEGC
FATES_VEGC_ABOVEGROUND
FATES_WOOD_PRODUCT

#ECOSYSTEM STATE
FATES_FRACTION
FATES_AREA_PLANTS
FATES_AREA_TREES
FATES_TRIMMING
FATES_LAI
FATES_ELAI
FATES_CANOPY_SPREAD
FATES_CA_WEIGHTED_HEIGHT
FATES_NCOHORTS
FATES_NPATCHES
FATES_TVEG

#ALLOCATION
FATES_CROOT_ALLOC
FATES_FROOT_ALLOC
FATES_LEAF_ALLOC
ATES_SEED_ALLOC
FATES_STEM_ALLOC
FATES_STORE_ALLOC

#FLUXES
FATES_GPP
FATES_NPP
FATES_NEP
FATES_STOMATAL_COND
FATES_AUTORESP
FATES_MAINT_RESP
FATES_GROWTH_RESP
FATES_HET_RESP

#SEEDS
FATES_SEEDS_IN
FATES_SEED_BANK
FATES_UNGERM_SEED_BANK

#LITTER
FATES_LITTER_IN
FATES_LITTER_OUT

#PHENOLOGY
FATES_COLD_STATUS
FATES_DAYSINCE_COLDLEAFOFF
FATES_DAYSINCE_COLDLEAFON
FATES_GDD
FATES_NCHILLDAYS
ATES_NCOLDDAYS

#FIRE
FATES_DISTURBANCE_RATE_FIRE
FATES_FDI
FATES_FIRE_CLOSS
FATES_FIRE_INTENSITY
FATES_FUELCONSUMED
FATES_FUEL_AMOUNT
FATES_IGNITIONS
FATES_NESTEROV_INDEX
FATES_ROS

[jmaerz]

Hi, Rosie and I, we were discussing potential steps towards coupling the land to ocean via riverine transport of tracers through MOSART. While Mariana, Mat and I are working on the technical side to enable a flexible tracer transport through MOSART, it would be helpful to keep an eye on land carbon and nutrients that end up in the river network to get the right amount of those to the river network. For this, it would be helpful to have land diagnostics in place that allow to trace those (potential) fluxes during the land tuning process. In terms of the receiving ocean biogeochemistry side, we currently have the riverine input of:
DIN, DIP, DSi (dissolved silicate), DIC/Alkalinity, Fe, DOC and detritus (POM) via a climatological database in place. Hence, the (if existent) equivalents from the land side would be good to diagnose and to evaluate.

[YanchunHe]

As talked during the land meeting, here is a status update on ILAMB installation on NIRD:

Installations

One can find the related ILAMB installations under the the NIRD IPCC node (ssh ipcc.nird.sigma2.no)

• /diagnostics/ILAMB
  • /diagnostics/ILAMB/ENVS : conda installations of environment
  • /diagnostics/ILAMB/ILAMB-Data : observational datasets
  • /diagnostics/ILAMB/SRC : source files for ILAMB and scripts for formatting observational datasets

Activation

First set two environment variables:

export ILAMB_ROOT=/diagnostics/ILAMB/ILAMB-Data
export CONFIG_ROOT=/diagnostics/ILAMB/SRC/ILAMB/src/ILAMB/data
export WWW_ROOT= /projects/NS2345K-datalake/www/diagnostics/ILAMB/$USER
export CASE_NAME=CLM40cn

Then activate Conda environment:

conda activate /diagnostics/ILAMB/ENVS

and, run ilamb with a sample (cmorized) model output by CLM40cn:

mkdir -p /scratch/$USER/diagnostics/ILAMB
cd  /scratch/$USER/diagnostics/ILAMB
ilamb-run --config ${CONFIG_ROOT}/ilamb.cfg --model_root ${ILAMB_ROOT}/MODELS/ --models ${CASE_NAME} --regions global

Finally, move the generated wepages to NS2345K-datalake/www:

mkdir -p $WWW_ROOT
mv _build $WWW_ROOT/$CASE_NAME}

The results will be available under:

https://ns2345k.web.sigma2.no/datalake/diagnostics/ILAMB/$USER/CLM40cn

Note that seems the program is not up-to-date, so that the generated index webpage is not fully correct, so it looks there are no output.

But at least one can see the albedo and upward SW:

• https://ns2345k.web.sigma2.no/datalake/diagnostics/ILAMB/yanchun/CLM40cn/RadiationandEnergyCycle/Albedo/CERES/CERES.html
• https://ns2345k.web.sigma2.no/datalake/diagnostics/ILAMB/yanchun/CLM40cn/RadiationandEnergyCycle/SurfaceUpwardSWRadiation/CERES/CERES.html

It is not working for NorESM original outpout now:

ilamb-run --config ${CONFIG_ROOT}/ilamb.cfg --model_root $ILAMB_ROOT/MODELS/ --models n1850.ne30_tn14.hybrid_fatessp.20241219 --model_year 1 1850

showing it does not find the right model output, which should be figure out how it works with the original model output.

@rosiealice @kjetilaas @maritsandstad @mvdebolskiy @monsieuralok

[maritsandstad]

Hi @YanchunHe ,

This is great, but paths with the ipcc node stuff are a bit confusing to me. This lives on /datalake/NS9560K/diagnostics, right? Also, can you open write permissions there. I thought datalake would be a better place to put other stuff that doesn't need backup for diagnostics. For the xesmf_land-one, I was planning to move the prebuilt conda-environement there (maybe a good place for prebuilt environments for other diagnostic packages as well?), and I also have a default weight-file that I want to put there.

[YanchunHe]

OK, on the IPCC node:

1. the datapeak is accessed from /projects/NS9560K/
2. the datalake is from /projects/NS9560K-datalake/
3. /diagnostics is the same as /projects/NS9560K/diagnostics

So currently, the Conda lib and source code are put under the datapeak, while the observational data is under datalake; the intermediate files are under the /scratch, and the results (figures/wepages) are under the datalake.

I will apply for the new projects with Sigma2 for the diagnostic tools and datasets. Sigma2 will have a new service for test, naming 'Central Data Library'. I don't have much knowledge on it either, but we will see how will it work.

I open the permissions under: /projects/NS9560K-datalake/diagnostics if this is the place you want to put more data?

[maritsandstad]

Ok, I'm not sure whether the diagnostics need to be under datapeak, but regardless thanks for opening up, I'll put some data on datalake (at least default weight-file fits there...)

[rosiealice]

Hi @YanchunHe
I finally got round to looking into this. First, thanks so much for setting up all the environments and making it run on NIRD. This is great. I managed to run the test case for the CLM4 output. There are only two radiation variables in the CMORized output in
/diagnostics/ILAMB/ILAMB-Data/MODELS/CLM40cn/

so only those outputs are created.

For me, /projects/NS2345K-datalake/www/diagnostics/ILAMB/$USER for me does not exist, and I don't have the permission to make it. Do you?

I do have noresm diagnostics in /datalake/NS9560K/www/diagnostics/noresm/rosief/
and so I made an ilamb folder under that path and moved the output there. It appears on the internet at
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/rosief/ilamb/clm4cn/_build/

There the output makes sense in terms of the fact that there are maps of the albedo comparison
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/rosief/ilamb/clm4cn/_build/RadiationandEnergyCycle/Albedo/CERESed4.2/CLM40cn_global_bias.png

The index.html and the dashboard and all the scores haven't worked. I don't know if this is because our outputs are so limited that the comparisons all crash? Anyway, this is what I found for the 'test case' so it is great that it runs!

[YanchunHe]

For me, /projects/NS2345K-datalake/www/diagnostics/ILAMB/$USER for me does not exist, and I don't have the permission to make it. Do you?

Sorry, and now I changed the permission, so that you should be able to create your own $USER folder underneath there.

[rosiealice]

On running with the raw output data, rather than the CMORized data, the command I have been using to run ILAMB like this looks something like this (adapted to my paths on NIRD from the script I had on derecho, and similar to https://github.com/adrifoster/fates-global-cal/blob/main/ilamb_scripts/ncar_ilambscripts/rosie_ilamb_diagnostics_fates_sp).

ilamb-run
--config /nird/home/rosief/rosie_ilamb_configs/config_files/ilamb_CLMFATES_supersimple_nird.cfg
--model_root /diagnostics/ILAMB/ILAMB-Data/MODELS/
--models nf2000_ne30pg3_ne30pg3_mtn14.fatessp_20250120
--model_setup /nird/home/rosief/rosie_ilamb_configs/model_setup/model_setup_fates_derecho.txt
--filter .clm2.h0.

There are two main differences I can see, first that it uses this 'model_setup file. And the second is that it uses the '--filter' argument. I am not sure which of these is the one that is critical.

When i do this on NIRD, I get actually stuck on the same problem that I was stuck on last time I tried to get ILAMB to work on derecho. Which is that the model tries to look at the data but for whatever reason cannot find it, but then the ILAMB log file leaves you in the dark about where it is trying to look, and does not report any useful info. I reported this problem to Nate, who helpfully appeared to have addressed it in:
rubisco-sfa/ILAMB#90 (comment)

But I am still getting no guidance on this from the log file, which looks like:
[DEBUG][0][extractTimeSeries][nf2000_ne30pg3_ne30pg3_mtn14.fatessp_20250120] Could not find [QVEGE] in the model results
[DEBUG][0][WorkConfront][Evapotranspiration/MODIS][nf2000_ne30pg3_ne30pg3_mtn14.fatessp_20250120]
Traceback (most recent call last):

The critical thing that I can't find out here is whether the model is trying to look in the right place for the files. I would add debugging code to the ilamb code, but I am unsure of how to recompile the environment.

So the first thing I will do is email Nate, Keith & adrianna to figure out if I cam on the right track wrt the non-CMORized variables. But it would also be good to understand how one can enact a 'debugging' process without resorting to getting the source code changed from afar...

[rosiealice]

Just also to briefly note, I found it somewhat confusing to navigate what can and cannot be seem from the IPCC node. It did not seem to be able to find the standard output location I have been using on NIRD, (/datalake/NS9560K/noresm3/cases/) for example.

[YanchunHe]

There are two main differences I can see, first that it uses this 'model_setup file. And the second is that it uses the '--filter' argument. I am not sure which of these is the one that is critical.

I think the model_setup file is critical, and the --filter is just to accelerate the process to find the right files and variables.

Could you also share the content of your model setup file?

Indeed, how the iLMAB understand and find out the corresponding variables in the raw model output is something I don't understand and don't find information on.

But it would also be good to understand how one can enact a 'debugging' process without resorting to getting the source code changed from afar...

And I will try to find a way to run the installed ILAMB under a so-called debugging or developer mode, so that you can override the default code with your changed 'source mode' code to debug.

[YanchunHe]

So here is a recipe to install an editable version of code, so that you can change and debug:

# activate the ILAMB envs
conda activate /diagnostics/ILAMB/ENVS

# clone a copy of code (here I use a noresm branch of ilamb).
mkdir ~/diagnostics && cd ~/diagnostics              # you can chose your other preferred place for the source
git clone -b noresm git@github.com:NorESMhub/ILAMB.git
cd ILAMB
pip install -e .

# test if the installation is valid
python -c "import ILAMB; print(your_package.__file__)"

# it should give output like:
# ~/diagnostics/ILAMB/src/ILAMB/__init__.py

Then make changes of source under: ~/diagnostics/ILAMB/src/ILAMB should incorporate your own development.

[maritsandstad]

Also, @rosiealice, files that you have under /nird/home/rosief/ are not generally accessible for other people, so maybe post copies of them somewhere other people can see them, for instance here: /datalake/NS9560K/diagnostics/temp (looks like your config and setup file at least...)

[YanchunHe]

I found it somewhat confusing to navigate what can and cannot be seem from the IPCC node. It did not seem to be able to find the standard output location I have been using on NIRD, (/datalake/NS9560K/noresm3/cases/) for example.

As I wrote above, the datapeak have the same address for the normal NIRD node and the IPCC node.

For datalake, on NIRD login node, it is from /datalake/NSxxxxK, and on IPCC node, it is under /projects/NSxxxxK-datalake/

So on IPCC node, it is /projects/NS9560K-datalake/noresm3/cases

[rosiealice]

Thanks so much for the recipe @YanchunHe! I will try this tomorrow and see if I can figure out whih files it is trying to look in...

[rosiealice]

Question @YanchunHe . Should I do the above recipe on the ipcc not, or not, or does it not matter?

OK, I did all the steps you mention above and they worked out :) Will look at the code now.

[YanchunHe]

Usually does not matter if you login to the IPCC node or the normal nird login nodes.

[rosiealice]

Hi @YanchunHe,

I have been doing a bunch of debugging which I haven't quite managed to summarise yet, but one question first...

When I try the first set of instructions for the pre-installed version of ILAMB, 'ilamb_run' tries to use the version of the ilamb-run specified in

/diagnostics/ILAMB/ENVS/bin/ilamb-run 

which is

/nird/home/yanchun/diagnostics/ILAMB/bin/ilamb-run

and it tells me:

 Permission denied: '/nird/home/yanchun/diagnostics/ILAMB/bin/ilamb-run'

I can get around this by running the script directly from the common bin directory:

 /diagnostics/ILAMB/SRC/ILAMB/bin/ilamb-run --config ${CONFIG_ROOT}/ilamb.cfg --model_root ${ILAMB_ROOT}/MODELS/ --models ${CASE_NAME} --regions global

but that doesn't seem like the right thing to be doing?

[YanchunHe]

Sorry and yes, before I spotted you post here, I released the central installation /diagnostics/ILAMB/ENVS/bin/ilamb-run was pointing to my personal copy, so I fixed that. So it should be OK since late yesterday.

And I believe calling directly ' /diagnostics/ILAMB/SRC/ILAMB/bin/ilamb-run' is just fine, as the real thing that matters is the activated library/modules that these scripts are actually calling.