Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace ncar_pylib because it is deprecated and will disappear soon #1925

Closed
wants to merge 1 commit into from
Closed

Replace ncar_pylib because it is deprecated and will disappear soon #1925

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 16 additions & 46 deletions cime_config/SystemTests/fsurdatmodifyctsm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,54 +65,24 @@ def _create_config_file(self):
def _run_modify_fsurdat(self):
tool_path = os.path.join(self._ctsm_root,
'tools/modify_input_files/fsurdat_modifier')
# Need to specify a specific python version that has the required
# dependencies
python_path = _get_python_path()
subprocess.check_call([python_path, tool_path, self._cfg_file_path])
# Prepare conda environment before running the tool
# TODO slevis (delete these lines when all steps are done):
# 1) DONE Confirm that
# ./create_test FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel -c /glade/p/cgd/tss/ctsm_baselines/ctsm5.1.dev115
# works: PASS
# 2) DONE Confirm that the README instructions work in interactive mode
# >>> module unload python
# >>> module load conda
# >>> ./py_env_create
# >>> conda activate ctsm_py
# >>> cd tools/modify_input_files
# >>> ./fsurdat_modifier islas_examples/modify_fsurdat/fill_indian_ocean/modify_slevis_16pft.cfg
# 3) Execute those here
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ekluzek @billsacks
Running the plan by you before I spend time on it because I think I remember Bill having hesitations about implementing the new commands inside the test:

  • I have removed references to python_path.
  • I will have the test execute the module load, module unload, py_env_create, and conda activate before running the tool.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another way to do it would be to just ensure that python and conda are in the users path. And that the ctsm_py environment is active for conda. This would expect the user to execute those things outside of the system test itself.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forget the details, but I remember having issues in the past with loading and unloading python modules from within a system test, since CIME does its own module load of python. It may have been that it worked when running a single system test, but messed things up when running a whole test suite... but again, I forget the details.

There is some additional discussion in #1798.

I guess I'd say go ahead and try it, and as long as it works even when running a whole test suite, then it's fine by me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to remind myself what @ekluzek had already accomplished in #1798.

@billsacks thanks for the reminder of the existence of #1798!


# Run the fsurdat_modifier tool
subprocess.check_call([tool_path, self._cfg_file_path])

def _modify_user_nl(self):
append_to_user_nl_files(caseroot = self._get_caseroot(),
component = "clm",
contents = "fsurdat = '{}'".format(self._fsurdat_out))

def _get_python_path():
"""Get path to ncar_pylib's python on cheyenne

This is needed because we need a python environment that includes xarray
and its dependencies. This is currently hard-coded for cheyenne until we
come up with a robust way in CIME of ensuring that the correc python
environment is loaded.

"""
out = subprocess.check_output(['/glade/u/apps/opt/ncar_pylib/ncar_pylib',
'-l'], universal_newlines=True)

# First look for a loaded ('L') python
path = _find_path_from_pylib_output(out, 'L')
# If no loaded python found, look for a default ('D') python
if path is None:
path = _find_path_from_pylib_output(out, 'D')

if path is None:
raise RuntimeError('No python found')

return os.path.join(path, 'bin', 'python')

def _find_path_from_pylib_output(ncar_pylib_output, char):
"""Given line-by-line output from ncar_pylib, return the path to python if found

Args:
- ncar_pylib_output: line-by-line output from ncar_pylib
- char: the character to look for in the leading parenthetical expression (typically 'L' or 'D')

Returns a path to python, or None if not found
"""
# The line of interest looks like the following (for char = 'L'):
# (L) ... /path/to/python
regex = r'\(' + char + r'\).* (/\S+)'
for line in ncar_pylib_output.splitlines():
match_line = re.match(regex, line)
if match_line:
return match_line.group(1)

return None