Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for no log files to be compressed #541

Closed
blimlim opened this issue Dec 16, 2024 · 0 comments · Fixed by #542
Closed

Allow for no log files to be compressed #541

blimlim opened this issue Dec 16, 2024 · 0 comments · Fixed by #542

Comments

@blimlim
Copy link
Contributor

blimlim commented Dec 16, 2024

The implementation of model log compression in #532 has a bug which causes OM2 runs to fail.

Compression is currently run from the cice.py driver, using the get_log_files() and compress_log_files() methods. The get_log_files_method() searches the work directory for files matching regex patterns contained in the model.logs_to_compress list.

For example in cice.py we set:

payu/payu/models/cice.py

Lines 58 to 61 in 0c2a837

self.logs_to_compress = [r"iceout[0-9]{3}",
r"debug\.root\.[0-9]{2}",
r"ice_diag\.d",
r"ice_diag_out"]

which matches all the desired log files for CICE4 in ESM1.5.

While for cice5.py we set:

# Empty list means no log files will be compressed
self.logs_to_compress = []

The intended behaviour is for no files to be compressed when model.logs_to_compress is empty.

In the actual implementation, the opposite happens when model.logs_to_compress is empty:

payu/payu/models/cice.py

Lines 335 to 339 in 0c2a837

log_files = []
for filename in os.listdir(self.work_path):
if re.match("|".join(self.logs_to_compress), filename):
log_files.append(os.path.join(self.work_path, filename))
return log_files

The "|".join(self.logs_to_compress) statement produces "", which then matches every file in the model's work directory. model.compress_log_files() then tries to compress and delete everything.

In OM2, this causes the following error.

Traceback (most recent call last):
  File "/g/data/vk83/prerelease/apps/payu/dev/bin/payu-run", line 8, in <module>
    sys.exit(runscript())
  File "/g/data/vk83/prerelease/apps/payu/dev/lib/python3.10/site-packages/payu/subcommands/run_cmd.py", line 135, in runscript
    expt.archive(force_prune_restarts=run_args.force_prune_restarts)
  File "/g/data/vk83/prerelease/apps/payu/dev/lib/python3.10/site-packages/payu/experiment.py", line 799, in archive
    model.archive()
  File "/g/data/vk83/prerelease/apps/payu/dev/lib/python3.10/site-packages/payu/models/cice.py", line 324, in archive
    self.compress_log_files()
  File "/g/data/vk83/prerelease/apps/payu/dev/lib/python3.10/site-packages/payu/models/cice.py", line 353, in compress_log_files
    os.remove(file)
IsADirectoryError: [Errno 21] Is a directory: '/scratch/tm70/sw6175/access-om2/work/cice-resdate-changes-test-release-025deg_jra55_ryf-bc5b6712/ice/OUTPUT'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant