Correctly reference mount paths

[reissbaker]

Description

Fixes the mount paths referenced in the Axolotl code that runs inside the Modal containers to correctly match the mount path set in src/axolotl/cli/cloud/__init__.py.

Motivation and Context

Fixes #2346

How has this been tested?

I ran axolotl train ./config.yaml --cloud ./cloud_config.yaml and it no longer crashed immediately as soon as the container booted inside Modal due to the hardcoded mismatched mount paths.

FWIW, it appears that preprocess and lm_eval actually never mount any local files or directories at all in __init__.py, and so I suspect they will continue to be broken since none of the configs will be copied over (they already are broken in master), but this fixes train, at least.

I'm working on a fix for preprocess, which I can add as a followup PR as it's a more involved change.

Types of changes

Bugfix

Social Handles (Optional)

@reissbaker on Twitter

[winglian]

Thanks, aren't mount paths specific to the configured cloud configuration? see https://github.com/axolotl-ai-cloud/axolotl-cookbook/blob/main/grpo/cloud.yaml#L2-L6 and https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/cloud/modal.yaml#L3-L6

Happy to accept this PR, but is this something we can solve with docs? is /workspace/mounts the default for modal already?

[reissbaker]

You are referencing volume mounts, which are configured via the cloud_config.yaml. However, where Axolotl itself uploads its configuration appears to be hardcoded in Axolotl, in two different places:

1. Where it actually uploads the configuration (line 48 of __init__.py: /workspace/mounts/config.yaml)
2. Where it later tries to read the configuration from (line 278, among others, of modal_.py: /workspace/artifacts/axolotl/config.yaml)

This means that the Modal integration is completely broken: it's impossible to have a working config, since Axolotl itself hardcodes two different paths (as described in the linked issue, #2346 ). Every attempt to use this feature will instantly crash on Modal.

Totally happy for you to make it configurable where Axolotl uploads its config files. However, currently there's no way in the cloud_config.yaml to specify where you want it to upload those files: it's just hardcoded in Axolotl. The only configurable thing in the cloud_config.yaml is volume mounts; none of them specify that they should be used for anything specific e.g. config file uploads (and you can have multiple... At which point, how do you decide which to use?). This was meant as a small bugfix PR to get the existing feature to work.

[winglian]

Oh, I see what you mean now. Thanks for walking me through it 🙏 . good to go!

[reissbaker]

Thanks @winglian! For some reason, here's what I see on my end:

Do I need specific permissions to be able to merge? Or, happy if you hit the merge button if I'm not able to.