Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: PT-541: rewrite basically the whole thing & add support for modelcache #20

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

ef0xa
Copy link
Contributor

@ef0xa ef0xa commented Feb 10, 2025

PT-541: runpod-workers/worker-sglang: update for modelcache

Took me ages to figure out how this worked. It's not really that complicated, but it's been quite obscured by the control flow jumping around all over the place. Should be a bit easier to follow now.

To summarize the changes:

  • combine all source files into single handler
  • actual comments
  • add support for huggingface modelcache
  • add type hints to the code
  • remove unreachable code
  • update version of runpod-python client to 1.7.7

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pandyamarut : what's the file used for? can we delete it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drives the config forms at quick deploy. If you need any env var that User should be able to configure, it goes here. This schema definition is parsed over to our console as form and inputs. This was created for vllm at first to support versions and changes in their configs. We're using it for all quick deploy templates in the future.

@ef0xa ef0xa changed the title rewrite basically the whole thing rewrite basically the whole thing & add support for modelcache Feb 10, 2025
@ef0xa ef0xa changed the title rewrite basically the whole thing & add support for modelcache feat: PT-541: rewrite basically the whole thing & add support for modelcache Feb 10, 2025
@@ -1,7 +1,7 @@
ray
pandas
pyarrow
runpod~=1.7.0
runpod>=1.7.7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't do this. ~= covers for 1.7.0-1.7.x. If/when we go 1.8.x it means breaking changes. We don't want this upgrading before we can test it. Revert this line please.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drives the config forms at quick deploy. If you need any env var that User should be able to configure, it goes here. This schema definition is parsed over to our console as form and inputs. This was created for vllm at first to support versions and changes in their configs. We're using it for all quick deploy templates in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants