|
1 | 1 | <div align="center">
|
2 | 2 |
|
3 |
| -<h1>Template | Worker</h1> |
| 3 | +<h1>OpenAI Compatible Tensort-LLM Worker | Worker</h1> |
4 | 4 |
|
5 |
| -[](https://github.com/runpod-workers/worker-template/actions/workflows/CI-test_handler.yml) |
6 |
| - |
7 |
| -[](https://github.com/runpod-workers/worker-template/actions/workflows/build-test-release.yml) |
8 |
| - |
9 |
| -🚀 | A simple worker that can be used as a starting point to build your own custom RunPod Endpoint API worker. |
10 |
| -</div> |
11 |
| - |
12 |
| -## 📖 | Getting Started |
13 |
| - |
14 |
| -1. Clone this repository. |
15 |
| -2. (Optional) Add DockerHub credentials to GitHub Secrets. |
16 |
| -3. Add your code to the `src` directory. |
17 |
| -4. Update the `handler.py` file to load models and process requests. |
18 |
| -5. Add any dependencies to the `requirements.txt` file. |
19 |
| -6. Add any other build time scripts to the`builder` directory, for example, downloading models. |
20 |
| -7. Update the `Dockerfile` to include any additional dependencies. |
21 |
| - |
22 |
| -### ⚙️ | CI/CD (GitHub Actions) |
23 |
| - |
24 |
| -As a reference this repository provides example CI/CD workflows to help you test your worker and build a docker image. The three main workflows are: |
25 |
| - |
26 |
| -1. `CI-test_handler.yml` - Tests the handler using the input provided by the `--test_input` argument when calling the file containing your handler. |
27 |
| - |
28 |
| -### Test Handler |
29 |
| - |
30 |
| -This workflow will validate that your handler works as expected. You may need to add some dependency installations to the `CI-test_handler.yml` file to ensure your handler can be tested. |
31 |
| - |
32 |
| -The action expects the following arguments to be available: |
33 |
| - |
34 |
| -- `vars.RUNNER_24GB` | The endpoint ID on RunPod for a 24GB runner. |
35 |
| -- `secrets.RUNPOD_API_KEY` | Your RunPod API key. |
36 |
| -- `secrets.GH_PAT` | Your GitHub Personal Access Token. |
37 |
| -- `vars.GH_ORG` | The GitHub organization that owns the repository, this is where the runner will be added to. |
38 |
| - |
39 |
| -### Test End-to-End |
40 |
| - |
41 |
| -This repository is setup to automatically build and push a docker image to the GitHub Container Registry. You will need to add the following to the GitHub Secrets for this repository to enable this functionality: |
42 |
| - |
43 |
| -- `DOCKERHUB_USERNAME` | Your DockerHub username for logging in. |
44 |
| -- `DOCKERHUB_TOKEN` | Your DockerHub token for logging in. |
45 |
| - |
46 |
| -Additionally, the following need to be added as GitHub actions variables: |
47 |
| - |
48 |
| -- `DOCKERHUB_REPO` | The name of the repository you want to push to. |
49 |
| -- `DOCKERHUB_IMG` | The name of the image you want to push to. |
50 |
| - |
51 |
| -The `CD-docker_dev.yml` file will build the image and push it to the `dev` tag, while the `CD-docker_release.yml` file will build the image on releases and tag it with the release version. |
52 |
| - |
53 |
| -The `CI-test_worker.yml` file will test the worker using the input provided by the `--test_input` argument when calling the file containing your handler. Be sure to update this workflow to install any dependencies you need to run your tests. |
54 |
| - |
55 |
| -## Example Input |
56 |
| - |
57 |
| -```json |
58 |
| -{ |
59 |
| - "input": { |
60 |
| - "name": "John Doe" |
61 |
| - } |
62 |
| -} |
63 |
| -``` |
64 |
| - |
65 |
| -## 💡 | Best Practices |
66 |
| - |
67 |
| -System dependency installation, model caching, and other shell tasks should be added to the `builder/setup.sh` this will allow you to easily setup your Dockerfile as well as run CI/CD tasks. |
68 |
| - |
69 |
| -Models should be part of your docker image, this can be accomplished by either copying them into the image or downloading them during the build process. |
70 |
| - |
71 |
| -If using the input validation utility from the runpod python package, create a `schemas` python file where you can define the schemas, then import that file into your `handler.py` file. |
72 |
| - |
73 |
| -## 🔗 | Links |
74 |
| - |
75 |
| -🐳 [Docker Container](https://hub.docker.com/r/runpod/serverless-hello-world) |
0 commit comments