Helm values.yaml with supabase and credentials JSON for GCP/Vertex AI (with Anthropic models)

[hnykda]

This is not going to be a complete guide, but I am using the "this would be still helpful for me if I had this couple of hours ago" (I am still lost to some degree). But this works, I can login to the UI, I can call various models, secrets get mounted, keys are being used, ... I am sharing this in a hope maintainers can see what's causing issues (thanks a lot for litellm ❤️ ) or some other lost soul will be helped and can help me to clean this out. This is all at v1.61.7.

Here it is:

replicaCount: 1

image:
  repository: litellm/litellm
  pullPolicy: Always
  tag: "v1.61.7"

nameOverride: "litellm"
fullnameOverride: ""

environmentSecrets:
  - litellm-secrets

service:
  type: ClusterIP
  port: 4000

serviceAccount:
  create: false

ingress:
  enabled: false

envVars:
  GCP_CREDENTIALS: "/gcp-credentials.json"
  GCP_PROJECT_ID: "some-project-ID"
  # extremely important to use 5432, not 6543, or you will see weird hang ups such
  # as this https://stackoverflow.com/a/78989212
  DATABASE_HOST: "aws-0-us-west-1.pooler.supabase.com:5432"
  DATABASE_NAME: "postgres"
  

volumes:
  - name: gcp-credentials
    secret:
      secretName: litellm-secrets
      optional: false
      items:
        - key: GCP_CREDENTIALS
          path: "gcp-credentials.json"

volumeMounts:
  - name: gcp-credentials
    mountPath: "/gcp-credentials.json"
    subPath: "gcp-credentials.json"
    readOnly: true

proxy_config:
  model_list:
    - model_name: claude-3-5-sonnet-20241022
      litellm_params:
        model: anthropic/claude-3-5-sonnet-20241022
        api_key: os.environ/ANTHROPIC_API_KEY

    - model_name: gpt-4
      litellm_params:
        model: openai/gpt-4o-2024-08-06
        api_key: os.environ/OPENAI_API_KEY

    - model_name: perplexity-llama
      litellm_params:
        model: perplexity/llama-3.1-sonar-huge-128k-online
        api_key: os.environ/PERPLEXITYAI_API_KEY

    # gemini-1.5-pro-002
    - model_name: gemini-1.5-pro-002
      litellm_params:
        model: vertex_ai/gemini-1.5-pro-002
        vertex_project: os.environ/GCP_PROJECT_ID
        vertex_credentials: os.environ/GCP_CREDENTIALS

    # anthropic/claude-3-5-sonnet-20241022
    - model_name: vertex_ai/claude-3-5-sonnet@20241022
      litellm_params:
        model: vertex_ai/claude-3-5-sonnet-v2@20241022
        vertex_project: os.environ/GCP_PROJECT_ID
        vertex_location: us-east5
        vertex_credentials: os.environ/GCP_CREDENTIALS

    - model_name: vertex_ai/claude-3-5-sonnet@20241022
      litellm_params:
        model: vertex_ai/claude-3-5-sonnet-v2@20241022
        vertex_project: os.environ/GCP_PROJECT_ID
        vertex_location: europe-west1
        vertex_credentials: os.environ/GCP_CREDENTIALS

    - model_name: anthropic/claude-3-5-sonnet@20241022
      litellm_params:
        model: anthropic/claude-3-5-sonnet-20241022
        api_key: os.environ/ANTHROPIC_API_KEY

    - model_name: bad_model_to_test_fallbacks
      litellm_params:
        model: vertex_ai/bad_model_to_test_fallbacks
        api_key: adsasdasd


  router_settings:
    routing_strategy: simple-shuffle
    redis_host: redis-master.default.svc.cluster.local
    redis_password: os.environ/REDIS_PASSWORD
    redis_port: 6379

    fallbacks:  # apparently could be under litellm_settings as well
      - vertex_ai/claude-3-5-sonnet@20241022:
        - anthropic/claude-3-5-sonnet@20241022
      - bad_model_to_test_fallbacks:
        - vertex_ai/claude-3-5-sonnet@20241022
      
  general_settings:
    master_key: os.environ/PROXY_MASTER_KEY
    cache: false
    allow_requests_on_db_unavailable: true
    proxy_batch_write_at: 60
    database_connection_pool_limit: 10
    disable_spend_logs: true
    disable_usage_logs: false
  
  litellm_settings:
    json_logs: false
    set_verbose: false


db:
  useExisting: true
  deployStandalone: false
  # same as above with host, based on https://stackoverflow.com/a/78989212, not sure which is actually needed
  endpoint: aws-0-us-west-1.pooler.supabase.com:5432
  database: postgres
  # same as above with host, based on https://stackoverflow.com/a/78989212, not sure which is actually needed
  # also see https://github.com/BerriAI/litellm/issues/8649
  url: postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)?pgbouncer=true&connection_limit=1
  secret:
    name: litellm-secrets
    usernameKey: DATABASE_USERNAME
    passwordKey: DATABASE_PASSWORD

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1000m
    memory: 1Gi

redis:
  enabled: false

migrationJob:
  enabled: true
  disableSchemaUpdate: false

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 2

podSecurityContext:
  runAsUser: 0
  runAsGroup: 0
  fsGroup: 0
  runAsNonRoot: false

securityContext:
  allowPrivilegeEscalation: true
  readOnlyRootFilesystem: false
  runAsNonRoot: false
  runAsUser: 0

where secrets.yaml are deployed separately as something like:

apiVersion: v1
kind: Secret
metadata:
  name: litellm-secrets
  namespace: llmproxy
type: Opaque
stringData:
  # this might be entirely useless
  DATABASE_URL: postgresql://<user>:<pwd>@aws-0-us-west-1.pooler.supabase.com:6543/postgres
  DATABASE_USERNAME: ....
  DATABASE_PASSWORD: ...
  LITELLM_SALT_KEY: ....
  REDIS_PASSWORD: ....
  PROXY_MASTER_KEY: sk-...
  ANTHROPIC_API_KEY: sk-...
  OPENAI_API_KEY: sk-proj-...
  PERPLEXITYAI_API_KEY: pplx-...
  GCP_CREDENTIALS: |
    {
      "type": "service_account",
      ...
      "universe_domain": "googleapis.com"
    }

I also changed the deployment.yaml to manually take the PROXY_MASTER_KEY from the secrets above instead of autogenerating them:

            - name: PROXY_MASTER_KEY
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: PROXY_MASTER_KEY

[krrishdholakia]

Thank you for this @hnykda

cc: @ishaan-jaff on helm issues

[mknet3]

@hnykda as a workaround you can configure the master key with other different environment key like CUSTOM_PROXY_MASTER_KEY:

  proxy_config:
    general_settings:
      master_key: os.environ/CUSTOM_PROXY_MASTER_KEY

But I think the way how the master key is generated is very opinionated and imho should be open to configure it in other ways.

In my case, I'm using Secrets Store CSI Driver with Hashicorp Vault and I can't configure PROXY_MASTER_KEY secret because it is already generated by the Chart. I had to do the workaround above to make it work.

I can collaborate improving the Helm Chart.

cc @krrishdholakia @ishaan-jaff

[ishaan-jaff]

Hi @mknet3 would love to improve it. I can see @hnykda filed issues around specific pain points. Can you help us address those first or any issues you faced during setup ?

[mknet3]

Sure, I will open some PRs in the following days. This was the first one -> #8636 but I will open more to improve it. Thanks guys.

[hnykda]

Thanks you both on working on this 🙇

[hnykda]

Two things that I recalled and don't think are clear:

1. the security context was necessary for the migration jobs going through (I think it's because prisma wants to write into .cache and if it doesn't have permission to do that, it just hangs without any information)
2. some values in the chart have to be there, i.e. do not have defaults or are not optional - if you just remove them, it won't deploy (I think helm charts usually don't behave like that)