Related to #3083 #1988

Ready for testing 🎉

cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project

Hi, there. I played around with using your fork to update some existing projects and eventually got them working but ended up reverting them to their production (pip) versions so that I could revisit when I had more time.

When working on an adapting an existing project, I got an error during the build, which was reproduced when I built a project from scratch using cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project:

[django python-build-stage 4/6] RUN --mount=type=cache,target=/root/.cache/uv --mount=type=bind,source=uv.lock,target=uv.lock --mount=type=bind,source=pyproject.toml,target=pyproject.toml uv sync --frozen --no-install-project --no-dev:
0.112 Using CPython 3.12.7 interpreter at: /usr/local/bin/python
0.112 Creating virtual environment at: .venv
0.113 error: Could not find root package my-awesome-project
'

I believe that it's caused by having the blank uv.lock. I thought since the local project had the host file rw that it would generate a usable uv.lock during the local build. Is there a way to get a usable version of uv.lock on the host filesystem without running uv sync on the host filesystem? Thanks for this--very cool!

@steveputman thanks looking into this!

During a build you cannot write to the host system.

Your interpreter is inside the container together with uv so you can run uv:

docker compose -f docker-compose.local.yml run django uv lock

Struggled a lot with this and it is not ideal. So hopefully someone has a better solution.

@foarsitter
The Docker documentation was sort of vague on that, but now I know.

Your solution is better than mine (I had uv installed on my host system so just ran uv lock in the project directory.

Thanks for the help! Will let you know if anything else comes up.

I think you need to run uv sync on various start scripts (like celery), at least locally. Because you're mounting a folder to /app/.venv we actually don't have the installed virtualenv created in the base Dockerfile.

Of course we're also not sharing a virtualenv here, by my read/attempt to use this. It might be possible to just share a mount across all the services, but I'm having trouble getting that to work.

Great work! This project really need to have a better tool than the plain pip

The dependabot would need updating too. I dont know if it supports pyproject declarations

The dependabot would need updating too. I dont know if it supports pyproject declarations

I believe it does, I've found some discussions on the project, here's the link to the issue.

The dependabot would need updating too. I dont know if it supports pyproject declarations

This has been added now: dependabot/dependabot-core#10478 (comment)

a git pull away from a merge 😿

Tried it. I wonder if the requirements currently in requirements/production.txt should be moved to an optional dependencies section in pyproject.toml, so something like:

[project.optional-dependencies]
production = [
  "gunicorn==23.0.0",
  # ...
]

Currently they appear in the main project dependencies, so they will be installed in the development virtualenv.

They can then be installed by running uv sync --extra=production - or explicitly excluded with uv sync --no-extra=production

uv docs about optional dependencies.

(And thanks for doing the work - I look forward to it being merged :) )

@foobacca I see development as a superset of production. Everything from production needs to be replicated in development but production doesn't need a test framework for example.

When the sentry_sdk is added as an extra dependency I cannot use capture_exception for example because the package is missing. Another example is testing the gunicorn command. I cannot do such thing when developing when I do not have the package.

@foarsitter I see your point (though you can of course install the production dependencies if you want with uv sync --extra=production ).

I don't have a strong preference, but it does seem like a change from the master branch, where there is a separate requirements/production.txt file, suggesting that the maintainers liked to keep the production dependencies separate. So I wanted to make sure that change was explicit and intended.

I'll leave it to the maintainers. If they're happy with the change I have no objection. (And I'm quite capable of changing my project after running cookiecutter anyway 🙂 )

One other idea. I would expect the [project] and [dependency-groups] section to be the first and second sections in pyproject.toml - putting the overall project definition and dependencies at the top where it's easy to find them.

I can obviously edit it myself after running cookiecutter, so not a big deal if others prefer it at the bottom, but thought I'd mention it.

I don't have a strong preference, but it does seem like a change from the master branch, where there is a separate requirements/production.txt file, suggesting that the maintainers liked to keep the production dependencies separate. So I wanted to make sure that change was explicit and intended.

The production deps are installed locally on the master branch too, so not so much of change - see #4838 for more context

One other idea. I would expect the [project] and [dependency-groups] section to be the first and second sections in pyproject.toml - putting the overall project definition and dependencies at the top where it's easy to find them.

I can obviously edit it myself after running cookiecutter, so not a big deal if others prefer it at the bottom, but thought I'd mention it.

I agree. Dependencies should be at the top of the file

We use https://github.com/tox-dev/pyproject-fmt for dictating the order of appearance, so you don't need to worry about it :)

According to the cookiecutter docs .bat files are supported too, should that be a solution @luzfcb otherwise we should go with your version.

@luzfcb adopted your version because the .bat file didn't work out of the box. I think we are ready!

@foarsitter I am trying to test the branch for an existing project. I created a blank project with the branch and then tried to merged the differences with mine. I am not sure I completely understood what has happened to package requirements. I copied the packages I had added to local.txt and base.txt to pyproject.toml, but it still gives a ModuleNotFound error. Where should the list of needed packages be when using uv? Thanks!

Are you using docker? How did you add the requirements? Wich module cannot be found?

Are you using docker? How did you add the requirements? Wich module cannot be found?

Yes, using docker. I had added them to base.txt (they worked fine with pip) and when I saw the error I then added them to pyproject.toml too to no avail. The error is for django-money, but I have added all of these:
"django-money==3.5.3",
"html2text==2024.2.26",
"mjml-python==1.3.5",
"mercadopago==2.3.0",
"stripe==12.0.0",
"nameparser==1.1.3",
"django-afip==13.2.0",
"django-weasyprint==2.4.0",

Error is as follows: ModuleNotFoundError: No module named 'djmoney' (on the django container)

You can add all your dependencies with uv as we do in the post gen hooks: https://github.com/cookiecutter/cookiecutter-django/pull/5434/files#diff-8601710441b0dde909812d3f5a1d83245fec1f1bf94b6e510d1b72cbd2fb86c1R525

First you need to remove the extending (-r) of product.txt from local.txt. I think it would be wise to remove all the dependencies you added manually from pyproject.toml so things don't get mixed up.

Then you need to execute the following commands:

# project dependencies
docker compose -f docker-compose.local.yml add -r ./requirements/base.txt 
docker compose -f docker-compose.local.yml add -r ./requirements/production.txt 
# dev dependency group
docker compose -f docker-compose.local.yml add -r --dev ./requirements/production.txt 

This will modify you pyproject.toml but doesn't alter your image. Therefore you need to build the image again with docker compose -f docker-compose.local.yml build django.

I think you didn't rebuild the image, and therefore, missing djmoney.

I get an error when trying to execute those commands (unknown docker command: "compose add")

Anyway, I had manually added the dependencies (including django-money) to pyproject.toml and did a build --no-cache and the error is still there. I actually did a test with --progress=plain to check whether django-money is every installed and no, it is never mentioned (although it sits there in pyproject.toml).__

I even tried running docker compose -f docker-compose.local.yml run django uv lock as you mention in an earlier comment before. uv.lock correctly references django-money, but the problem persists (uv doesn't seem to realize it has to install django-money). Not sure where it is taking the list of packages it needs to install from.

I might have missed merging something when bringing code over from the empty project I generated using this branch🤷🏻‍♂️

There is a mistake in the commands, it has to be uv add.

Do you have a .venv in your directory? If you have files in the directory it will override your .venv in your container.

Did you update the compose file according to this PR?

Cool, I managed to run uv add

Error is still there. I didn't have a .venv in my directory. django-money is inside the generated .venv/lib/python3.12/site-packages

I merged all of the changes in django's Dockerfile. I didn't find any particular changes in the actual compose file.

Do you have the following volumes listed in docker-compose.local.yml for the docker service and 100% sure there is no .venv folder? (rm -rf .venv)

volumes:
      - /app/.venv
      - .:/app:z

@acyment made some changes to the Dockerfile, can you try these?

Do you have the following volumes listed in docker-compose.local.yml for the docker service and 100% sure there is no .venv folder? (rm -rf .venv)

volumes:
      - /app/.venv
      - .:/app:z

@acyment made some changes to the Dockerfile, can you try these?

Error still there. Made sure .venv is not there. Here are my local Dockerfile and compose files, in case you can spot what might be at fault. Thanks so much!

Github won't let me attach a Dockerfile or a compose file, so pasting them here:
/compose/local/django/Dockerfile

# define an alias for the specific python version used in this file.
FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim AS python

# Python build stage
FROM python AS python-build-stage

ARG APP_HOME=/app

WORKDIR ${APP_HOME}

# we need to move the virtualenv outside of the $APP_HOME directory because it will be overriden by the docker compose mount
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=0

# Install apt packages
RUN apt-get update && apt-get install --no-install-recommends -y \
  # dependencies for building Python packages
  build-essential \
  # psycopg dependencies
  libpq-dev \
  gettext \
  wait-for-it \
  weasyprint

# Requirements are installed here to ensure they will be cached.
RUN --mount=type=cache,target=/root/.cache/uv \
  --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
  --mount=type=bind,source=uv.lock,target=uv.lock:rw \
  uv sync --no-install-project

COPY . ${APP_HOME}

RUN --mount=type=cache,target=/root/.cache/uv \
  --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
  --mount=type=bind,source=uv.lock,target=uv.lock:rw \
  uv sync


# devcontainer dependencies and utils
RUN apt-get update && apt-get install --no-install-recommends -y \
  sudo git bash-completion nano ssh

# Create devcontainer user and add it to sudoers
RUN groupadd --gid 1000 dev-user \
  && useradd --uid 1000 --gid dev-user --shell /bin/bash --create-home dev-user \
  && echo dev-user ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/dev-user \
  && chmod 0440 /etc/sudoers.d/dev-user


ENV PATH="/${APP_HOME}/.venv/bin:$PATH"
ENV PYTHONPATH="${APP_HOME}/.venv/lib/python3.12/site-packages:$PYTHONPATH"

COPY ./compose/production/django/entrypoint /entrypoint
RUN sed -i 's/\r$//g' /entrypoint
RUN chmod +x /entrypoint

COPY ./compose/local/django/start /start
RUN sed -i 's/\r$//g' /start
RUN chmod +x /start


COPY ./compose/local/django/celery/worker/start /start-celeryworker
RUN sed -i 's/\r$//g' /start-celeryworker
RUN chmod +x /start-celeryworker

COPY ./compose/local/django/celery/beat/start /start-celerybeat
RUN sed -i 's/\r$//g' /start-celerybeat
RUN chmod +x /start-celerybeat

COPY ./compose/local/django/celery/flower/start /start-flower
RUN sed -i 's/\r$//g' /start-flower
RUN chmod +x /start-flower


ENTRYPOINT ["/entrypoint"]

local.yml (I haven't renamed the compose files yet)

volumes:
  cyment_com_local_postgres_data: {}
  cyment_com_local_postgres_data_backups: {}
  cyment_com_local_redis_data: {}

services:
  django: &django
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile
      tags:
        - 'acyment/django'
    image: cyment_com_local_django
    container_name: cyment_com_local_django
    volumes:
      - /app/.venv
      - .:/app:z
    depends_on:
      - postgres
      - redis
      - mailpit
    env_file:
      - ./.envs/.local/.django
      - ./.envs/.local/.postgres
    ports:
      - '8000:8000'
      - '5678:5678'
    command: /start

  postgres:
    build:
      context: .
      dockerfile: ./compose/production/postgres/Dockerfile
      tags:
        - 'acyment/postgres'
    image: cyment_com_production_postgres
    container_name: cyment_com_local_postgres
    volumes:
      - cyment_com_local_postgres_data:/var/lib/postgresql/data
      - cyment_com_local_postgres_data_backups:/backups
    env_file:
      - ./.envs/.local/.postgres

  mailpit:
    image: docker.io/axllent/mailpit:latest
    container_name: cyment_com_local_mailpit
    ports:
      - '8025:8025'

  redis:
    image: docker.io/redis:6
    container_name: cyment_com_local_redis

    volumes:
      - cyment_com_local_redis_data:/data

  celeryworker:
    <<: *django
    image: cyment_com_local_celeryworker
    container_name: cyment_com_local_celeryworker
    depends_on:
      - redis
      - postgres
      - mailpit
    ports: []
    command: /start-celeryworker

  celerybeat:
    <<: *django
    image: cyment_com_local_celerybeat
    container_name: cyment_com_local_celerybeat
    depends_on:
      - redis
      - postgres
      - mailpit
    ports: []
    command: /start-celerybeat

  flower:
    <<: *django
    image: cyment_com_local_flower
    container_name: cyment_com_local_flower
    ports:
      - '5555:5555'
    command: /start-flower

  node:
    build:
      context: .
      dockerfile: ./compose/local/node/Dockerfile
      tags:
        - 'acyment/node'
    image: cyment_com_local_node
    container_name: cyment_com_local_node
    environment:
      NODE_ENV: development
      NODE_OPTIONS: '--inspect=0.0.0.0:9229' # Expose the debugging port
    depends_on:
      - django
    volumes:
      - .:/app:z
      # http://jdlm.info/articles/2016/03/06/lessons-building-node-app-docker.html
      - /app/node_modules
    command: bun run dev
    ports:
      - '3000:3000'
      - '9229:9229'
        # Expose browsersync UI: https://www.browsersync.io/docs/options/#option-ui
      - '3001:3001'

Your files seems fine, can you try these two things?

1. sometimes celeryworker is still runing and therefore the image ins't updated. Can you remove all the running containers (docker compose down) and very with docker compose ps? After that run docker compose up django --build.

docker compose -f local.yml down
docker compose -f local.yml ps
docker compose -f local.yml up django --build

2. can you try creating a new project using this branch and check if it works:

cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project

Your files seems fine, can you try these two things?

1. sometimes celeryworker is still runing and therefore the image ins't updated. Can you remove all the running containers (`docker compose down`) and very with `docker compose ps`? After that run `docker compose up django --build`.

docker compose -f local.yml down
docker compose -f local.yml ps
docker compose -f local.yml up django --build

2. can you try creating a new project using this branch and check if it works:

cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project

1. Made it work by deleting all containers, images and volumes and rebuilding from scratch. Thanks for the patience!

Just to clarify, when I need to add a new package dependency, how should I proceed from now on? My previous workflow was editing the corresponding requirements.txt file (base.txt / local.txt / production.txt) and then rebuilding the django image. Should I manually add it to pyproject.toml or rather run uv add with docker compose run django?