feat: no longer use build_depends.repos

[youtalk]

Description

If we keep specifying external repositories with the main branch in build_depends.repos, version control becomes unclear, and redundant build time increases.

This PR revises the colcon-build action to overlay the prebuilt /opt/autoware workspace, ensuring that only the version-controlled repositories in autoware:core-devel image are used.

Related links

• feat(colcon-build): source /opt/autoware/setup.sh if exists autoware-github-actions#337

How was this PR tested?

Notes for reviewers

None.

Interface changes

None.

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[xmfcx]

Since we have moved to tagged versions / nightly versions system, we need 2 versions of this build_depend.repos as well.

If we only keep the tagged versions as you've proposed, until we make a release, the CI will stop working if a repo that we depend makes an unreleased change.

This will slow down the development.

• Please check ci(build-depends.repos): separate repos for humble and nightly autoware.universe#10148 for why and how we integrated this with:
  • build_depends_humble.repos
  • build_depends_nightly.repos

[youtalk]

The current autoware.core is not completely reliable even if build-and-test-differential and build-and-test pass, as all autoware-related dependent repositories are pointing to the main branch.

I believe that strict version control is the correct decision in the long run. To achieve this, we will proceed with creating a release automation bot. However, since autoware.universe changes too frequently, build_depends.repos may still be necessary for the time being. On the other hand, I don’t think this is the case for autoware.core.

[xmfcx]

If you compare side by side:

• autoware.core/build_depends.repos
• autoware.universe/build_depends_nightly.repos

The only difference is that the for mainstream development of the autoware.universe, only the autoware_msgs version is tagged from what we already have in the autoware.core.

• So my suggestion would be to add the same solution as ci(build-depends.repos): separate repos for humble and nightly autoware.universe#10148 here too.

If we go with your solution, we'll have many failing CIs where we probably will have to bypass the restrictions, and, this will make it more unstable in the long run.

[youtalk]

I think it would be good to create build-and-test*-nightly workflows similar to autoware’s health-check-nightly workflow, and if only that workflows refer to the build_depends.repos.
https://github.com/autowarefoundation/autoware/blob/main/.github/workflows/health-check-nightly.yaml

[xmfcx]

Instead, we've made sure that the build-and-test-differential workflows use:

• nightly.repos for PRs against main.
• humble.repos for PRs against humble:

https://github.com/autowarefoundation/autoware.universe/blob/92da62945cac1df1f1ce89526e74c43f76efa118/.github/workflows/build-and-test-differential.yaml#L99-L110

      - name: Prepare build_depends.repos file (main branch)
        if: ${{ github.event.pull_request.base.ref == 'main' }}
        uses: ./.github/actions/combine-repos-action
        with:
          base_file: build_depends_humble.repos
          overlay_file: build_depends_nightly.repos
          output_file: build_depends.repos

      - name: Prepare build_depends.repos file (humble branch)
        if: ${{ github.event.pull_request.base.ref == 'humble' }}
        run: cp build_depends_humble.repos build_depends.repos
        shell: bash

And similarly, we make use of the nightly.repos for the build-and-test-daily workflows:
https://github.com/autowarefoundation/autoware.universe/blob/92da62945cac1df1f1ce89526e74c43f76efa118/.github/workflows/build-and-test-daily.yaml#L83-L88

      - name: Prepare build_depends.repos file
        uses: ./.github/actions/combine-repos-action
        with:
          base_file: build_depends_humble.repos
          overlay_file: build_depends_nightly.repos
          output_file: build_depends.repos

This covers all cases:

• The daily check are done against nightly.repos
• Per-PR changes are done against corresponding .repos file.

[youtalk]

@xmfcx This PR is for autoaware.core not for autoware.universe.

#239 (comment)

However, since autoware.universe changes too frequently, build_depends.repos may still be necessary for the time being. On the other hand, I don’t think this is the case for autoware.core.

[xmfcx]

It doesn't matter, as we keep moving in more and more packages from universe to the core, the activity rate of both repositories will equalize.

[youtalk]

That is incorrect. Autoware Core strictly manages version control and releases from the ROS build farm, which clearly differentiates it from Autoware Universe. That’s why I am trying to remove the reference of the main branches here, as it easily causes API-breaking changes.

[mitsudome-r]

Here are some comments:

• This will help us reduce the CI on Autoware.Core because we won't have to build packages like autoware_msgs, autoware_utils, and autoware_lanelet2_extensions every time we make changes to Autoware.Core.
• The drawback is that we would have to wait until the new autoware:core-devel is pushed from the autoware CI if there are updates in the dependent repositories so we really should automate the release and tagging to reduce the effort of updating the image.
• Autoware.Universe nightly.repos can stay as is for now, but I agree with Fatih that it's better to match versions of common dependent packages with Autoware.Core versions to avoid potential complications when analyzing build failure on Universe CI (The green state in the following image). This can be done as a separate task.
  
• We should be careful with the docker image that we use for the CI. Since autoware:core-devel includes Autoware Core packages in /opt/autoware, it won't be able to detect build error when we remove a package in Autoware Core. We might need a image that only has dependent packages installed.
  • For example, if there is a PR that removes autoware_localization_utils package, the build CI should fail because autoware_ekf_localizer package depends on it. However, if we source the latest /opt/autoware/setup.bash in autoware:core-devel, it finds the prebuilt package from the previous build and CI succeeds falsely.