Upgrade to Airflow 3.0.2

[Lee-W]

Types of changes

• New feature
• Refactoring
• Breaking change (any change that would cause existing functionality to not work as expected)
• Documentation Update

Description

• Update to airflow 3.0.2
  • build(pyproject.toml,Dockerfile): upgrade to airflow 3.0.2
  • feat(patch): update migration patch to 3.0.2 version
    • this includes the fixes that will be released in 3.0.3
  • fix(airflow.cfg): migrate to airflow 3
  • build(gitignore): add logs and simple_auth_manager_passwords.json.generated
  • build(docker-compose)
    • add dag-processor and rename airflow-related component
    • set up health checks for each airflow component
  • fix(dags)
    • replace schedule_interval as schedule since it's changed in airflow 3
    • upgrade to Airflow 3 syntax using ruff==0.11.13 AIR rules (basically just some import update)
  • feat: add apache-airflow-providers-fab for user management
    • it used to be the default in past airflow version, but it needs to be installed to enable it
• feat(dags)
  • rewrite airflow-log-cleanup as airflow_log_cleanup
  • remove airflow-db-cleanup as db access is forbidden now. also discussed in the last meeting, this is not our bottleneck

Checklist

• Add test cases to all the changes you introduce
• Run make lint and make test locally to ensure all linter checks and testing pass
• Update the documentation if necessary

Steps to Test This Pull Request

Expected behavior

Related Issue

Additional context

After this is merged and deployed, we'll need to do the following things.

1. Disable the currently active dags and keep a record of the list.
2. Back up the database.
3. Upgrade the database.
4. Delete the record of airflow-db-cleanup and airflow-log-cleanup (the new one is airflow_log_cleanup, keeping the old one can be confusing)
5. Re-enable the dags that were previously turned on.

[Lee-W]

This PR is based on #158.

[Lee-W]

We still have a few todos now.

1. https://github.com/apache/airflow/blob/5a670c6d5ac631e6526d97693e2e081b9f76a027/airflow-core/src/airflow/serialization/serialized_objects.py#L2170-L2173 This is causing problem in our cases.
2. auth

[Lee-W]

the root cause of 1 is that older dag run can have data_interval_start, data_interval_end and conf as none

[Lee-W]

the root cause of 1 is that older dag run can have data_interval_start, data_interval_end and conf as none

Fortunately, the conf part is fixed in 3.0.2. still have the data_interval part needs to be taken care

[Lee-W]

1. is pretty much resolved. 2 should be an easy one. We're close to Airflow 3 ready

[Lee-W]