Add Asynchronous External Task Client and Worker

[kobiche]

Description

Summary

This Pull Request introduces a fully asynchronous variant of the external task client and worker. It provides the same core functionality as the existing synchronous modules, but leverages asyncio and httpx.AsyncClient to enable non-blocking I/O and concurrent task handling.

What’s New

1. Async External Task Client (async_external_task_client.py):

   • Exposes async methods (fetch_and_lock, complete, failure, etc.).
   • Allows for parallel fetching and handling of tasks using coroutines.
   • Includes a configurable timeout, max concurrency, and authentication just like the synchronous version.

2. Async External Task Executor (async_external_task_executor.py):

   • Manages async “execute task” logic (success, failure, BPMN error) without blocking the event loop.
   • Mirrors the pattern in external_task_executor.py, but uses await and returns coroutines.

3. Async External Task Worker (async_external_task_worker.py):

   • Subscribes to one or more topics in a non-blocking loop.
   • Spawns background tasks to process external tasks concurrently.
   • Supports graceful shutdown via stop() that cancels running jobs and fetch loops.
   • In case the task execution fails, the message is returned back to Camunda engine with a retry counter updated

4. Tests:

   • Added new test modules (test_async_external_task_executor.py and test_async_external_task_worker.py) using unittest.IsolatedAsyncioTestCase.
   • They verify success, failure, BPMN error flows, and cancellation logic for background tasks.

Why This Is Needed

• Non-blocking performance: Async I/O lets the worker handle multiple tasks concurrently without blocking, potentially improving throughput in environments with many waiting or long-running requests.
• Easier concurrency: Instead of multiple threads or processes, developers can rely on Python’s event loop for scheduling tasks.

How It Works

• To use the async variant, import AsyncExternalTaskClient or AsyncExternalTaskWorker.
• Write async callback functions (i.e., async def handle_task(task)) which can do I/O without blocking the rest of the system.
• The worker spawns tasks in the background and awaits them, respecting a max concurrency (via asyncio.Semaphore).

Backwards Compatibility

• All existing synchronous classes remain unchanged.
• The new async classes are optional and do not break the old interface.

[CLAassistant]

All committers have signed the CLA.

[yogeshrnaik]

Great contribution @kobiche Thank you.

[yogeshrnaik]

@kobiche
Can you please update the version number in the setup.cfg file?

camunda-external-task-client-python3/setup.cfg

Line 3 in 296cbd0

version = 4.4.0

I can then release next version in pypi with new version number.

[yogeshrnaik]

@kobiche
Can you please make this change in current PR and see if the build passes? Thank you.

[kobiche]

seems like building with python 3.7 is not possible. Should I remove it? I mean, even python 3.8 is already outdated: https://devguide.python.org/versions/

[yogeshrnaik]

seems like building with python 3.7 is not possible. Should I remove it? I mean, even python 3.8 is already outdated: https://devguide.python.org/versions/

@kobiche
Yes pls remove. Thank you.

[yogeshrnaik]

@kobiche
Some unit tests are failing. Please have a look. Thank you.

=========================== short test summary info ============================
FAILED camunda/client/tests/test_engine_client.py::EngineClientTest::test_get_process_instance_server_error_raises_exception - AssertionError: False is not true
FAILED camunda/client/tests/test_engine_client.py::EngineClientTest::test_start_process_server_error_raises_exception - AssertionError: False is not true
FAILED camunda/client/tests/test_engine_client_auth.py::EngineClientAuthTest::test_auth_basic_get_process_instance_server_error_raises_exception - AssertionError: False is not true
FAILED camunda/client/tests/test_engine_client_auth.py::EngineClientAuthTest::test_auth_basic_start_process_server_error_raises_exception - AssertionError: False is not true
FAILED camunda/client/tests/test_engine_client_bearer.py::EngineClientAuthTest::test_auth_basic_get_process_instance_server_error_raises_exception - AssertionError: False is not true
FAILED camunda/client/tests/test_engine_client_bearer.py::EngineClientAuthTest::test_auth_basic_start_process_server_error_raises_exception - AssertionError: False is not true
================== 6 failed, 115 passed, 7 warnings in 2.27s ===================

[kobiche]

Seems like there were some differences between Python versions (I tested with 3.11) in how the exception messages were parsed in the assertion. I fixed them.