New User Struggling To Understand Why WebApplicationFactory:CreateClient() Blocking In Docker-Compose Environment On Remote GitLab.com CI Server

[dcs3spp]

Hi,

I am learning ASP.NET Core and stuggling to understand why WebApplicationFactory:CreateClient() is blocking on remote GitLab.com CI Server. I have created a small GitLab project to illustrate the issue that I am encountering.

I have a docker-compose stack, containing the following services within the same network (please see listing below):

• Kafka + Zookeeper
• netclient-run: ASP.NET Core WebApp that spawns a Background/Hosted Service that requests topic creation from the Kafka service. This is done using the confluent-kafka-dotnet client.
• netclient-test: ASP.NET Core xUnit test that uses WebApplicationFactory to start a TestServer instance and create a Http Client. The Test Server bootstraps the WebApp.

Upon startup of the live WebApp the background/hosted service runs and successfully makes a request to the kafka service. This is successful, both locally and on the GitLab CI Server.

However the test blocks continously on the Gitlab CI Server when the background service makes a request to kafka to create the topic. The same test runs successfully on local development environment. I am using WebApplicationFactory to spin-up a TestServer and create a HttpClient.

Does anyone have any ideas as to why the test is blocking on GitLab CI server?

xUnit Test Stub

namespace KafkaAdmin.Kafka.IntegrationTests
{
    public class ApiControllerTest
    {
        [Fact]
        public void ApiControllerTest_Ping_Returns_Pong()
        {
            var appFactory = new WebApplicationFactory<WebApp.Startup>();

            /** THIS CODE HANGS IN DOCKER-COMPOSE STACK ON GITLAB.COM CI SERVER BUT RUNS SUCCESSFULLY LOCALLY IN THE SAME DOCKER-COMPOSE STACK**/
            using (var client = appFactory.CreateClient())
            {
                Console.WriteLine("WE ARE IN THE TEST HERE");
            }
        }
    }
}

Output from GitLab CI Server

# WEB APP RUNS SUCCESSFULLY, REQUEST IS MADE FROM BACKGROUND SERVICE TO KAFKA DOCKER-COMPOSE SERVICE TO CREATE TOPIC
netclient-run     | .NET Run Web App Ready. Starting WebApp that contains KafkaAdmin background service.
netclient-test    | Giving netclient-run a bit of time to start up…
netclient-run     | warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
netclient-run     |       Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
netclient-run     | warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
netclient-run     |       No XML encryptor configured. Key {395ba0f4-cde9-49af-8fb4-fd16b9f05bae} may be persisted to storage in unencrypted form.
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Admin service trying to create Kafka Topic...
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Topic::eventbus, ReplicationCount::1, PartitionCount::3
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Bootstrap Servers::kafka:9092
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Admin service successfully created topic eventbus
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Now listening on: http://[::]:80
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Application started. Press Ctrl+C to shut down.
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Hosting environment: Docker
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Content root path: /KafkaAdmin/src/KafkaAdmin.WebApp

# TEST DOCKER-COMPOSE SERVICE BLOCKS CONTINOUSLY WHEN MAKING A REQUEST TO KAFKA SERVICE 
netclient-test    | .NET Client test container ready. Running test that uses WebApplicationFactory TestServer to start WebApp with KafkaAdmin background service
netclient-test    | This runs successfully in a local development environment on MacOS and Ubuntu Linux 16.04.
netclient-test    | This fails when running on a GitLab CI Server. It can be seen that the test server bootstraps the WebApp.....
netclient-test    | The KafkaAdmin background service blocks when requesting topic creation from the kafka service
netclient-test    | Test run for /KafkaAdmin/tests/KafkaAdmin.Kafka.IntegrationTests/bin/Release/netcoreapp3.1/linux-musl-x64/KafkaAdmin.Kafka.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
netclient-test    | Starting test execution, please wait...
netclient-test    | 
netclient-test    | A total of 1 test files matched the specified pattern.
netclient-test    | warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
netclient-test    |       Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
netclient-test    | warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
netclient-test    |       No XML encryptor configured. Key {2b234f03-01b4-472d-9621-db8e056db173} may be persisted to storage in unencrypted form.
netclient-test    | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-test    |       Admin service trying to create Kafka Topic...
netclient-test    | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-test    |       Topic::eventbus, ReplicationCount::1, PartitionCount::3
netclient-test    | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-test    |       Bootstrap Servers::kafka:9092
# BLOCKS INDEFINITELY HERE

GitLab Pipeline

stages:
  - test

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

services:
  - docker:dind
stages:
- test


test:
  stage: test
  image: docker/compose:alpine-1.27.4
  before_script:
    - docker version
    - docker-compose version
    - docker info
    - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - cd docker
    - docker-compose -f docker-compose.yml build
    - docker-compose -f docker-compose.yml up --exit-code-from netclient-test --abort-on-container-exit

Docker Compose Environment

---
version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.0.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - camnet
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_LOG4J_ROOT_LOGLEVEL: WARN

  kafka:
    image: confluentinc/cp-kafka:6.0.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    networks:
      - camnet
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_HEAP_OPTS: -Xmx512M -Xms512M
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_LOG4J_LOGGERS: "org.apache.zookeeper=WARN,org.apache.kafka=WARN,kafka=WARN,kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN"
  
  netclient-test:
    build:
      context: ../
      dockerfile: docker/Dockerfile
    container_name: netclient-test
    image: dcs3spp/netclient
    networks:
      - camnet
    depends_on:
      - kafka
      - netclient-run
    entrypoint: []
    command:
      - bash
      - -c
      - |-
        echo 'Giving Kafka a bit of time to start up…'
        while ! nc -z kafka 9092;
        do
          sleep 1;
        done;

        echo 'Giving netclient-run a bit of time to start up…'
        while ! nc -z netclient-run 80;
        do
          sleep 1;
        done;

        echo .NET Client test container ready. Running test that uses WebApplicationFactory TestServer to start WebApp with KafkaAdmin background service
        echo This runs successfully in a local development environment on MacOS and Ubuntu Linux 16.04.
        echo This fails when running on a GitLab CI Server. It can be seen that the test server bootstraps the WebApp.....
        echo The KafkaAdmin background service blocks when requesting topic creation from the kafka service
        dotnet test --runtime linux-musl-x64 -c Release --no-restore  --nologo tests/KafkaAdmin.Kafka.IntegrationTests/

  netclient-run:
    build:
      context: ../
      dockerfile: docker/Dockerfile
    container_name: netclient-run
    image: dcs3spp/netclient
    networks:
      - camnet
    depends_on:
      - kafka
    entrypoint: []
    command:
      - bash
      - -c
      - |-
        echo 'Giving Kafka a bit of time to start up…'
        while ! nc -z kafka 9092;
        do
          sleep 1;
        done;
        echo .NET Run Web App Ready. Starting WebApp that contains KafkaAdmin background service.
        dotnet run --runtime linux-musl-x64 -c Release --no-restore --project src/KafkaAdmin.WebApp/
    
networks:
  camnet:

Dockefile used for ASP.NET Core service in docker-compose environment

##
# Dockerfile for building and running .NET client Kafka Admin test for CI server
# This client tries to create a test topic on a kafka server. The .NET client and kafka
# server are running within a docker-compose stack on the same network
# 
# For faster image build, use in conjunction with docker buildkit
#
# Stages:
# base:             Contains alpine packages bash, curl and zip and sets prompt
#                   for bash shell to include user@host and working directory.
# build-test:       Restores packages for .NET Client and add entrypoint to run the client
#
# Usage:            DOCKER_BUILDKIT=1 docker build -f Docker/Dockerfile -t dcs3spp/netclient .
##


##
# Base Image
# Image: SDK
# 
# - Sets PS1 prompt coloured to include user@host and working directory
##
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-alpine AS base

ENV PS1A="[\d \[\e[33m\]\t\[\e[m\]] \[\e[31m\]\u\[\e[m\]@\[\e[31m\]\h\[\e[m\] \[\e[36m\]\w\[\e[m\]\n $ "

RUN apk add --update --no-cache git bash curl zip \
    && echo 'PS1=$PS1A' >> ~/.bashrc


##
# Build image:
# - Restore Dependencies for client and add entrypoint
# - Use environment Docker
# - Expose port 80
# - Add entrypoint to run KafkaAdmin.WebApp project
##
FROM base AS build-test

ENV ASPNETCORE_ENVIRONMENT Docker
ENV ASPNETCORE_URLS=http://+:80

EXPOSE 80

WORKDIR /KafkaAdmin/src/KafkaAdmin.Kafka
COPY ./src/KafkaAdmin.Kafka/ .

WORKDIR /KafkaAdmin/src/KafkaAdmin.Kafka.AutofacModule
COPY ./src/KafkaAdmin.Kafka.AutofacModule/ .

WORKDIR /KafkaAdmin/src/KafkaAdmin.WebApp
COPY ./src/KafkaAdmin.WebApp/ .

WORKDIR /KafkaAdmin/tests/KafkaAdmin.Kafka.IntegrationTests
COPY ./tests/KafkaAdmin.Kafka.IntegrationTests/ .

WORKDIR /KafkaAdmin

RUN dotnet restore -r linux-musl-x64 --no-cache tests/KafkaAdmin.Kafka.IntegrationTests/KafkaAdmin.Kafka.IntegrationTests.csproj

ENTRYPOINT [\
    "dotnet",\
    "run",\
    "-p:/KafkaAdmin/src/KafkaAdmin.WebApp/KafkaAdmin.WebApp.csproj",\
    "-r:linux-musl-x64",\
    "-c:Release",\
    "--no-restore"\
    ]

[dcs3spp]

After reading this aspnetcore issue discovered that the problem was with the implementation
of my IHostedService implementation.

The StartAsync method was performing the task, running until the request completed. By design this method is meant to be fire and forget, i.e. start the task and then continue. Updated my KafkaAdmin service to be a BackgroundService, overriding ExecuteAsync method, as listed below.
Subsequently, tests no longer blocks.

using System;
using System.Threading;
using System.Threading.Tasks;

using Confluent.Kafka;
using Confluent.Kafka.Admin;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

using KafkaAdmin.Kafka.Config;


namespace KafkaAdmin.Kafka
{
    public delegate IAdminClient KafkaAdminFactory(KafkaConfig config);

    /// <summary>Background Service to make a request from Kafka to create a topic</summary>
    public class KafkaAdminService : BackgroundService, IDisposable
    {
        private KafkaAdminFactory _Factory { get; set; }
        private ILogger<KafkaAdminService> _Logger { get; set; }
        private KafkaConfig _Config { get; set; }


        /// <summary>
        /// Retrieve KafkaConfig from appsettings
        /// </summary>
        /// <param name="config">Config POCO from appsettings file</param>
        /// <param name="clientFactory"><see cref="KafkaAdminFactory"/></param>
        /// <param name="logger">Logger instance</param>
        public KafkaAdminService(
            IOptions<KafkaConfig> config,
            KafkaAdminFactory clientFactory,
            ILogger<KafkaAdminService> logger)
        {
            if (clientFactory == null)
                throw new ArgumentNullException(nameof(clientFactory));

            if (config == null)
                throw new ArgumentNullException(nameof(config));

            _Config = config.Value ?? throw new ArgumentNullException(nameof(config));
            _Factory = clientFactory ?? throw new ArgumentNullException(nameof(clientFactory));
            _Logger = logger ?? throw new ArgumentNullException(nameof(logger));
        }


        /// <summary>
        /// Create a Kafka topic if it does not already exist
        /// </summary>
        /// <param name="token">Cancellation token required by IHostedService</param>
        /// <exception name="CreateTopicsException">
        /// Thrown for exceptions encountered except duplicate topic
        /// </exception>
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            using (var client = _Factory(_Config))
            {
                try
                {
                    _Logger.LogInformation("Admin service trying to create Kafka Topic...");
                    _Logger.LogInformation($"Topic::{_Config.Topic.Name}, ReplicationCount::{_Config.Topic.ReplicationCount}, PartitionCount::{_Config.Topic.PartitionCount}");
                    _Logger.LogInformation($"Bootstrap Servers::{_Config.Consumer.BootstrapServers}");

                    await client.CreateTopicsAsync(new TopicSpecification[] {
                        new TopicSpecification {
                            Name = _Config.Topic.Name,
                            NumPartitions = _Config.Topic.PartitionCount,
                            ReplicationFactor = _Config.Topic.ReplicationCount
                        }
                    }, null);

                    _Logger.LogInformation($"Admin service successfully created topic {_Config.Topic.Name}");
                }
                catch (CreateTopicsException e)
                {
                    if (e.Results[0].Error.Code != ErrorCode.TopicAlreadyExists)
                    {
                        _Logger.LogInformation($"An error occured creating topic {_Config.Topic.Name}: {e.Results[0].Error.Reason}");
                        throw e;
                    }
                    else
                    {
                        _Logger.LogInformation($"Topic {_Config.Topic.Name} already exists");
                    }
                }
            }

            _Logger.LogInformation("Kafka Consumer thread started");

            await Task.CompletedTask;
        }


        /// <summary>
        /// Call base class dispose
        /// </summary>
        public override void Dispose()
        {
            base.Dispose();
        }
    }
}

Still confused as to why the live WebApp starts successfully. Why is this just an issue for the TestServer?