RoadRunner HTTP i/o timeout

[Kaczmi]

Hello,
we're experiencing an issue with RoadRunner where, under certain conditions, HTTP workers are being blocked for an unexpectedly long time — in some cases over 700 seconds.

Here is a log example:

2025-04-30T12:50:28+0000	ERROR	http	write response (chunk) error	{"start": "2025-04-30T12:43:58+0000", "elapsed": 390262, "error": "write tcp 10.233.102.12:8080->10.233.102.58:43250: i/o timeout"}
2025-04-30T12:50:28+0000	INFO	http	http access log	{"read_bytes": 56034, "write_bytes": 3925, "status": 200, "method": "POST", "URI": "/evaluate-basket", "remote_address": "10.233.102.58:43250", "request_time": "2025-04-30T12:50:28+0000", "start": "2025-04-30T12:43:58+0000", "elapsed": 390262}

As shown, this request blocked an HTTP worker for 390 seconds. We've seen similar cases with even longer durations. What's strange is that this seems to bypass the expected default timeout (pool.allocation_timeout = 60s) and does not return early.

Some context:

• RoadRunner version: 2023.3.12
• Running inside Kubernetes as a standalone pod
• PHP worker command: command: php -dapc.enable_cli=1 -dopcache.enable_cli=1 -dopcache.validate_timestamps=0 packages/marketing/src/Infrastructure/Server/app.php
• relay: pipes is used

The HTTP worker pool has 20 workers with a max of 1000 jobs each

Here is our full RoadRunner configuration:

version: '3'

server:
  command: "php -dapc.enable_cli=1 -dopcache.enable_cli=1 -dopcache.validate_timestamps=0 packages/marketing/src/Infrastructure/Server/app.php"
  relay: pipes

status:
  address: 0.0.0.0:2114

http:
  address: 0.0.0.0:8080
  access_logs: true
  internal_error_code: 503
  pool:
    debug: false
    num_workers: 20
    max_jobs: 1000
    supervisor:
      ttl: 14400s
      idle_ttl: 900s
  middleware: [ "headers", "gzip" ]

  headers:
    response:
      X-Powered-By: "RoadRunner"

logs:
  mode: development
  level: debug

rpc:
  listen: tcp://127.0.0.1:6001

We haven't been able to consistently reproduce the issue — it seems random — but it happens often enough in production to become a serious bottleneck.

We're looking for insight into:

• What could cause workers to be blocked for such extended periods?
• Why does pool.allocation_timeout appear to have no effect in these cases?

Any suggestions for mitigating or debugging this?

[rustatian]

Hey @Kaczmi 👋🏻
Elapsed time means how much time you waited for the worker + how much time you spent inside the worker. pool.allocate_timeout does not work for you because it seems that these 309+ seconds are spent inside the worker, when the worker was already allocated.
I suppose that you need to use pool.supervisor.exec_ttl link timeout. ttl and idle_ttl are soft timeouts, which means RR won't kill the worker (PHP process) after that timeout elapses, but rather return the response and then kill that worker. Moreover, you're using too many let's say, restart factors. Your workers would be restarted:

1. Every 14400s (4h).
2. Every 900s (15m).
3. Every 1000 requests.

Also, feel free to update RR, you're using a fairly outdated version. Keep in mind, when updating to v2024+ you should install PHP protobuf extension (via pecl or directly)