undici instrumentation can add duplicate trace-context headers, which can break Elasticsearch

[trentm]

Sending duplicate traceparent headers in an HTTP request to Elasticsearch breaks the request, e.g.:

% curl -v https://[REDACTED].es.us-west2.gcp.elastic-cloud.com/ -H traceparent:asdf -H traceparent:qwerty
...
> GET / HTTP/2
> Host:[REDACTED].es.us-west2.gcp.elastic-cloud.com
> User-Agent: curl/8.4.0
> Accept: */*
> traceparent:asdf
> traceparent:qwerty
>
* received GOAWAY, error=0, last_stream=1
< HTTP/2 502
< content-type: application/json; charset=UTF-8
< x-cloud-request-id: xHAxKfEoQKiYGTbKqHzfzw
< x-found-handling-cluster: [REDACTED]
< x-found-handling-instance: instance-0000000000
< content-length: 51
< date: Wed, 10 Apr 2024 21:58:48 GMT
<
{"ok":false,"message":"backend closed connection"}

This can happen with undici@5, because undici instrumentation uses request.addHeader(k, v) to add trace-context headers and in undici@5 that API does not check for duplicate headers. (The reason it doesn't is that in undici@5 the request.headers are maintained as a string; .addHeader() just appends another header.

The reason one can get duplicate traceparent headers is if something else in the application is adding traceparent. It could be the user themselves, e.g. via elasticsearch client observability events which I think can be used this way, or it could be via a second APM agent in play.

[trentm]

Actually, I might be wrong in my assumption that undici@6 gets the situation we want. For example, using undici@6.11.1 with this call:

  const { statusCode, body } = await undici.request(url, {
    headers: {
      foo: 'bar',
      traceparent: 'asdf',
    },
  });

the target url got this request:

server req: GET /ping {
  host: 'localhost:51272',
  connection: 'keep-alive',
  foo: 'bar',
  traceparent: 'asdf, 00-c595479ad31182af71f0f4503872a2bc-99600c4e4ad62c2b-01',
  tracestate: 'es=s:1'
}

[trentm]

The ES issue is here, BTW: elastic/elasticsearch#107338