Network stats which seem improbable

[cachedout]

Kibana version:

Serverless

Elasticsearch version:
Serverless
Server OS version:
Serverless
Browser version:
Island 1.47.22
Browser OS version:
OS X
Original install method (e.g. download page, yum, from source, etc.):
Serveress
Describe the bug:
I'm seeing results for network traffic which seem unlikely to be correct when looking at the Hosts view in Observability.

Below is the current graph.

This is a Kubernetes node and as such has many network interfaces, however it seems quite unlikely that they would be producing anything close to this level of traffic. Below is the interface configuration:

root@gke-serverless-produ-serverless-produ-ffe25127-xv6b:/# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 169.254.123.1  netmask 255.255.255.0  broadcast 169.254.123.255
        ether 02:42:e0:56:a7:96  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.128.1.90  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::4001:aff:fe80:15a  prefixlen 64  scopeid 0x20<link>
        ether 42:01:0a:80:01:5a  txqueuelen 1000  (Ethernet)
        RX packets 56357914  bytes 90461616724 (90.4 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 63109660  bytes 131654042980 (131.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 5733477  bytes 7062023205 (7.0 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5733477  bytes 7062023205 (7.0 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth2d2a1ea8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::6427:4cff:fe6a:6129  prefixlen 64  scopeid 0x20<link>
        ether 66:27:4c:6a:61:29  txqueuelen 0  (Ethernet)
        RX packets 1931847  bytes 141778823 (141.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2545015  bytes 214617613 (214.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth348973a2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::78e3:69ff:fe5f:8634  prefixlen 64  scopeid 0x20<link>
        ether 7a:e3:69:5f:86:34  txqueuelen 0  (Ethernet)
        RX packets 49863  bytes 13246279 (13.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 48885  bytes 18777456 (18.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth41b870d5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::40c:1ff:fe36:e4ce  prefixlen 64  scopeid 0x20<link>
        ether 06:0c:01:36:e4:ce  txqueuelen 0  (Ethernet)
        RX packets 3789482  bytes 658119933 (658.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3076755  bytes 363162171 (363.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth529a891a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::905d:2cff:feaa:2d59  prefixlen 64  scopeid 0x20<link>
        ether 92:5d:2c:aa:2d:59  txqueuelen 0  (Ethernet)
        RX packets 972617  bytes 341209788 (341.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 807331  bytes 283536018 (283.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth74f2452d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::8ce2:aaff:fe10:385e  prefixlen 64  scopeid 0x20<link>
        ether 8e:e2:aa:10:38:5e  txqueuelen 0  (Ethernet)
        RX packets 2071231  bytes 3038664370 (3.0 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2609200  bytes 2225660055 (2.2 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethd1d7e3f4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::984e:7dff:fe1a:ba46  prefixlen 64  scopeid 0x20<link>
        ether 9a:4e:7d:1a:ba:46  txqueuelen 0  (Ethernet)
        RX packets 2812801  bytes 322956753 (322.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2370957  bytes 1058381944 (1.0 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethd44f3d62: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::811:47ff:fefa:ebb3  prefixlen 64  scopeid 0x20<link>
        ether 0a:11:47:fa:eb:b3  txqueuelen 0  (Ethernet)
        RX packets 8950049  bytes 1324623887 (1.3 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5995848  bytes 757420044 (757.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethdd840581: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::d073:44ff:fe36:1a97  prefixlen 64  scopeid 0x20<link>
        ether d2:73:44:36:1a:97  txqueuelen 0  (Ethernet)
        RX packets 1791257  bytes 122626014 (122.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2314577  bytes 195370838 (195.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethdd37c222: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::7866:2ff:fec6:1381  prefixlen 64  scopeid 0x20<link>
        ether 7a:66:02:c6:13:81  txqueuelen 0  (Ethernet)
        RX packets 298  bytes 26544 (26.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 352  bytes 1751135 (1.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethee808314: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.120.10.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f8cc:fcff:fee0:4f6c  prefixlen 64  scopeid 0x20<link>
        ether fa:cc:fc:e0:4f:6c  txqueuelen 0  (Ethernet)
        RX packets 1183750  bytes 467549248 (467.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1191725  bytes 335772521 (335.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Running bmon on the machine shows the following:

Somewhat interestingly, it appears that the veth interfaces seem to created and destroyed dynamically.

Regardless, while this node is certainly busy and definitely sending traffic, a sustained 80 Gbit/sec of traffic is unlikely. There may be something happening.

May be also related to #184099

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[smith]

@cachedout we recently made some changes to those metrics: #189281. These changes are deployed to serverless so maybe the PR doesn't address this bug.

@jennypavlova do you know if the chart shown in the screenshot above was updated to use the revised formulas?

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.