The mgmtd service is always bound to the IP address of TCP,causing the meta service to be unable to deploy.

[feeyman]

Hello everyone, have you encountered the same problem before?
There are two network cards on my metadata node, one TCP and one RDMA;
In the cluster.xml configuration file, the RDMA (172. * * *) network address is configured, *but the mgmtd service is located on another IP address (10. * * ), causing the meta service to be unable to deploy. What is the reason for this? Thank you for your response.

IB is ok:
CA 'mlx5_0'
CA type: MT4119
Number of ports: 1
Firmware version: 16.35.4506
Hardware version: 0
Node GUID: 0x1c34da03005de8f0
System image GUID: 0x1c34da03005de8f0
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 4
LMC: 0
SM lid: 1
Capability mask: 0xa651e848
Port GUID: 0x1c34da03005de8f0
Link layer: InfiniBand

[feeyman]

@diabloneo Hello, thank you very much for your open source project. Have you encountered this situation before? The 8000 port is not bound to the RDMA IB card, but a regular TCP network card. Sorry for the inconvenience, I look forward to your reply.

[diabloneo]

@feeyman

I have studied this issue. The screenshot shows that ibdev2netdev script failed to execute ip command. The ip command is contained in iproute2 package in Ubuntu which was not installed in image 3fs:20250327.

There are some situations when ibdev2netdev will use ip command:

1. The nic card is old style ( If /sys/class/net/$d/dev_id file exist) and the card is a ib card.
2. The nic card is new style.

We tested on Mallanox CX-4 which is old style RoCE card, so it didn't use the ip command.

To fix this problem. I have upgrade the 3fs-runtime image to ship with iproute2 package and publised m3fs v0.1.6 to use the new image.

@feeyman Please trying the new version. If you still have problems, feel free to post new feedback.

[feeyman]

@feeyman

I have studied this issue. The screenshot shows that ibdev2netdev script failed to execute ip command. The ip command is contained in iproute2 package in Ubuntu which was not installed in image 3fs:20250327.

There are some situations when ibdev2netdev will use ip command:

1. The nic card is old style ( If /sys/class/net/$d/dev_id file exist) and the card is a ib card.
2. The nic card is new style.

We tested on Mallanox CX-4 which is old style RoCE card, so it didn't use the ip command.

To fix this problem. I have upgrade the 3fs-runtime image to ship with iproute2 package and publised m3fs v0.1.6 to use the new image.

@feeyman Please trying the new version. If you still have problems, feel free to post new feedback.

@diabloneo Thank you for your reply. I just experimented with the new image and the error message "no IP command" disappeared. However, the mgmtd service is still on the TCP network card (10. * * *: 8000); When importing the meta configuration, it pointed to the RDMA network card (172.0 * * *: 8000), but the final result still failed, unable to import or start the meta container.

Is this a problem with network card recognition or command filtering? May I ask if there is only one network card on the machine you are experimenting with? But establishing an RDMA connection still requires using TCP. If you have time, I hope you can give some advice. Thank you very much.

[diabloneo]

I have tested on either one nic server or two nics server.

Will you please post following information:

1. ip -4 addr on host
2. The content of cluster.yml

@feeyman

[feeyman]

I have tested on either one nic server or two nics server.

Will you please post following information:

1. ip -4 addr on host
2. The content of cluster.yml

@feeyman
@diabloneo okay, thank you; For some reasons, I have hidden some network segments.

[diabloneo]

After done some research with @feeyman, I found two issues require further investigation:

First is if the ibs7 interface is added to the list returned by IfAddrs::load()? After enabling debug in the config, I still didn't see any log printed in IfAddrs::load() method.

Second one is the network interface named with prefix ib is mapped to IPoIB type in checkNicType() function in listener.cc* file. I have posted a issue to 3FS repo: deepseek-ai/3FS#223

[diabloneo]

After digging the 3FS code and made some experiments, I and @feeyman finally deployed a 3FS cluster on a IB enviroment.

Here are some conclusions:

1. The mgmtd servivce's network_type must be set to IPoIB to let the mgmtd service listening on the IB interface.
2. Other services' (except monitor) network_type still use the value RDMA.
3. The monitor service didn't work on the environment's IB interface. It's can be deployed on the other IP address on ethernet interface.

I published https://github.com/open3fs/m3fs/releases/tag/v0.1.7 to solve this problem. If you want to deploy 3FS on a IB environment, just set networkType: IB in the cluster.yml.