Why checkNicType function treat "ib" prefix as Address:IPoIB ?

[diabloneo]

In the checkNicType function (

3FS/src/common/net/Listener.cc

Line 33 in dfadd59

static bool checkNicType(std::string_view nic, Address::Type type) {

)

Nic name starts with ib has been mapped to IPoIB address type:

static bool checkNicType(std::string_view nic, Address::Type type) {
  switch (type) {
    case Address::TCP:
      return nic.starts_with("en") || nic.starts_with("eth") || nic.starts_with("bond") || nic.starts_with("xgbe");
    case Address::IPoIB:
      return nic.starts_with("ib");
    case Address::RDMA:
      return nic.starts_with("en") || nic.starts_with("eth") || nic.starts_with("bond") || nic.starts_with("xgbe");
    case Address::LOCAL:
      return nic.starts_with("lo");
    default:
      return false;
  }
}

This condition causes the nic named like "ib0" to be skipped in Listener::setup() for RDMA group:

  for (auto [name, addr] : nics.value()) {
    if (addr.up && (filters.empty() || filters.count(name) != 0) && checkNicType(name, networkType_)) {
      addressList_.push_back(Address{addr.ip.toLong(),
                                     config_.listen_port(),
                                     networkType_ == Address::LOCAL ? Address::TCP : networkType_});
    }
  }

Is there any reason for doing this?

[diabloneo]

#156

I found a similar issue.

Does this mean that: if I want to use ib nic, I have to setup ipoib, then use IPoIB:// prefix in the configuration?

[SF-Zhou]

Establishing an RDMA connection actually requires both a TCP network card and an RDMA network card, and RDMA does not need to be in IPoIB mode.

[diabloneo]

I solved the problem to a certain extent. I just quote the conclusion here:

After digging the 3FS code and made some experiments, I and @feeyman finally deployed a 3FS cluster on a IB enviroment.

Here are some conclusions:

The mgmtd servivce's network_type must be set to IPoIB to let the mgmtd service listening on the IB interface.
Other services' (except monitor) network_type still use the value RDMA.
The monitor service didn't work on the environment's IB interface. It's can be deployed on the other IP address on ethernet interface.
I published https://github.com/open3fs/m3fs/releases/tag/v0.1.7 to solve this problem. If you want to deploy 3FS on a IB environment, just set networkType: IB in the cluster.yml.

Visit open3fs/m3fs#88 for more information.