Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

monitor_collector_main start failed, throws address not mapped to object error #183

Open
iamazy opened this issue Mar 17, 2025 · 1 comment
Open

monitor_collector_main start failed, throws address not mapped to object error #183

iamazy opened this issue Mar 17, 2025 · 1 comment

Comments

@iamazy
Copy link

iamazy commented Mar 17, 2025 •
edited
Loading

OS: openEuler 22.10
kernel version: 5.10
arch: aarch64

error message is

[2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_bond_0 port 1 ==> bond1 (Up)
[2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] mlx5_bond_1 port 1 ==> bond2 (Up)
[2025-03-19T15:05:43.726121180+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_0 => bond1
[2025-03-19T15:05:43.726128590+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_1 => bond2
[2025-03-19T15:05:43.727988730+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond2.2295, addr 10.155.10.171/27, subnet 10.155.10.160/27, up true
[2025-03-19T15:05:43.728012760+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond1.2288, addr 10.153.16.171/27, subnet 10.153.16.160/27, up true
[2025-03-19T15:05:43.728027980+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond0.1318, addr 10.151.18.23/24, subnet 10.151.18.0/24, up true
[2025-03-19T15:05:43.728046960+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of eno1, addr 10.153.16.171/24, subnet 10.153.16.0/24, up true
[2025-03-19T15:05:43.731316560+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_0:1 -> bond1 not found, maybe running in container!
[2025-03-19T15:05:43.731346210+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_0:1 can't set zone by IP, fallback to UNKNOWN
[2025-03-19T15:05:43.731444940+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_0 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:99:10:ab
[2025-03-19T15:05:43.731456520+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_0, id 0, 1 available ports
[2025-03-19T15:05:43.734180830+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_1:1 -> bond2 not found, maybe running in container!
[2025-03-19T15:05:43.734192510+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_1:1 can't set zone by IP, fallback to UNKNOWN
[2025-03-19T15:05:43.734269090+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_1 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:9b:a:ab
[2025-03-19T15:05:43.734278690+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_1, id 1, 1 available ports
[2025-03-19T15:05:43.735039780+08:00 IBManager:927324 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] Folly log json configure: {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   "categories": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     ".": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "level": "INFO",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "inherit": true,
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "propagate": "NONE",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "handlers": [
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "normal",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "err",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "fatal",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "debug"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       ]
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   "handlers": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "normal": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "path": "/var/log/3fs/monitor_collector_main.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "async": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "debug": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "path": "/var/log/3fs/monitor_collector_main-debug.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "async": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "err": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "level": "ERR",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "path": "/var/log/3fs/monitor_collector_main-err.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "async": "false",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "fatal": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "stream",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "level": "FATAL",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "stream": "stderr"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735312910+08:00 monitor_collect:927195 OnePhaseApplication.h:87 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal","err","fatal","debug"]}},"handlers":{"normal":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"debug":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main-debug.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"err":{"type":"file","options":{"level":"ERR","path":"/var/log/3fs/monitor_collector_main-err.log","async":"false","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"fatal":{"type":"stream","options":{"level":"FATAL","stream":"stderr"}}}}
*** Aborted at 1742367943 (Unix time, try 'date -d @1742367943') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 927195 (pthread TID 0xffffb3914020) (linux TID 927195) (code: address not mapped to object), stack trace: ***
Segmentation fault (core dumped)

stack trace is

#0  0x0000000000000000 in ?? ()
#1  0x00000000003e0080 in folly::goodMallocSize (minSize=64) at /home/nfsdeplyer/3FS/third_party/folly/folly/memory/Malloc.h:390
#2  folly::fbstring_core<char>::reserveSmall (this=this@entry=0xffffffffa688, minCapacity=<optimized out>, disableSSO=<optimized out>)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:836
#3  0x00000000003dffa4 in folly::fbstring_core<char>::expandNoinit (this=this@entry=0xffffffffa688, delta=<optimized out>, 
    expGrowth=<optimized out>, disableSSO=false) at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:872
#4  0x00000000003dfe18 in folly::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, folly::fbstring_core<char> >::append (
    this=0xffffffffa688, s=0xffffffff95d0 "_ZN5folly10symbolizer13getStackTraceEPmm", n=40)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:1834
#5  0x0000000000613240 in folly::symbolizer::SymbolizePrinter::print (this=this@entry=0xffffffffa678, frame=...)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:100
#6  0x000000000061372c in folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frame=...)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:151
#7  folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frames=<optimized out>, frameCount=<optimized out>)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:179
#8  0x00000000006105e0 in folly::symbolizer::SymbolizePrinter::println<100ul> (this=0xffffffffa678, fa=..., skip=0)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.h:80
#9  folly::symbolizer::(anonymous namespace)::getStackTraceStrImpl<100ul, bool (*)(folly::symbolizer::FrameArray<100ul>&)> (func=<optimized out>)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/Symbolizer.cpp:440
#10 0x0000000000433b18 in hf3fs::net::ThreadPoolGroup::ThreadPoolGroup (this=0x8f2280, name="Svr", config=...)
    at /home/nfsdeplyer/3FS/src/common/net/ThreadPoolGroup.h:33
#11 0x000000000045b354 in hf3fs::net::Server::Server (this=0x8f2240, config=...) at /home/nfsdeplyer/3FS/src/common/net/Server.cc:13
#12 0x00000000003d1fa8 in hf3fs::monitor::MonitorCollectorServer::MonitorCollectorServer (this=0x8f2240, config=...)
    at /home/nfsdeplyer/3FS/src/monitor_collector/service/MonitorCollectorServer.cc:8
#13 0x00000000003cfe7c in std::make_unique<hf3fs::monitor::MonitorCollectorServer, hf3fs::monitor::MonitorCollectorServer::Config&> (__args=...)
    at /usr/lib/gcc/aarch64-linux-gnu/10.3.1/../../../../include/c++/10.3.1/bits/unique_ptr.h:962
#14 hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::initApplication (
    this=0x7aaed0 <hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::instance()::app>)
    at /home/nfsdeplyer/3FS/src/common/app/OnePhaseApplication.h:99
#15 0x00000000003f1c7c in hf3fs::ApplicationBase::run (
    this=0x7aaed0 <hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::instance()::app>, argc=1, argv=0xfffffffff5c8)
    at /home/nfsdeplyer/3FS/src/common/app/ApplicationBase.cc:65
#16 0x0000fffff71a3000 in ?? () from /usr/lib64/libc.so.6
#17 0x0000fffff71a30d8 in __libc_start_main () from /usr/lib64/libc.so.6
#18 0x000000000032e9f0 in _start ()

maybe it's a folly exception, but I don't know how to address this issue.

@SF-Zhou please take a look at this issue :)
@iamazy iamazy changed the title monitor_collector_main start failed, throws RoCE v2 GID not found error monitor_collector_main start failed, throws address not mapped to object error Mar 19, 2025
@iamazy
Copy link
Author

iamazy commented Mar 19, 2025

I disable libunwind in folly and finally start monitor_collector_main success

cmake -S . -B build -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@iamazy