Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Boostd crashed after startup, we imported more than 6000 offline orders at once #1996

Open
6 of 12 tasks
zelin44913 opened this issue Feb 11, 2025 · 2 comments
Open
6 of 12 tasks
Labels

Comments

@zelin44913
Copy link

zelin44913 commented Feb 11, 2025

Checklist

  • This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
  • This is not a new feature request. If it is, please file a feature request instead.
  • This is not an enhancement request. If it is, please file a improvement suggestion instead.
  • I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
  • I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to boost.

Boost component

  • boost daemon - storage providers
  • boost client
  • boost UI
  • boost data-transfer
  • boost index-provider
  • Other

Boost Version

boostd version 2.4.0+mainnet+git.3965a2c

Describe the Bug

Boostd crashed after startup, we imported more than 6000 offline orders at once

We repeatedly tried to start the process, and it crashed every time, including restarting the server, updating the system, etc.

Logging Information

{"level":"info","ts":"2025-02-11T14:16:49.677+0800","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"add piece called","id":"3e419fbe-41ea-42b0-9793-e3faf40c7435"}
{"level":"info","ts":"2025-02-11T14:16:49.678+0800","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"add piece called","id":"6ed2437d-2098-4e61-9aaa-e0f14c00f14b"}
{"level":"info","ts":"2025-02-11T14:16:49.678+0800","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"add piece called","id":"18ba451f-4ea2-40ca-ad82-9095772a8b2f"}
{"level":"info","ts":"2025-02-11T14:16:49.678+0800","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"add piece called","id":"b5dbdac5-5ccd-4910-81fd-960b06596d78"}
unexpected fault address 0x7f0697dbf000
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f0697dbf000 pc=0x5c89bc]

goroutine 166504 gp=0xc0004eec40 m=26 mp=0xc001601008 [running]:
runtime.throw({0x56b2d17?, 0x19b7380?})
	/usr/local/go/src/runtime/panic.go:1023 +0x5c fp=0xc00de72f20 sp=0xc00de72ef0 pc=0x58c31c
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:878 +0x116 fp=0xc00de72f80 sp=0xc00de72f20 pc=0x5a4f56
runtime.memmove()
	/usr/local/go/src/runtime/memmove_amd64.s:149 +0xfc fp=0xc00de72f88 sp=0xc00de72f80 pc=0x5c89bc
golang.org/x/exp/mmap.(*ReaderAt).ReadAt(0x5800000000?, {0xc007612680?, 0x1, 0xc8f3480?}, 0x55c07a?)
	/root/go/pkg/mod/golang.org/x/exp@v0.0.0-20240904232852-e7e105dedf7e/mmap/mmap_unix.go:75 +0xc5 fp=0xc00de72fd8 sp=0xc00de72f88 pc=0x1ca7725
github.com/ipld/go-car/v2/internal/io.(*offsetReadSeeker).Read(0xc007612660, {0xc007612680?, 0xc001466360?, 0xc00de73068?})
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/internal/io/offset_read_seeker.go:59 +0x2c fp=0xc00de73010 sp=0xc00de72fd8 pc=0x1c8bf0c
github.com/ipld/go-car/v2/internal/io.(*offsetReadSeeker).ReadByte(0xc007612660)
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/internal/io/offset_read_seeker.go:82 +0x26 fp=0xc00de73040 sp=0xc00de73010 pc=0x1c8c0a6
github.com/multiformats/go-varint.ReadUvarint({0x7f1e20501148, 0xc007612660})
	/root/go/pkg/mod/github.com/multiformats/go-varint@v0.0.7/varint.go:80 +0x51 fp=0xc00de73088 sp=0xc00de73040 pc=0x66acf1
github.com/ipld/go-car/v2/internal/carv1/util.LdReadSize({0x7f1e20501128, 0xc007612660}, 0x0, 0x2000000)
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/internal/carv1/util/util.go:69 +0x9d fp=0xc00de730c0 sp=0xc00de73088 pc=0x1c8cd7d
github.com/ipld/go-car/v2/internal/carv1/util.LdRead({0x7f1e20501128, 0xc007612660}, 0x0?, 0xc00de73150?)
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/internal/carv1/util/util.go:87 +0x25 fp=0xc00de73110 sp=0xc00de730c0 pc=0x1c8cee5
github.com/ipld/go-car/v2/internal/carv1.ReadHeader({0x7f1e20501128?, 0xc007612660?}, 0x401?)
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/internal/carv1/car.go:52 +0x2c fp=0xc00de73190 sp=0xc00de73110 pc=0x1c8d02c
github.com/ipld/go-car/v2.ReadVersion({0x7f1e20501128, 0xc007612660}, {0x0?, 0x0?, 0x800?})
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/reader.go:363 +0x90 fp=0xc00de73278 sp=0xc00de73190 pc=0x1caad90
github.com/ipld/go-car/v2.NewReader({0x87d41a0, 0xc005067068}, {0x0, 0x0, 0x0})
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/reader.go:63 +0x245 fp=0xc00de73398 sp=0xc00de73278 pc=0x1caa685
github.com/ipld/go-car/v2.OpenReader({0xc004a1baa0?, 0x5d?}, {0x0, 0x0, 0x0})
	/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.13.1/reader.go:37 +0x53 fp=0xc00de733d8 sp=0xc00de73398 pc=0x1caa393
github.com/filecoin-project/boost/storagemarket.openReader({0xc004a1baa0, 0x5d}, 0x7f0000000)
	/code/boost-2.4.0/storagemarket/deal_execution.go:581 +0x109 fp=0xc00de73480 sp=0xc00de733d8 pc=0x2d08989
github.com/filecoin-project/boost/storagemarket.(*Provider).addPiece(0xc00161a288, {0x87f6f28, 0xc001458320}, {0x87e4850, 0xc005e11440}, 0xc004a52fc0)
	/code/boost-2.4.0/storagemarket/deal_execution.go:538 +0x106 fp=0xc00de73980 sp=0xc00de73480 pc=0x2d082c6
github.com/filecoin-project/boost/storagemarket.(*Provider).execDealUptoAddPiece(0xc00161a288, {0x87f6f28, 0xc001458320}, 0xc004a52fc0, 0xc004e268c0)
	/code/boost-2.4.0/storagemarket/deal_execution.go:215 +0x568 fp=0xc00de73b40 sp=0xc00de73980 pc=0x2d05708
github.com/filecoin-project/boost/storagemarket.(*Provider).execDeal(0xc00161a288, 0xc004a52fc0, 0xc004e268c0)
	/code/boost-2.4.0/storagemarket/deal_execution.go:123 +0x446 fp=0xc00de73ca8 sp=0xc00de73b40 pc=0x2d04ce6
github.com/filecoin-project/boost/storagemarket.(*Provider).runDeal(0xc00161a288, 0xc004a52fc0, 0xc004e268c0)
	/code/boost-2.4.0/storagemarket/deal_execution.go:54 +0x1f9 fp=0xc00de73f38 sp=0xc00de73ca8 pc=0x2d04459
github.com/filecoin-project/boost/storagemarket.(*Provider).startDealThread.func1()
	/code/boost-2.4.0/storagemarket/provider_loop.go:553 +0xa5 fp=0xc00de73fe0 sp=0xc00de73f38 pc=0x2d1e545
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00de73fe8 sp=0xc00de73fe0 pc=0x5c7ae1
created by github.com/filecoin-project/boost/storagemarket.(*Provider).startDealThread in goroutine 387
	/code/boost-2.4.0/storagemarket/provider_loop.go:546 +0xb2

goroutine 1 gp=0xc0000061c0 m=nil [chan receive]:
runtime.gopark(0xc001311458?, 0x1465540?, 0xa0?, 0xa0?, 0x1465620?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0041b13d8 sp=0xc0041b13b8 pc=0x58f36e
runtime.chanrecv(0xc0045b17a0, 0x0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x3bf fp=0xc0041b1450 sp=0xc0041b13d8 pc=0x5583bf
runtime.chanrecv1(0xc0007e9ec0?, 0xc00acdd0b0?)
	/usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0041b1478 sp=0xc0041b1450 pc=0x557fd2
main.init.func45(0xc000603480)
	/code/boost-2.4.0/cmd/boostd/run.go:183 +0x19a5 fp=0xc0041b1950 sp=0xc0041b1478 pc=0x2ea19e5
github.com/urfave/cli/v2.(*Command).Run(0xc6b4e80, 0xc000603480, {0xc0007089c0, 0x1, 0x1})
	/root/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:274 +0x93f fp=0xc0041b1bd8 sp=0xc0041b1950 pc=0x140cd1f
github.com/urfave/cli/v2.(*Command).Run(0xc0006be9a0, 0xc000603280, {0xc0000520a0, 0x2, 0x2})
	/root/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0xb7d fp=0xc0041b1e60 sp=0xc0041b1bd8 pc=0x140cf5d
github.com/urfave/cli/v2.(*App).RunContext(0xc000466780, {0x87f6c88, 0xc8f3480}, {0xc0000520a0, 0x2, 0x2})
	/root/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332 +0x566 fp=0xc0041b1ec0 sp=0xc0041b1e60 pc=0x1409606
github.com/urfave/cli/v2.(*App).Run(...)
	/root/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:309
main.main()
	/code/boost-2.4.0/cmd/boostd/main.go:53 +0x4b5 fp=0xc0041b1f50 sp=0xc0041b1ec0 pc=0x2ea57d5
runtime.main()
	/usr/local/go/src/runtime/proc.go:271 +0x29d fp=0xc0041b1fe0 sp=0xc0041b1f50 pc=0x58ef1d
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0041b1fe8 sp=0xc0041b1fe0 pc=0x5c7ae1

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c6fa8 sp=0xc0000c6f88 pc=0x58f36e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:326 +0xb3 fp=0xc0000c6fe0 sp=0xc0000c6fa8 pc=0x58f1d3
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c6fe8 sp=0xc0000c6fe0 pc=0x5c7ae1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c7780 sp=0xc0000c7760 pc=0x58f36e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc0000ee000)
	/usr/local/go/src/runtime/mgcsweep.go:318 +0xdf fp=0xc0000c77c8 sp=0xc0000c7780 pc=0x578a9f
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc0000c77e0 sp=0xc0000c77c8 pc=0x56d385
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c77e8 sp=0xc0000c77e0 pc=0x5c7ae1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000007340 m=nil [runnable]:
runtime.gopark(0x10000?, 0x37cae0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c7f78 sp=0xc0000c7f58 pc=0x58f36e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0xc88a940)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc0000c7fa8 sp=0xc0000c7f78 pc=0x576429
runtime.bgscavenge(0xc0000ee000)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc0000c7fc8 sp=0xc0000c7fa8 pc=0x5769d9
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc0000c7fe0 sp=0xc0000c7fc8 pc=0x56d325
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c7fe8 sp=0xc0000c7fe0 pc=0x5c7ae1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0xc00488e318?, 0x30?, 0x40?, 0x1000000010?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c6620 sp=0xc0000c6600 pc=0x58f36e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc0000c67e0 sp=0xc0000c6620 pc=0x56c347
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c67e8 sp=0xc0000c67e0 pc=0x5c7ae1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc000007dc0 m=nil [select]:
runtime.gopark(0xc0000c8760?, 0x2?, 0x0?, 0x0?, 0xc0000c871c?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000d6db0 sp=0xc0000d6d90 pc=0x58f36e
runtime.selectgo(0xc0000d6f60, 0xc0000c8718, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0000d6ed0 sp=0xc0000d6db0 pc=0x5a11e5
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc00018a870)
	/root/go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:71 +0x10e fp=0xc0000d6fc8 sp=0xc0000d6ed0 pc=0xa479ce
github.com/ipfs/go-log/writer.NewMirrorWriter.gowrap1()
	/root/go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:36 +0x25 fp=0xc0000d6fe0 sp=0xc0000d6fc8 pc=0xa47805
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000d6fe8 sp=0xc0000d6fe0 pc=0x5c7ae1
created by github.com/ipfs/go-log/writer.NewMirrorWriter in goroutine 1
	/root/go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:36 +0xbf

goroutine 7 gp=0xc0002f28c0 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e42f81d?, 0x3?, 0x53?, 0x9e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c8f50 sp=0xc0000c8f30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c8fe0 sp=0xc0000c8f50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c8fe8 sp=0xc0000c8fe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 18 gp=0xc000102700 m=nil [GC worker (idle)]:
runtime.gopark(0x843af50?, 0xc000118020?, 0x1a?, 0xa?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c8750 sp=0xc0000c8730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c87e0 sp=0xc0000c8750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c87e8 sp=0xc0000c87e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 19 gp=0xc0001028c0 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0x7c?, 0x35?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c2750 sp=0xc0000c2730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c27e0 sp=0xc0000c2750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c27e8 sp=0xc0000c27e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 20 gp=0xc000102a80 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0xd7?, 0x9?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c2f50 sp=0xc0000c2f30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c2fe0 sp=0xc0000c2f50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c2fe8 sp=0xc0000c2fe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc000438000 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e436024?, 0x3?, 0xef?, 0xf7?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043e750 sp=0xc00043e730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043e7e0 sp=0xc00043e750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043e7e8 sp=0xc00043e7e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc0004381c0 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0x29?, 0x5d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043ef50 sp=0xc00043ef30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043efe0 sp=0xc00043ef50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043efe8 sp=0xc00043efe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 8 gp=0xc0002f2a80 m=nil [GC worker (idle)]:
runtime.gopark(0x843af50?, 0xc000057760?, 0x1a?, 0xa?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c9750 sp=0xc0000c9730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c97e0 sp=0xc0000c9750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c97e8 sp=0xc0000c97e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 9 gp=0xc0002f2c40 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0x77?, 0xc7?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c9f50 sp=0xc0000c9f30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c9fe0 sp=0xc0000c9f50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c9fe8 sp=0xc0000c9fe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 36 gp=0xc000438380 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0xc5?, 0xa7?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043f750 sp=0xc00043f730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043f7e0 sp=0xc00043f750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043f7e8 sp=0xc00043f7e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 10 gp=0xc0002f2e00 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0x21?, 0x37?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043a750 sp=0xc00043a730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043a7e0 sp=0xc00043a750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043a7e8 sp=0xc00043a7e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 37 gp=0xc000438540 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e4600bd?, 0x1?, 0x33?, 0x96?, 0xc000445c40?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043ff50 sp=0xc00043ff30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043ffe0 sp=0xc00043ff50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043ffe8 sp=0xc00043ffe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 38 gp=0xc000438700 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e435fd4?, 0x3?, 0xbf?, 0xde?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000440750 sp=0xc000440730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004407e0 sp=0xc000440750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004407e8 sp=0xc0004407e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 21 gp=0xc000102c40 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0xf4?, 0xbf?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c3750 sp=0xc0000c3730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c37e0 sp=0xc0000c3750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c37e8 sp=0xc0000c37e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 39 gp=0xc0004388c0 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e42f7cd?, 0x3?, 0x98?, 0x35?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000440f50 sp=0xc000440f30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000440fe0 sp=0xc000440f50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000440fe8 sp=0xc000440fe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 40 gp=0xc000438a80 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e467648?, 0x1?, 0x70?, 0x49?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000441750 sp=0xc000441730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004417e0 sp=0xc000441750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004417e8 sp=0xc0004417e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 11 gp=0xc0002f2fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27d9dc205?, 0x3?, 0x4?, 0xb?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043af50 sp=0xc00043af30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043afe0 sp=0xc00043af50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043afe8 sp=0xc00043afe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 22 gp=0xc000102e00 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e446ffc?, 0x3?, 0xa5?, 0x4d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c3f50 sp=0xc0000c3f30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c3fe0 sp=0xc0000c3f50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c3fe8 sp=0xc0000c3fe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 12 gp=0xc0002f3180 m=nil [GC worker (idle)]:
runtime.gopark(0x8a26f99d52c?, 0x3?, 0x99?, 0x4d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043b750 sp=0xc00043b730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043b7e0 sp=0xc00043b750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043b7e8 sp=0xc00043b7e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 23 gp=0xc000102fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e42f8db?, 0x3?, 0xf0?, 0xf7?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c4750 sp=0xc0000c4730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c47e0 sp=0xc0000c4750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c47e8 sp=0xc0000c47e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 24 gp=0xc000103180 m=nil [GC worker (idle)]:
runtime.gopark(0x8a27e42fad9?, 0x3?, 0xda?, 0x21?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c4f50 sp=0xc0000c4f30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c4fe0 sp=0xc0000c4f50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c4fe8 sp=0xc0000c4fe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 13 gp=0xc0002f3340 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x1?, 0xc4?, 0x52?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00043bf50 sp=0xc00043bf30 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00043bfe0 sp=0xc00043bf50 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00043bfe8 sp=0xc00043bfe0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 25 gp=0xc000103340 m=nil [GC worker (idle)]:
runtime.gopark(0xc8f7440?, 0x3?, 0x9e?, 0x4d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000c5750 sp=0xc0000c5730 pc=0x58f36e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000c57e0 sp=0xc0000c5750 pc=0x56f465
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000c57e8 sp=0xc0000c57e0 pc=0x5c7ae1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

Repo Steps

  1. Run 'export $(lotus auth api-info --perm=admin) && nohup boostd run >> /data/logs/boostd.log 2>&1 &'
  2. Do '...'
  3. See error '...'
    ...
@zelin44913
Copy link
Author

Updated to 2.4.1, the problem still exists

@LexLuthr
Copy link
Collaborator

Not that surprising. 6000 is simply too much for Boost architecture. Are you using remote commP?
Are these full sector deals? 6000 * 32 GiB = 192000 GiB = 187.5 TiB?

I am surprised you are seeing a crash and not OOO. Can you check dmesg and system logs as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Development

No branches or pull requests

2 participants