Server unexpectedly crashed. #163
Comments
|
@Hobzar what version of the Merlin server and agent are you using? What operating system is the server running on? Did you build the server yourself or use the pre compiled one? |
|
I created http2 listener, but there was error like below, then server crashed. |
|
@Hobzar This typically happens when the system time is not synchronized to the same source. This is common for VMs that don't use NTP. The best option is to get the time synchronized between the server and the place where the agents are running. Alternatively, you can increase the |
|
Thank you. After I set JWTLeeway, the error did not occur again. But the server still crashes. |
|
@Hobzar Is that ALL of the output during the crash? Can you run the server with the |
|
@Hobzar For sake of being thorough, can you share with me the output from running the server with the |
|
Kali host is "Linux ser483446610093 6.10.9-amd64 #1 SMP PREEMPT_DYNAMIC Kali 6.10.9-1kali1 (2024-08-09) x86_64 GNU/Linux" goroutine 35595 [running]: goroutine 1 [IO wait, 16 minutes]: goroutine 19 [chan receive, 16 minutes]: goroutine 20 [chan receive, 2 minutes]: goroutine 10 [chan receive, 2 minutes]: goroutine 6 [select]: goroutine 7 [select, 16 minutes]: goroutine 8 [IO wait]: goroutine 16 [select]: goroutine 15 [semacquire, 16 minutes]: goroutine 23 [IO wait, 2 minutes]: goroutine 28 [select]: goroutine 34 [select]: goroutine 41 [select]: goroutine 25 [IO wait]: goroutine 40 [select]: goroutine 31 [IO wait]: goroutine 255 [IO wait]: goroutine 51 [select]: goroutine 52 [select]: goroutine 120 [select]: goroutine 42 [IO wait]: goroutine 108 [IO wait]: goroutine 48 [IO wait]: goroutine 70 [select]: goroutine 57 [IO wait]: goroutine 105 [select]: goroutine 73 [IO wait]: goroutine 279 [select]: goroutine 88 [IO wait]: goroutine 250 [IO wait]: goroutine 277 [select]: goroutine 132 [select]: goroutine 177 [select]: goroutine 135 [IO wait]: goroutine 247 [select]: goroutine 196 [IO wait]: goroutine 3115 [select]: goroutine 267 [IO wait]: goroutine 3255 [IO wait]: goroutine 209 [select]: goroutine 210 [select]: goroutine 217 [select]: goroutine 189 [IO wait]: goroutine 228 [IO wait]: goroutine 220 [IO wait]: goroutine 223 [select]: goroutine 343 [select]: goroutine 290 [select]: goroutine 293 [IO wait]: goroutine 330 [select]: goroutine 288 [IO wait]: goroutine 310 [IO wait]: goroutine 300 [select]: goroutine 301 [select]: goroutine 342 [select]: goroutine 304 [IO wait]: goroutine 327 [IO wait]: goroutine 416 [IO wait]: goroutine 344 [select]: goroutine 334 [IO wait]: goroutine 374 [IO wait]: goroutine 370 [IO wait]: goroutine 355 [IO wait]: goroutine 510 [IO wait]: goroutine 361 [IO wait]: goroutine 492 [select]: goroutine 928 [IO wait]: goroutine 580 [IO wait]: goroutine 389 [select]: goroutine 452 [select]: goroutine 467 [select]: goroutine 491 [select]: goroutine 429 [IO wait]: goroutine 482 [IO wait]: goroutine 845 [select]: goroutine 505 [IO wait]: goroutine 549 [select]: goroutine 550 [select]: goroutine 963 [IO wait]: goroutine 561 [IO wait]: goroutine 1140 [select]: goroutine 1547 [IO wait]: goroutine 1565 [select]: goroutine 854 [IO wait]: goroutine 878 [IO wait]: goroutine 685 [IO wait]: goroutine 920 [select]: goroutine 659 [select]: goroutine 873 [select]: goroutine 765 [IO wait]: goroutine 730 [select]: goroutine 747 [select]: goroutine 754 [IO wait]: goroutine 1162 [IO wait]: goroutine 949 [select]: goroutine 28321 [select]: goroutine 1594 [select]: goroutine 1030 [select]: goroutine 1038 [IO wait]: goroutine 1057 [select]: goroutine 1088 [IO wait]: goroutine 1506 [select]: goroutine 1192 [select]: goroutine 1244 [IO wait]: goroutine 1208 [IO wait]: goroutine 9276 [IO wait]: goroutine 1504 [IO wait]: goroutine 1511 [IO wait]: goroutine 1240 [select]: goroutine 1371 [IO wait]: goroutine 1520 [select]: goroutine 1372 [select]: goroutine 1357 [select]: goroutine 1400 [IO wait]: goroutine 1472 [select]: goroutine 9120 [select]: goroutine 8341 [IO wait]: goroutine 1477 [select]: goroutine 1612 [IO wait]: goroutine 1468 [IO wait]: goroutine 1577 [IO wait]: goroutine 9127 [IO wait]: goroutine 8338 [select]: goroutine 19713 [select]: goroutine 6971 [select]: goroutine 7111 [select]: goroutine 2918 [IO wait]: goroutine 2914 [select]: goroutine 3187 [select]: goroutine 6985 [IO wait]: goroutine 3118 [IO wait]: goroutine 3198 [IO wait]: goroutine 3226 [select]: goroutine 28289 [IO wait]: goroutine 4024 [select]: goroutine 4805 [select]: goroutine 4016 [IO wait]: goroutine 4822 [IO wait]: goroutine 7114 [IO wait]: goroutine 9268 [select]: goroutine 30504 [IO wait]: goroutine 11133 [select]: goroutine 11235 [select]: goroutine 11023 [select]: goroutine 11223 [IO wait]: goroutine 11042 [IO wait]: goroutine 11096 [IO wait]: goroutine 11166 [IO wait]: goroutine 11076 [select]: goroutine 30484 [select]: goroutine 35578 [runnable]: goroutine 19758 [IO wait]: goroutine 35596 [runnable]: |
|
How many agents can connect to one listener? Is there limit? |
|
@Hobzar Thank you for sharing that crash information. It had exactly what I was looking for. No, there is not a limit on how many agents can connect to one listener. The amount of memory & CPU the host has could be factor. How many agents were you checking in? I tried with 250 agents and things seemed stable. Can you try to run this fix and see if the problem persists? |
|
When I run "sudo go run main.go", error occured. |
|
@Hobzar it looks like your hard drive is full? |
|
Free space is enough. |
|
Where is your |
|
Thank you. /tmp directory was full. I'm running the server. I'll contact you if any errors occur later. |
|
Hi, Ne0nd0g. The server never crashes. But I have one more request. |
Sometimes, merlin server crashed.
If I create http2 listener, then I connect agents. But after a few minutes, it crashed.
Output was like this.
goroutine 4791 [IO wait]:
internal/poll.runtime_pollWait(0x7fd7cd0a3698, 0x72)
/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc00036b800?, 0xc0004de600?, 0x0)
/opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00036b800, {0xc0004de600, 0x1300, 0x1300})
/opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc00036b800, {0xc0004de600?, 0xc0004de605?, 0x1e?})
/opt/hostedtoolcache/go/1.21.9/x64/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0004567b0, {0xc0004de600?, 0x0?, 0xc000394fb8?})
/opt/hostedtoolcache/go/1.21.9/x64/src/net/net.go:179 +0x45
crypto/tls.(*atLeastReader).Read(0xc00062cab0, {0xc0004de600?, 0xc00062cab0?, 0x0?})
/opt/hostedtoolcache/go/1.21.9/x64/src/crypto/tls/conn.go:805 +0x3b
bytes.(*Buffer).ReadFrom(0xc0003950a8, {0xd16180, 0xc00062cab0})
/opt/hostedtoolcache/go/1.21.9/x64/src/bytes/buffer.go:211 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000394e00, {0xd16360?, 0xc0004567b0}, 0x1300?)
/opt/hostedtoolcache/go/1.21.9/x64/src/crypto/tls/conn.go:827 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0xc000394e00, 0x0)
/opt/hostedtoolcache/go/1.21.9/x64/src/crypto/tls/conn.go:625 +0x250
crypto/tls.(*Conn).readRecord(...)
/opt/hostedtoolcache/go/1.21.9/x64/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc000394e00, {0xc0006b4f20, 0x9, 0x44db26?})
/opt/hostedtoolcache/go/1.21.9/x64/src/crypto/tls/conn.go:1369 +0x158
io.ReadAtLeast({0x7fd7cd162ac0, 0xc000394e00}, {0xc0006b4f20, 0x9, 0x9}, 0x9)
/opt/hostedtoolcache/go/1.21.9/x64/src/io/io.go:335 +0x90
io.ReadFull(...)
/opt/hostedtoolcache/go/1.21.9/x64/src/io/io.go:354
net/http.http2readFrameHeader({0xc0006b4f20, 0x9, 0x0?}, {0x7fd7cd162ac0?, 0xc000394e00?})
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/h2_bundle.go:1635 +0x65
net/http.(*http2Framer).ReadFrame(0xc0006b4ee0)
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/h2_bundle.go:1899 +0x85
net/http.(*http2serverConn).readFrames(0xc0002f9860)
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/h2_bundle.go:4666 +0x87
created by net/http.(*http2serverConn).serve in goroutine 4752
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/h2_bundle.go:4778 +0x56a
goroutine 3513 [select]:
net/http.(*http2serverConn).serve(0xc0006111e0)
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/h2_bundle.go:4786 +0x88f
net/http.(*http2Server).ServeConn(0xc0001003c0, {0xd20778?, 0xc0000ed880}, 0xc00063bb18)
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/h2_bundle.go:4374 +0xbcc
net/http.http2ConfigureServer.func1(0xc0002a80f0, 0xd20778?, {0xd175a0?, 0xc0006207a0})
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/h2_bundle.go:4164 +0xfc
net/http.(*conn).serve(0xc0006ba240, {0xd1c9a8, 0xc0002721e0})
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/server.go:1917 +0x1213
created by net/http.(*Server).Serve in goroutine 30
/opt/hostedtoolcache/go/1.21.9/x64/src/net/http/server.go:3086 +0x5cb
The text was updated successfully, but these errors were encountered: