SIGTRAP when calling get_all_fs()

[chbuescher]

My program uses github.com/shirou/gopsutil/v3/disk for getting disk usage data once a minute, which for aix calls perfstat.FileSystemStat(). About once a day the program dies with SIGTRAP and is restartet by the system resource controller.
When analyzing the /core file i get the following stack trace:

_internal_error(??, ??, ??) at 0xd057df80
_cleanup_push@AF83_37(??, ??, ??, ??) at 0xd058300c
pthread_cleanup_push(??, ??) at 0xd05815ec
__modinit_lock(??) at 0xd0129974
__modfini() at 0xd0196fc8
exit(??) at 0xd018b368
runtime.throw(??, ??) at 0x10054acc
runtime.sigpanic() at 0x1006e51c
runtime.sigtrampgo(??, ??, ??) at 0x10086614
runtime.sigtramp(??, ??, ??) at 0x108168a0
strncpy() at 0x10306768
getfsinfo(??, ??, ??, ??, ??, ??, ??) at 0x103f7f30
get_all_fs(??) at 0x103f80c4
_cgo_022023b15f11_Cfunc_get_all_fs(??) at 0x103f7178
github_0com_1power_x2ddevops_1perfstat.FileSystemStat() at 0x103dc334
github_0com_1shirou_1gopsutil_1v3_1disk.UsageWithContext(??, ??, ??, ??) at 0x10622278
github_0com_1shirou_1gopsutil_1v3_1disk.Usage(??, ??) at 0x1062264c
main.startMetrics(??) at 0x10801438
runtime.kickoff() at 0x10092e90

After some Google-ing i've found https://peeterjoot.wordpress.com/2010/05/05/fun-with-aix-nested-signal-handlers/.
Maybe the SIGTRAP occurs because devname or fsname sometimes is NULL or calloc failed in getfsinfo()

[aklyachkin]

Can you make some local changes into the code and check if it works for you on your machine?

File c_helpers.c in function getfsinfo() (line 102 and below).

After line 111:

if (fsname == NULL) {
  return 1
}

if (devname == NULL) {
  return 1
}

After line 122 (calloc() calls):

if (fs->devname == NULL) {
  return 1
}
if (fs->fsname == NULL) {
  return 1
}

[chbuescher]

Yes, i'll test it. Thanks !

[chbuescher]

The good: Your fix works.
The bad: It seems to be a memory issue with gccgo. My program keeps allocating memory and finally i get this error:
malloc failed: Not enough space
Thanks for your help !

[aklyachkin]

OK, thank for you for the feedback. I have my own perfstat-based agent the whole time running and got no problems. But you mentioned the difference - mine is compiled using Golang compiler and not gcc-go, even if it uses gcc as backend for C bindings.

[aklyachkin]

Can you check it with Golang and POWER8 (or newer), if the same problem occurs? I will try meanwhile to recompile my agent with gcc-go and check if I get the problem.

[aklyachkin]

I was still unable to bring my part under gccgo, but can you please do the following:

• stop your program
• enable the full core:

chdev -l sys0 -a fullcore=true

• export the variable:

export MALLOCDEBUG=log:extended,stack_depth:20

• start your program again
• find the PID of your program and run:

gencore <PID> /tmp/core.first

• after some time run it again:

gencore <PID> /tmp/core.second

• then start dbx and produce the list of memory allocations:

dbx </path/to/program> /tmp/core.first
malloc allocation >/tmp/alloc.first
quit

dbx </path/to/program> /tmp/core.second
malloc allocation >/tmp/alloc.second
quit

Then please send me both alloc.first and alloc.second. I will compare them with what I see on my systems.

[chbuescher]

dbx ./fropTest /tmp/core.first
Type 'help' for help.
[using memory image in /tmp/core.first]
reading symbolic information ...internal error: assertion failed at line 6704 in file object.c
internal error: assertion failed at line 6685 in file object.c
internal error: assertion failed at line 6685 in file object.c
internal error: assertion failed at line 6685 in file object.c
...
Signal 0 in sem_wait at 0xd031f79c ($t1)
alloc.third.gz
alloc.first.gz

0xd031f79c (sem_wait+0xdc) 80410014 lwz r2,0x14(r1)
(dbx)

[chbuescher]

When compiled with golang & Power8 my program still was restarted by srcmstr, but no core file was generated.
I added code to record the heap size, and it keeps more or less constant.
I don't think it's your code that has a memory leak.
My program uses shirou/gopsutil for platform independent performance metrics, and for aix gopsutil uses your perfstat lib.
Seems to be an gccgo issue, but we still have to use gccgo for our Power7 machines.

[aklyachkin]

Do you use gcc8? If yes, can you compile your code using gcc10? I compiled my code with gccgo10 and don't see any memory problems. But I have only rootvg on my test system.

[chbuescher]

I compiled my code with go version go1.16.5 gccgo (GCC) 11.3.0 aix/ppc - self compiled with the same options as offical gcc 8.3.0.

[aklyachkin]

I got a panic today in the code compiled with gccgo 10

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x33 addr=0xb8222e7 pc=0x900000000047968]

goroutine 19420 [syscall]:
runtime.dopanic_m
        ../../../../gcc-10.3.0/libgo/go/runtime/panic.go:1208

Now I have to understand, if it has something to do with the perfstat library, but I'll do it tomorrow.