sccd memory ballooning

[KenSharp]

Memory usage for sccd balloons and has to be restarted every few days. This isn't ideal for a daemon that should be running continuously.

I don't know what causes the ballooning but it's fairly easy to recreate: just leave sccd running.

$ sccd --version
SCC Core Daemon version v3.2.0.0-0d10c8e58

$ uname -srvmpio
Linux 5.13.0-44-generic #49~20.04.1-Ubuntu SMP Wed May 18 18:44:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

The result being that resources on the server end up starved and the OOM killer kicks in. This can kill the daemon causing corruption of the local files and results in even longer downtime as it needs to re-sync.

[Liquid369]

Hello!

We are working on testing some fixes for memory leaks like such,
We have put up #21 which should help you along this issue some.
Able to test with us for some good data gathering?

[KenSharp]

I tried, after installing/adding/compiling a load of dependencies, but finally fell down at threshold.hpp

./bls/bls.h:21:10: fatal error: bls-dash/threshold.hpp: No such file or directory
 #include <bls-dash/threshold.hpp>

I cannot find a source for that file.

[KenSharp]

Tried again, tried build from the depends, but no matter what flags I add it refuses to find relic_conf.h

./bls-dash/privatekey.hpp:18:10: fatal error: relic_conf.h: No such file or directory
   18 | #include "relic_conf.h"

If someone can tell me how to build this thing...

[Liquid369]

Can you provide me how you are compiling step by step please and I will guide you from there?

[KenSharp]

I can't even get bls-signatures to compile today, I have no idea why - it worked fine yesterday. I give up for now.

[KenSharp]

Okay, first issue is that this version of bls-signatures does not include threshold.hpp, it was removed. This is the only references to bls-signatures I can find with a search. So I checked out an older version which does include this file, but this has problems building relic.

I gave up on that and tried a “depends” build.

$ cd depends
$ make -j16

At this point the build still cannot find its dependencies

$ cd ../src
$ ln -s ../depends/x86_64-pc-linux-gnu/include/bls-dash
$ cd ..
$ make -j16

[...]

/usr/bin/ld: cannot find -lbls-dash

So I grabbed the 1-2.0 archive from depends, built and installed it. This worked. Until..

/usr/bin/ld: /usr/local/lib/libbls-dash.a(bls.cpp.o): in function `bls::BLS::Init()':
bls.cpp:(.text+0x6c): undefined reference to `sodium_init'
/usr/bin/ld: bls.cpp:(.text+0x7b): undefined reference to `sodium_free'
/usr/bin/ld: bls.cpp:(.text+0x82): undefined reference to `sodium_malloc'

This one I haven't figured out.

This was all on an Ubuntu Focal chroot because all other version of Ubuntu/Debian I tried failed for varying reasons.

I'm sure you know how you're compiling this. Why can't you share that information?

[Liquid369]

Hello,

I usually just ask to see if I can see a problem in the process you are using to build.
No reason to think we are trying to 'hide' how to compile the wallet, below is what process I follow.

Ubuntu 18.04
Clone StakeCubeCoin repository

git clone https://github.com/stakecube/StakeCubeCoin.git
cd StakeCubeCoin/depends
make HOST=x86_64-pc-linux-gnu
cd ..
bash autogen.sh
./configure --prefix=`pwd`/depends/x86_64-pc-linux-gnu --enable-glibc-back-compat --enable-reduce-exports --disable-bench --disable-gui-tests --disable-tests --enable-crash-hooks
make

Now this is also based on having the main depends such as build-essential for being able to compile with gcc/clang/make

Follow for depends here:

sudo apt-get install curl build-essential libtool autotools-dev automake pkg-config python3 bsdmainutils

Hopefully this helps.

[KenSharp]

First time running 3.3.0 and I saw the memory balloon again albeit at a slower rate (had to restart after around 24 hours).

Been running another 24 hours and the memory use is significantly better. My machine isn't even touching swap! I'll leave it running for a week and report back.

Massive improvement, thank you!

[KenSharp]

Turns out I wasn't hitting swap because the MN were POSE_BANNED (I wasn't aware of another update).

Anyway, after sorting that out the memory use has vastly improved. I think it might still balloon, albeit a lot more slowly. I'll have to look at this when I get chance but the improvement is incredible. Excellent work, thank you again.

[KenSharp]

Been running for months now. Memory ballooning has reduced dramatically, but the memory does continue to increase over time, so there's likely to be some more memory leaks, though these could be in dependencies.

To workaround the leaks I restart all MN daily. When doing this I have no issues at all. Thank you for working on this!

[squidicuzz]

This should be a bit better now. Please open a new ticket if this is still an issue for you.