ThreadLocal Hubs should be cleaned up

[adinauer]

Problem Statement

At the moment Hub instances are stored in ThreadLocal variables. They are never cleaned up and thus cause servers like Tomcat to issue warnings as can be seen here #2074, here #420 and here #487. While there are mechanisms to clean up these leaks in Tomcat we should not rely on them and clean up instead.

sentry-java/sentry/src/main/java/io/sentry/Sentry.java

Line 21 in 9f1af1c

private static final @NotNull ThreadLocal<IHub> currentHub = new ThreadLocal<>();

Solution Brainstorm

We may have to move away from ThreadLocal variables at some point to support Webflux and similar. Until then we could try and unset the ThreadLocal after a request is done (ServletRequestListener.requestDestroyed)

[bruno-garcia]

Closed through #2076 (somehow GitHub didn't automatically close it)

[adinauer]

#2076 only allows Threads that had any Sentry.* method called before Sentry.init to self heal. This issue is still unresolved.

[adinauer]

We could do it similar to this: sentry/src/main/java/io/sentry/servlet/SentryServletRequestListener.java

[adamnegaard]

Is there any way to avoid the memory leaks in tomcat 9 caused by this issue? It's is causing developers to have to restart tomcat whenever we have redeployed applications a few times.
If any one could give me a tip on how to get started fixing this issue, i might have time to try and solve it , assuming it's a small effort.

[adinauer]

Hello @adamnegaard what frameworks are you using? Is it Spring? Spring Boot? WebMVC? WebFlux?

Basically the idea here is to call Sentry.setCurrentHub(NoOpHub.getInstance()); once we know we're done with a request. In the latest versions of the SDK this should already be the case for WebFlux. This should cleanup the hubs created on the request threads. However there may still be more thread locals depending on thread / SDK usage.

[adamnegaard]

Hi @adinauer, thanks for taking the time to get back to me.
We are using Spring Boot, for developing API's. We build our applications as .WAR files, and deploy them on a tomcat server. We are using logback for logging, and thus using the SentryAppender found here:
https://github.com/getsentry/sentry-java/blob/a78496a9730742d6837339f2b359e66776f94eb1/sentry-logback/src/main/java/io/sentry/logback/SentryAppender.java

Whenever we undeploy an app, tomcat is not able to unload classes, which have been loaded by the app. It seems like the reason for this is the static reference to a ThreadLocal here, which cannot be terminated on application undeploy:
https://github.com/getsentry/sentry-java/blob/a78496a9730742d6837339f2b359e66776f94eb1/sentry/src/main/java/io/sentry/Sentry.java#LL33C10-L33C10
On every reploy, tomcat loads all the needed classes for the app, but never unloads the classes from the undeployed application. This essentially causes a PermGen memory leak.
It is worth mentioning, whenever we restart tomcat, it is able to unload all the classes, but this is not ideal for our current setup.

What i have tried so far, is creating a CustomSentryAppender that extends SentryAppender, and using that with logback. I have tried overriding the stop() method as follows:

@Override
public void stop() {
    Sentry.setCurrentHub(NoOpHub.getInstance());
    Sentry.close();
    super.stop();
}

I can see that this method gets invoked on undeploy, but it does not seem to be fixing our memory leak.

Is there any known workaround for this issue? Otherwise, could you give me any pointers to where Sentry requests are performed, so i can try unsetting the hub reference after a request is done?
Thanks a lot 😄 .