Slf4j java.lang.LinkageError

[wolframhaussig]

As requested by @eyalkoren I open a separate issue for this. When using the elastic-apm-agent-1.28.1.jar attached to a Tomcat instance I see the following error:

java.lang.LinkageError: loader constraint violation: loader co.elastic.apm.agent.bci.classloading.IndyPluginClassLoader @489488de wants to load interface org.slf4j.Marker. A different interface with the same name was previously loaded by org.apache.catalina.loader.ParallelWebappClassLoader @193a4a55. (org.slf4j.Marker is in unnamed module of loader org.apache.catalina.loader.ParallelWebappClassLoader @193a4a55, parent loader java.net.URLClassLoader @6807989e)
        at co.elastic.logging.logback.EcsEncoder.serializeMarker(EcsEncoder.java:139)
        at co.elastic.logging.logback.EcsEncoder.serializeMarkers(EcsEncoder.java:132)
        at co.elastic.logging.logback.EcsEncoder.encode(EcsEncoder.java:103)
        at co.elastic.logging.logback.EcsEncoder.encode(EcsEncoder.java:43)
        at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:230)
        at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:235)
        at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at co.elastic.apm.agent.logback.LogbackAppenderAppendAdvice.shadeLoggingEvent(LogbackAppenderAppendAdvice.java:49)
        at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
        at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.info(Logger.java:599)
        ...
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:668)
        ...
        at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
        at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199)
        at org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter.attemptAuthentication(UsernamePasswordAuthenticationFilter.java:94)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:124)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.multipart.support.MultipartFilter.doFilterInternal(MultipartFilter.java:122)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        ...
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:645)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1506)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Originally posted by @wolframhaussig in #2350 (comment)

[eyalkoren]

Copying my comment to here: this seems related to the mechanism added in #2167 - now the ECS logging classes have visibility to the logging framework, but they may also have duplicated visibility to slf4j.

[eyalkoren]

@wolframhaussig I think #2376 may fix your issue as well since I excluded the slf4j classes from the IndyPlugin CL. Please try this snapshot and let us know if it resolves your issue.

[wolframhaussig]

@eyalkoren Unfortunately, I still get the following error:

java.lang.LinkageError: loader constraint violation: loader co.elastic.apm.agent.bci.classloading.IndyPluginClassLoader @6ce997f3 wants to load interface org.slf4j.Marker. A different interface with the same name was previously loaded by org.apache.catalina.loader.ParallelWebappClassLoader @3b4b0490. (org.slf4j.Marker is in unnamed module of loader org.apache.catalina.loader.ParallelWebappClassLoader @3b4b0490, parent loader java.net.URLClassLoader @6d7e2795)
        at co.elastic.logging.logback.EcsEncoder.serializeMarker(EcsEncoder.java:148)
        at co.elastic.logging.logback.EcsEncoder.serializeMarkers(EcsEncoder.java:141)
        at co.elastic.logging.logback.EcsEncoder.encode(EcsEncoder.java:103)
        at co.elastic.logging.logback.EcsEncoder.encode(EcsEncoder.java:43)
        at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:230)
        at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:235)
        at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at co.elastic.apm.agent.logback.LogbackAppenderAppendAdvice.shadeLoggingEvent(LogbackAppenderAppendAdvice.java:49)
        at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
        at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.info(Logger.java:599)
        ...
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:668)
        ...
        at org.springframework.context.support.ApplicationContextAwareProcessor.invokeAwareInterfaces(ApplicationContextAwareProcessor.java:121)
        at org.springframework.context.support.ApplicationContextAwareProcessor.postProcessBeforeInitialization(ApplicationContextAwareProcessor.java:97)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:407)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1611)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
        ...
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4699)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1125)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1859)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

[eyalkoren]

OK, so even through slf4j is not loaded by the IndyPluginCL, the fact that it is loaded by the agent CL when it is already loaded by the second IndyPluginCL parent (the webapp CL), is enough to trigger this.
Once I get the chance, I'll find a way to overcome that.
In the mean time, use 1.26.2.
Thanks for trying and reporting back!

[eyalkoren]

I thought about restoring slf4j shading, but this means adding all related complications we had. Two that I can remember:

• Escape shading wherever we want to do something with the application's slf4j, for example in MdcActivationListener and Slf4jLoggerErrorCapturingInstrumentation
• Duplicated shading in the ECS-reformatting modules so that we end up looking up the application's slf4j and not the one bundled with the agent

If this doesn't simplify things, we might as well find a solution for this specific loading issue

[felixbarny]

If we can find a solution for the specific issue, I think that would be much preferable.

[felixbarny]

The issue is related to this comment:

apm-agent-java/apm-agent-core/src/main/java/co/elastic/apm/agent/bci/classloading/IndyPluginClassLoader.java

Line 70 in ce271a8

// This implies, we can't reference instrumented slf4j classes in plugins, though.

The assumption that we never need to reference the application's slf4j falls short when doing ecs_log_reformatting for logback. That's because in logback, the ch.qos.logback.classic.spi.ILoggingEvent references org.slfj4.Marker. The agent tries to resolve the Marker class from the agent class loader whereas in this instance, we need the Marker class from the application.

[eyalkoren]

I am currently thinking of several options:

1. adding some per-plugin capability to require specific packages to always or never be loaded by the agent CL. Probably by extending co.elastic.apm.agent.bci.PluginClassLoaderRootPackageCustomizer. Then, use it within the ECS-reformatting plugins for slf4j. However, we will have to handle or remove agent logging within these plugins
2. since shading has its problems, we can instead add a minimal proprietary facade over the slf4j facade, so that plugins can only use it for logging. We can provide this through the SDK and then there's no reason for external plugins to ever package slf4j as well. Since the facade implementation will only be loaded by the agent CL (delegating to slf4j) - it will be safe
3. replace slf4j with our own facade...

[felixbarny]

we can instead add a minimal proprietary facade over the slf4j facade

In a similar vain, we could also just re-package slf4j into the plugin-sdk and into a custom namespace, such as co.elastic.apm.agent.slf4j.