java.lang.VerifyError: Bad return type

[manish160993]

Exception Details: Location: org/apache/catalina/connector/Request.getReader()Ljava/io/BufferedReader; @359: areturn Reason: Type 'java/lang/Object' (current frame, stack[0]) is not assignable to 'java/io/BufferedReader' (from method signature) Current Frame: bci: @359 flags: { } locals: { 'org/apache/catalina/connector/Request', 'datadog/trace/api/gateway/RequestContext', 'java/lang/Object' } stack: { 'java/lang/Object' } Bytecode: 0000000: 014c 2a4d 002c b400 c899 0014 bb02 b859 0000010: b201 d213 0359 b601 d9b7 02bb bf2c b401 0000020: 3cb6 028a c700 1f2c b602 8c4e 2dc6 0016 0000030: 2db9 0291 0100 3a04 1904 c600 092c 1904 0000040: b603 5c2c 04b5 00ca 2cb4 00ba b603 5f2c 0000050: b400 c6c7 0012 2cbb 00c3 592c b400 bab7 0000060: 00c4 b500 c62c b400 c6a7 0003 4db8 02c1 0000070: 59c6 00dd b902 c501 0059 c600 d459 b202 0000080: cbb9 02d1 0200 c600 c84c 2cc7 0006 a700 0000090: d8b8 02c1 4e2d c700 06a7 00cd 2a13 02d3 00000a0: b902 d402 003a 0419 04c7 000a 2cc1 0361 00000b0: 9900 06a7 00b3 2db9 02c5 0100 3a05 1905 00000c0: c700 06a7 00a3 b802 d9b2 02cb b902 df02 00000d0: 003a 0619 06b2 02e5 b602 e9b9 02ef 0200 00000e0: c002 f13a 0719 06b2 02e5 b602 f4b9 02ef 00000f0: 0200 c002 f13a 0819 07c6 0008 1908 c700 0000100: 06a7 0065 2a13 02d3 b202 f7b9 02f8 0300 0000110: 0336 092a 1302 fab9 02fd 0200 3a0a 190a 0000120: c600 0f19 0ab8 0303 3609 a700 053a 0bbb 0000130: 0363 5919 0519 0719 0815 09b7 0366 3a0b 0000140: bb03 6159 2c19 0bb7 0369 4da7 001b 57a7 0000150: 0017 1301 7db8 0183 5f13 0185 5fb9 018b 0000160: 0300 a700 0457 2cb0 Exception Handler Table: bci [291, 298] => handler: 301 bci [109, 338] => handler: 338 bci [338, 354] => handler: 357 Stackmap Table: append_frame(@29,Null,Object[#3]) same_frame(@67) same_frame(@101) same_locals_1_stack_item_frame(@108,Object[#195]) full_frame(@145,{Object[#3],Object[#717],Object[#195]},{}) append_frame(@156,Object[#356]) append_frame(@179,Object[#5]) same_frame(@182) append_frame(@198,Object[#717]) append_frame(@257,Object[#747],Object[#753],Object[#753]) same_frame(@260) full_frame(@301,{Object[#3],Object[#717],Object[#195],Object[#356],Object[#5],Object[#717],Object[#747],Object[#753],Object[#753],Integer,Object[#660]},{Object[#694]}) same_frame(@303) full_frame(@334,{Object[#3],Null,Object[#195]},{Object[#5]}) full_frame(@338,{Object[#3],Object[#717],Object[#5]},{Object[#155]}) same_locals_1_stack_item_frame(@357,Object[#155]) same_frame(@358) at org.apache.catalina.connector.Connector.createRequest(Connector.java:922) at org.apache.catalina.connector.CoyoteAdapter.log(CoyoteAdapter.java:477) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:429) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source)

[mcculls]

Hi @manish160993 are you using any other -javaagent options on the command-line?

Could you add -Dnet.bytebuddy.dump=path-to-writable-directory and recreate the issue. This JVM option will dump all instrumented classes before and after any advice was added to the given directory. The classes saved there will include org.apache.catalina.connector.Request-original.<number>.class and org.apache.catalina.connector.Request.<number>.class

If you could add those files here then we can check the bytecode against what we'd expect to be added.

It would also be useful to see the DATADOG TRACER CONFIGURATION logged during startup - thanks

[manish160993]

Hi @mcculls,
Thanks for the quick response.
We do add -javaagent:/usr/app/newrelic/newrelic.jar too in the command line.
org.apache.catalina.connector.Request-original.1676399002129.class
org.apache.catalina.connector.Request.1676399002129.class

[mcculls]

Hi @manish160993 - what happens if you remove the newrelic agent, but keep the datadog agent?

Another thing to try would be making sure the datadog agent appears before the newrelic agent on the command-line.

BTW, is it possible to attach the actual class-files listed above?

[manish160993]

@mcculls
Please find the attached zip file. It contains the class files mentioned above.
class files.zip

DATADOG TRACER CONFIGURATION {"version":"1.5.0-SNAPSHOT~4de17c1807","os_name":"Linux","os_version”:”X.X.X-X.X.amzn2.x86_64","architecture":"amd64","lang":"jvm","lang_version":"17.0.6","jvm_vendor":"Azul Systems, Inc.","jvm_version":"17.0.6+10-LTS","java_class_version":"61.0","http_nonProxyHosts":"null","http_proxyHost":"null","enabled":true,"service”:<service-name>,”agenl":"http://XX.XX.XX.XX:8126","agent_error":false,"debug":false,"analytics_enabled":false,"sampling_rules":[{},{}],"priority_sampling_enabled":true,"logs_correlation_enabled":true,"profiling_enabled":false,"remote_config_enabled":true,"debugger_enabled":false,"appsec_enabled":"ENABLED_INACTIVE","telemetry_enabled":true,"dd_version":"","health_checks_enabled":true,"configuration_file":"no config file present","runtime_id":"69ebcdfd-7400-4c17-bd88-74c7a6edf2b1","logging_settings":{"levelInBrackets":false,"dateTimeFormat":"'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'","logFile":"System.err","configurationFile":"simplelogger.properties","showShortLogName":false,"showDateTime":true,"showLogName":true,"showThreadName":true,"defaultLogLevel":"INFO","warnLevelString":"WARN","embedException":false},"cws_enabled":false,"cws_tls_refresh":5000,"datadog_profiler_enabled":true,"datadog_profiler_safe":true}

[rsurenderreddy]

is this isue fixed? can you share work a round to use to both newrelic and datadog?

[pc2459]

+1 on hoping someone has a workaround; we're seeing a similar issue when trying to run both agents simultaneously to evaluate both platforms, though in our case it's Jetty:

java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/eclipse/jetty/server/Request.getReader()Ljava/io/BufferedReader; @381: areturn
  Reason:
    Type 'java/lang/Object' (current frame, stack[0]) is not assignable to 'java/io/BufferedReader' (from method signature)
  Current Frame:
    bci: @381
    flags: { }
    locals: { 'org/eclipse/jetty/server/Request', 'datadog/trace/api/gateway/RequestContext', 'java/lang/Object' }
    stack: { 'java/lang/Object' }
  Bytecode: