Netty exceptions during Pantheon 1.2.3 startup on Amazon Linux 1

Description

after a brief chat with in gitter, i'm leaving this note here.

we recently began updating our private IBFT network nodes with pantheon-1.2.3-SNAPSHOT built from the sources. we'd previously been running pantheon-1.1.3-SNAPSHOT, again built from the sources. we did not see this issue in the earlier code. upon launching our nodes (with ./bin/pantheon --config-file=config.toml) , we get the following two exceptions back-to-back:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 2019-08-31 03:37:39.550+00:00 | main | DEBUG | PlatformDependent0 | direct buffer constructor: unavailable java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31) ~[netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.internal.PlatformDependent0$4.run(PlatformDependent0.java:224) ~[netty-common-4.1.34.Final.jar:4.1.34.Final] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:218) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:213) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:81) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:269) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.<init>(SingleThreadEventExecutor.java:165) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.SingleThreadEventLoop.<init>(SingleThreadEventLoop.java:58) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:135) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.Vertx.vertx(Vertx.java:85) [vertx-core-3.8.0.jar:3.8.0] at tech.pegasys.pantheon.cli.PantheonCommand.synchronize(PantheonCommand.java:1244) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.PantheonCommand.startSynchronization(PantheonCommand.java:793) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.PantheonCommand.run(PantheonCommand.java:704) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at picocli.CommandLine.execute(CommandLine.java:1160) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine.access$800(CommandLine.java:141) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine$RunLast.handle(CommandLine.java:1367) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine$RunLast.handle(CommandLine.java:1335) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526) [picocli-3.9.6.jar:3.9.6] at tech.pegasys.pantheon.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:59) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:29) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526) [picocli-3.9.6.jar:3.9.6] at tech.pegasys.pantheon.cli.PantheonCommand.parse(PantheonCommand.java:789) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.PantheonCommand.parse(PantheonCommand.java:696) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.Pantheon.main(Pantheon.java:47) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] 2019-08-31 03:37:39.615+00:00 | main | DEBUG | PlatformDependent0 | java.nio.Bits.unaligned: available, true 2019-08-31 03:37:39.645+00:00 | main | DEBUG | PlatformDependent0 | jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @2e32fc22 at jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:361) ~[?:?] at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:591) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:558) ~[?:?] at io.netty.util.internal.PlatformDependent0$6.run(PlatformDependent0.java:334) ~[netty-common-4.1.34.Final.jar:4.1.34.Final] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:325) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:213) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:81) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:269) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.<init>(SingleThreadEventExecutor.java:165) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.SingleThreadEventLoop.<init>(SingleThreadEventLoop.java:58) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:135) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [netty-common-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59) [netty-transport-4.1.34.Final.jar:4.1.34.Final] at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32) [vertx-core-3.8.0.jar:3.8.0] at io.vertx.core.Vertx.vertx(Vertx.java:85) [vertx-core-3.8.0.jar:3.8.0] at tech.pegasys.pantheon.cli.PantheonCommand.synchronize(PantheonCommand.java:1244) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.PantheonCommand.startSynchronization(PantheonCommand.java:793) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.PantheonCommand.run(PantheonCommand.java:704) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at picocli.CommandLine.execute(CommandLine.java:1160) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine.access$800(CommandLine.java:141) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine$RunLast.handle(CommandLine.java:1367) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine$RunLast.handle(CommandLine.java:1335) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526) [picocli-3.9.6.jar:3.9.6] at tech.pegasys.pantheon.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:59) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:29) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243) [picocli-3.9.6.jar:3.9.6] at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526) [picocli-3.9.6.jar:3.9.6] at tech.pegasys.pantheon.cli.PantheonCommand.parse(PantheonCommand.java:789) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.cli.PantheonCommand.parse(PantheonCommand.java:696) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] at tech.pegasys.pantheon.Pantheon.main(Pantheon.java:47) [pantheon-1.2.3-SNAPSHOT.jar:1.2.3-dev-dfec928c] 2019-08-31 03:37:39.647+00:00 | main | DEBUG | PlatformDependent0 | java.nio.DirectByteBuffer.<init>(long, int): unavailable

identified these exceptions as known issues (annoying warnings), although i wasn't able to find a relevant issue in the Jira. so far we have been safely able to ignore them, but they happen every single time we start up the node.

we did manage to find a possibly related issue among the bugs in the Netty project, but it was for an earlier version of Java. i include it here in the hope that it is helpful in some way:
https://github.com/netty/netty/issues/7768

our setup may be a bit peculiar, so i'll include some details here:

  • we're running in AWS, on a server which cohabitates with apache2 and tomcat. we run some application WAR files in this configuration that control the install of Pantheon through JSON-RPC.

  • the kernel we use in this configuration is : Linux 4.14.138-89.102.amzn1.x86_64 #1 SMP Thu Aug 15 15:41:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux.

  • this is derived from an Amazon AMI that is their minimal configuration for something they call "Amazon Linux 1," a variant of RHEL.

  • because Amazon Linux 1 doesn't natively support Java 11, we have utilized Amazon's variant of OpenJDK called Corretto.

  • the specific java version is : openjdk version "11.0.4" 2019-07-16 LTS
    OpenJDK Runtime Environment Corretto-11.0.4.11.1 (build 11.0.4+11-LTS)
    OpenJDK 64-Bit Server VM Corretto-11.0.4.11.1 (build 11.0.4+11-LTS, mixed mode)

  • our pantheon version is: pantheon/v1.2.3-dev-dfec928c/linux-x86_64/corretto-java-11

i'm unsure about how your folks use labels and components here, but i've made my best guesses.
-madfu

Status

Assignee

Unassigned

Reporter

wallet.AI codebot

Labels

Refinement State

Not Started

Scrum Team

None

Components

Priority

P4