HiveServer test failed at Hive View startup in Ambari

When you try to start Hive View in Ambari UI you may encounter an error like this. There may be a couple of reasons for this error, the important thing is knowing where to look to locate the source of a problem.

Error:

This is the Stack Trace from the UI:

Service ‘hiveserver’ check failed:
java.util.concurrent.TimeoutException: deadline passed

java.util.concurrent.TimeoutException: deadline passed
at akka.actor.dsl.Inbox$InboxActor$$anonfun$receive$1.applyOrElse(Inbox.scala:117)
at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:189)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.dsl.Inbox$InboxActor.aroundReceive(Inbox.scala:62)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

 

The first place to quickly check is the panel on the left of the Ambari home screen:

If you cannot locate the problem there (in this case we can), check the logs of the HiveServer.  cat /var/log/hive/hiveserver2.log and observe the logs. To be able to do that you need to ssh into the machine running the Hadoop ecosystem services. In my case, I am doing this on my local machine so I do  ssh root@127.0.0.1 -p 2222 . The default password is “hadoop”. On a windows machine, you will need a tool like Putty. On Linux or Mac, you can use your terminal.

 

This is the line that gives me a hint about the problem:

FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(210)) – Unable to create HiveServer2 namespace: hiveserver2 on ZooKeeper

A quick check of the zookeeper service indicates that the service is actually down.

Checking the logs of the zookeeper I see this:

[[email protected] ~]# cat /var/log/zookeeper/zookeeper-zookeeper-server-sandbox.hortonworks.com.out
2019-01-12 13:21:45,423 – INFO [main:[email protected]] – Reading configuration from: /usr/hdp/current/zookeeper-server/conf/zoo.cfg
2019-01-12 13:21:45,437 – ERROR [main:[email protected]] – Invalid configuration, only one server specified (ignoring)
2019-01-12 13:21:45,441 – INFO [main:[email protected]] – autopurge.snapRetainCount set to 30
2019-01-12 13:21:45,441 – INFO [main:[email protected]] – autopurge.purgeInterval set to 24
2019-01-12 13:21:45,450 – WARN [main:[email protected]] – Either no config or no quorum defined in config, running in standalone mode
2019-01-12 13:21:45,450 – INFO [PurgeTask:[email protected]] – Purge task started.
2019-01-12 13:21:45,486 – INFO [PurgeTask:[email protected]] – Purge task completed.
2019-01-12 13:21:45,509 – INFO [main:[email protected]] – Reading configuration from: /usr/hdp/current/zookeeper-server/conf/zoo.cfg
2019-01-12 13:21:45,510 – ERROR [main:[email protected]] – Invalid configuration, only one server specified (ignoring)
2019-01-12 13:21:45,510 – INFO [main:[email protected]] – Starting server
2019-01-12 13:21:45,524 – INFO [main:[email protected]] – Server environment:zookeeper.version=3.4.6-1245–1, built on 08/26/2016 00:47 GMT
2019-01-12 13:21:45,524 – INFO [main:[email protected]] – Server environment:host.name=sandbox.hortonworks.com
2019-01-12 13:21:45,524 – INFO [main:[email protected]] – Server environment:java.version=1.8.0_111
2019-01-12 13:21:45,525 – INFO [main:[email protected]] – Server environment:java.vendor=Oracle Corporation
2019-01-12 13:21:45,525 – INFO [main:[email protected]] – Server environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/jre
2019-01-12 13:21:45,525 – INFO [main:[email protected]] – Server environment:java.class.path=/usr/hdp/current/zookeeper-server/bin/../build/classes:/usr/hdp/current/zookeeper-server/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-server/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/current/zookeeper-server/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/current/zookeeper-server/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/current/zookeeper-server/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/current/zookeeper-server/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/current/zookeeper-server/bin/../lib/wagon-http-2.4.jar:/usr/hdp/current/zookeeper-server/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/current/zookeeper-server/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/current/zookeeper-server/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/current/zookeeper-server/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/current/zookeeper-server/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/current/zookeeper-server/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeeper-server/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zookeeper-server/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/commons-io-2.2.jar:/usr/hdp/current/zookeeper-server/bin/../lib/commons-codec-1.6.jar:/usr/hdp/current/zookeeper-server/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/current/zookeeper-server/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/current/zookeeper-server/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/current/zookeeper-server/bin/../lib/ant-1.8.0.jar:/usr/hdp/current/zookeeper-server/bin/../zookeeper-3.4.6.2.5.0.0-1245.jar:/usr/hdp/current/zookeeper-server/bin/../src/java/lib/*.jar:/usr/hdp/current/zookeeper-server/conf::/usr/share/zookeeper/*:/usr/share/zookeeper/*
2019-01-12 13:21:45,529 – INFO [main:[email protected]] – Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-01-12 13:21:45,532 – INFO [main:[email protected]] – Server environment:java.io.tmpdir=/tmp
2019-01-12 13:21:45,532 – INFO [main:[email protected]] – Server environment:java.compiler=<NA>
2019-01-12 13:21:45,534 – INFO [main:[email protected]] – Server environment:os.name=Linux
2019-01-12 13:21:45,535 – INFO [main:[email protected]] – Server environment:os.arch=amd64
2019-01-12 13:21:45,535 – INFO [main:[email protected]] – Server environment:os.version=3.10.0-327.el7.x86_64
2019-01-12 13:21:45,536 – INFO [main:[email protected]] – Server environment:user.name=zookeeper
2019-01-12 13:21:45,536 – INFO [main:[email protected]] – Server environment:user.home=/home/zookeeper
2019-01-12 13:21:45,536 – INFO [main:[email protected]] – Server environment:user.dir=/home/zookeeper
2019-01-12 13:21:45,538 – INFO [main:[email protected]] – tickTime set to 2000
2019-01-12 13:21:45,539 – INFO [main:[email protected]] – minSessionTimeout set to -1
2019-01-12 13:21:45,539 – INFO [main:[email protected]] – maxSessionTimeout set to -1
2019-01-12 13:21:45,590 – INFO [main:[email protected]] – binding to port 0.0.0.0/0.0.0.0:2181
2019-01-12 13:21:45,730 – INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:[email protected]] – Accepted socket connection from /172.17.0.2:37689
2019-01-12 13:21:45,896 – ERROR [main:[email protected]] – Unexpected exception, exiting abnormally
java.io.IOException: Transaction log: /hadoop/zookeeper/version-2/log.380 has invalid magic number 0 != 1514884167
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:578)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:595)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:561)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:643)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:158)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:272)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:399)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:113)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

I tried to investigate the contents of log.380 ( vi /hadoop/zookeeper/version-2/log.380 ) and file seemed to be corrupt, so run  mv /hadoop/zookeeper/version-2/log.380 /hadoop/zookeeper/version-2/Xlog.380X to rename the file. Then I started zookeeper service expecting zookeeper service to create a new file. As I expected, zookeeper created a new uncorrupt file with the same name and started running. After successfully starting the zookeeper service, I started HiveServer2 service and Hive View was working!

Note:

I am running on Hortonworks sandbox (HDP 2.5) which can be found here: https://hortonworks.com/downloads/

Hope this helps.
Good Luck,
Serdar