Sunday, 19 June 2016

Spark-shell: :10: error: not found: value sqlContext on windows




WARN Connection: BoneCP specified but not present in    
 CLASSPATH (or one of dependencies)
WARN Connection: BoneCP specified but not present in                 CLASSPATH (or one of dependencies)
WARN ObjectStore: Version information not found in     metastore. hive.metastore.schema.verification is not enabled so recording the     schema version 1.2.0
WARN ObjectStore: Failed to get database default,         returning NoSuchObjectException
WARN : Your hostname, DESKTOP-8JS2RD5 resolves to a     loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:103%net1, but we couldn't     find any external IP address!
java.lang.RuntimeException: java.lang.NullPointerException
        at     org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.spark.sql.hive.client.ClientWrapper.<init>    (ClientWrapper.scala:171)
    at     org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala    :163)
        at     org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
        at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:168)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
    at $iwC$$iwC.<init>(<console>:9)
    at $iwC.<init>(<console>:18)
    at <init>(<console>:20)
    at .<init>(<console>:24)
    at .<clinit>(<console>)
    at .<init>(<console>:7)
    at .<clinit>(<console>)
    at $print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
    at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132)
    at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
    at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
    at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
    at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)

          at       org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
          at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
          at       org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.sca      la:108)
          at       org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
          at       org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$proc      ess$1.apply$mcZ$sp(SparkILoop.scala:991)
          at       org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$proc      ess$1.apply(SparkILoop.scala:945)
          at       org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$proc      ess$1.apply(SparkILoop.scala:945)
          at       scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scal      a:135)
          at       org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(Spark      ILoop.scala:945)
          at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
          at org.apache.spark.repl.Main$.main(Main.scala:31)
          at org.apache.spark.repl.Main.main(Main.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.NullPointerException
    at java.lang.ProcessBuilder.start(Unknown Source)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
    at org.apache.hadoop.util.Shell.run(Shell.java:418)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
    at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:559)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:534)
    at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    ... 56 more

  <console>      :10: error: not found: value sqlContext
               import sqlContext.implicits._
                ^
  <console>:10: error: not found: value sqlContext
         import sqlContext.sql
          ^

  scala>

Fix:
  Fix:
  1) Copy winutils.exe from here(https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin) to some folder say, C:\Hadoop\bin. Set HADOOP_HOME to C:\Hadoop.
  2) Open admin command prompt. Run C:\Hadoop\bin\winutils.exe chmod 777 /tmp/hive

Result:

9 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.


    rpa training in chennai | best rpa training in chennai | rpa training in chennai | rpa training in bangalore
    rpa training in pune | rpa online training

    ReplyDelete
  5. I found this informative and interesting blog so i think so its very useful and knowledge able.I would like to thank you for the efforts you have made in writing this article.
    java training in omr | oracle training in chennai

    java training in annanagar | java training in chennai

    ReplyDelete
  6. Inspiring writings and I greatly admired what you have to say , I hope you continue to provide new ideas for us all and greetings success always for you..Keep update more information..
    python training in rajajinagar
    Python training in btm
    Python training in usa

    ReplyDelete
  7. Nice post ! Thanks for sharing valuable information with us. Keep sharing.. Big Data Hadoop Online Training Bangalore


    ReplyDelete
  8. I simply wanted to thank you so much again. I am not sure the things
    that I might have gone through without the type of hints revealed by
    you regarding that situation.
    ASP Dot Net Training in Chennai
    C Sharp Training in Chennai

    ReplyDelete