NoSuchMethodError for "list.toMap" in spark-submit -

-

when ran spark-submit following simple spark program of:

import org.apache.spark.sparkcontext._ import org.apache.spark.sparkconf import org.apache.spark.rdd.rdd import org.apache.spark.sparkcontext import org.apache.spark._ import sparkcontext._  object test2{ def main(args:array[string]) {   val conf = new sparkconf().setappname("test")   val sc=new sparkcontext(conf)    val list=list(("aa",1),("bb",2),("cc",3))   val maps=list.tomap }  }

i got java.lang.nosuchmethoderror line of "val maps=list.tomap". in spark-shell or scala, has no problem:

scala> val list=list(("aa",1),("bb",2),("cc",3)) list: list[(string, int)] = list((aa,1), (bb,2), (cc,3))  scala> val maps=list.tomap maps: scala.collection.immutable.map[string,int] = map(aa -> 1, bb -> 2, cc -> 3)

so use "tomap" method, missing in spark-submit? use "sbt package" compile program , without problem. thanks!

p.s: build.sbt file as:

name := "test2" version := "1.0" scalaversion := "2.11.6"

sbt package creates jar containing only project, not dependencies.

when using spark-submit, need jar contains project , dependencies except spark libraries (already provided spark installation on cluster)

this uber-jar can created using sbt assembly plugin:

  • in project folder, create file project/assembly.sbt line :

addsbtplugin("com.eed3si9n" % "sbt-assembly" % "0.14.0")

  • in build.sbt, spark dependencies should tagged "provided":

    librarydependencies ++= seq(   ...   "org.apache.spark"  %% "spark-core" % sparkversion % "provided"   ... )

  • run sbt assembly, create uber-jar target/scala-2.11/test2-1.0.jar


Comments

Post a Comment

Popular posts from this blog

python - pip install -U PySide error -

-
does know how avoid following error when running pip install -u pyside per official website instructions here: https://pypi.python.org/pypi/pyside/#installing-pyside-on-a-mac-os-x-system note have done brew install qt successfully. you using pip version 7.0.3, version 7.1.2 available. should consider upgrading via 'pip install --upgrade pip' command. collecting pyside using cached pyside-1.2.4.tar.gz installing collected packages: pyside running setup.py install pyside complete output command /applications/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-build-cpxemt/pyside/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-0r7hri-record/install-record.txt --single-version-externally-managed --compile:
Read more

arrays - C++ error: a brace-enclosed initializer is not allowed here before ‘{’ token -

-
simple question: i'm trying initialize array within c++ class declaration: using namespace std; #include <string> class myclass{ public: string myarray[] = {"a","b","c"}; }; and i'm getting error: error: brace-enclosed initializer not allowed here before ‘{’ token no, without complied c++11 compiler, cannot initialize member array in declaration. have initialize array member in constructor. , don't use open array if know number of elements initialize array.
Read more

cytoscape.js - How to add nodes to Dagre layout with Cytoscape -

-
i trying add nodes dagre graph through cityscape.js. problem nodes not painted. code have: cy.add({ group: "nodes", data: { id:"n0",name:"n0",weight: 75 }, }); $.ajax(url).done(function(jsontree){ var newnodes=[]; for(var i=1;i<6;i++){ //var child = jsontree.result[i]; newnodes.push( { group: "nodes", data: { id: "n"+i, name:"n"+i}}, { group: "edges", data: { id: "e"+i, source: "n0", target: "n"+i } } ); } cy.add(newnodes); any ideas? have used loaded function reload graph not work. not know if have use specific function reload graph. thank in advance. you need add positions new nodes. cytoscape.js docs : it important note positions of newly added nodes must defined when calling cy.add(). nodes can not placed in graph without valid position — otherwise not displayed. yo
Read more