Multicore usage in Spark and Scala and memory sharing -

-

i'm new spark, scala , have few questions -

  1. when set master local[2], mean that, whenever perform rdd.map( ... ).collect(), map function execute on 2 cores?
  2. following on 1, optimal if number of items in rdd equal number of cores? (so each map done in 1 core.)

  3. how memory shared if reuse values?

    i.e.

    val conf = new sparkconf().setmaster("local["+cores+"]") val sc = new sparkcontext(conf)  // in practice following sparse matrix want shared between threads val myhugevalue = 100  // in practice following interact myhugevalue sc.parallelize(array.range(0, cores)).map(myhugevalue + _).collect()

    in above, myhugevalue copied each mapper function uses myhugevalue? if so, how change impl doesn't copied over?

edit 1:

to more specific 2., suppose there 10 nodes or worker cores in cluster, won't processing done efficiently if data split 10 parts such each node can process batch? there reason split data smaller chunks , increase network overhead?

as far reason not want copy value myhugevalue because huge object , take time copying every map operation. map operations called lot of iterations. rather want cache object once per cluster , call map function repeatedly. there spark way of doing this?


Comments

Post a Comment

Popular posts from this blog

python - pip install -U PySide error -

-
does know how avoid following error when running pip install -u pyside per official website instructions here: https://pypi.python.org/pypi/pyside/#installing-pyside-on-a-mac-os-x-system note have done brew install qt successfully. you using pip version 7.0.3, version 7.1.2 available. should consider upgrading via 'pip install --upgrade pip' command. collecting pyside using cached pyside-1.2.4.tar.gz installing collected packages: pyside running setup.py install pyside complete output command /applications/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-build-cpxemt/pyside/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-0r7hri-record/install-record.txt --single-version-externally-managed --compile: ...
Read more

arrays - C++ error: a brace-enclosed initializer is not allowed here before ‘{’ token -

-
simple question: i'm trying initialize array within c++ class declaration: using namespace std; #include <string> class myclass{ public: string myarray[] = {"a","b","c"}; }; and i'm getting error: error: brace-enclosed initializer not allowed here before ‘{’ token no, without complied c++11 compiler, cannot initialize member array in declaration. have initialize array member in constructor. , don't use open array if know number of elements initialize array.
Read more

cytoscape.js - How to add nodes to Dagre layout with Cytoscape -

-
i trying add nodes dagre graph through cityscape.js. problem nodes not painted. code have: cy.add({ group: "nodes", data: { id:"n0",name:"n0",weight: 75 }, }); $.ajax(url).done(function(jsontree){ var newnodes=[]; for(var i=1;i<6;i++){ //var child = jsontree.result[i]; newnodes.push( { group: "nodes", data: { id: "n"+i, name:"n"+i}}, { group: "edges", data: { id: "e"+i, source: "n0", target: "n"+i } } ); } cy.add(newnodes); any ideas? have used loaded function reload graph not work. not know if have use specific function reload graph. thank in advance. you need add positions new nodes. cytoscape.js docs : it important note positions of newly added nodes must defined when calling cy.add(). nodes can not placed in graph without valid position — otherwise not displayed. yo...
Read more