spark worker insufficient memory -

-

i have spark/cassandra setup using spark cassandra java connector query on table. far, have 1 spark master node (2 cores) , 1 worker node (4 cores). both of them have following spark-env.sh under conf/: 

#!/usr/bin/env bash export spark_local_ip=127.0.0.1 export spark_master_ip="192.168.4.134" export spark_worker_memory=1g export spark_executor_memory=2g

here spark execution code:

    sparkconf conf = new sparkconf();     conf.setappname("testing");     conf.setmaster("spark://192.168.4.134:7077");     conf.set("spark.cassandra.connection.host", "192.168.4.129");     conf.set("spark.logconf", "true");     conf.set("spark.driver.maxresultsize", "50m");     conf.set("spark.executor.memory", "200m");     conf.set("spark.eventlog.enabled", "true");     conf.set("spark.eventlog.dir", "/tmp/");     conf.set("spark.executor.extraclasspath", "/home/enlighted/ebd.jar");     conf.set("spark.cores.max", "1");     javasparkcontext sc = new javasparkcontext(conf);       javardd<string> cassandrarowsrdd = cassandrajavautil.javafunctions(sc).cassandratable("testing", "ec")     .map(new function<cassandrarow, string>() {         private static final long serialversionuid = -6263533266898869895l;         @override         public string call(cassandrarow cassandrarow) throws exception {             return cassandrarow.tostring();         }     });     system.out.println("data cassandrarows: \n" + stringutils.join(cassandrarowsrdd.toarray(), "\n"));     sc.close();

now start master spark on first node , worker on second node , thhen run above code. creates executor thread on worker, see following message on application side logs:

[timer-0] warn org.apache.spark.scheduler.taskschedulerimpl  - initial job has not accepted resources; check cluster ui ensure workers registered , have sufficient resources

now keeping same setup, when run spark/sbin/start-all.sh on master server, creates master instance worker instance on first node. again when run same code , worker assigned new worker, works fine.

what issue original worker running on node different master node?

figured out root cause. master randomly assigning port worker communication. because of firewall on master, worker couldn't send out messages master (maybe resource details). weird worker didn't bother throw error also.


Comments

Post a Comment

Popular posts from this blog

python - pip install -U PySide error -

-
does know how avoid following error when running pip install -u pyside per official website instructions here: https://pypi.python.org/pypi/pyside/#installing-pyside-on-a-mac-os-x-system note have done brew install qt successfully. you using pip version 7.0.3, version 7.1.2 available. should consider upgrading via 'pip install --upgrade pip' command. collecting pyside using cached pyside-1.2.4.tar.gz installing collected packages: pyside running setup.py install pyside complete output command /applications/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-build-cpxemt/pyside/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-0r7hri-record/install-record.txt --single-version-externally-managed --compile: ...
Read more

arrays - C++ error: a brace-enclosed initializer is not allowed here before ‘{’ token -

-
simple question: i'm trying initialize array within c++ class declaration: using namespace std; #include <string> class myclass{ public: string myarray[] = {"a","b","c"}; }; and i'm getting error: error: brace-enclosed initializer not allowed here before ‘{’ token no, without complied c++11 compiler, cannot initialize member array in declaration. have initialize array member in constructor. , don't use open array if know number of elements initialize array.
Read more

cytoscape.js - How to add nodes to Dagre layout with Cytoscape -

-
i trying add nodes dagre graph through cityscape.js. problem nodes not painted. code have: cy.add({ group: "nodes", data: { id:"n0",name:"n0",weight: 75 }, }); $.ajax(url).done(function(jsontree){ var newnodes=[]; for(var i=1;i<6;i++){ //var child = jsontree.result[i]; newnodes.push( { group: "nodes", data: { id: "n"+i, name:"n"+i}}, { group: "edges", data: { id: "e"+i, source: "n0", target: "n"+i } } ); } cy.add(newnodes); any ideas? have used loaded function reload graph not work. not know if have use specific function reload graph. thank in advance. you need add positions new nodes. cytoscape.js docs : it important note positions of newly added nodes must defined when calling cy.add(). nodes can not placed in graph without valid position — otherwise not displayed. yo...
Read more