apache spark - SparkSQL, Thrift Server and Tableau -

-

i wondering if there way make sparksql table in sqlcontext directly visible other processes, example tableau.

i did research on thrift server, didn't find specific explanation it. middleware between hive(database) , application(client)? if so, need write hive table in spark program?

when use beeline check tables thrift server, there's field istemptable. know mean? i'm guessing temp table in sqlcontext of thrift server, because read spark driver program , cached tables visible through multiple programs. confusion here is, if driver program, workers?

to summarize,

  1. where should write dataframe, or tables in sqlcontext to? method should use(like dataframe.write.mode(savemode.append).saveastable())?
  2. should default settings used thrift server? or changes necessary?

thanks

i assume you've moved on now, comes across answer, thrift server broker between jdbc connection , sparksql.

once you've got thrift running (see spark docs basic intro), connect on jdbc using hive jdbc drivers thrift, , in turn relays sql queries spark using hivecontext.

if have full hive metastore , running, should able see hive tables in jdbc client immediately, otherwise can create tables on demand running commands in jdbc client:

create table data1 using org.apache.spark.sql.parquet options (path "/path/to/parquetfile"); create table data2 using org.apache.spark.sql.json options (path "/path/to/jsonfile");

hope helps little.


Comments

Post a Comment

Popular posts from this blog

python - pip install -U PySide error -

-
does know how avoid following error when running pip install -u pyside per official website instructions here: https://pypi.python.org/pypi/pyside/#installing-pyside-on-a-mac-os-x-system note have done brew install qt successfully. you using pip version 7.0.3, version 7.1.2 available. should consider upgrading via 'pip install --upgrade pip' command. collecting pyside using cached pyside-1.2.4.tar.gz installing collected packages: pyside running setup.py install pyside complete output command /applications/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-build-cpxemt/pyside/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/dy/ttgqhqqx3g9bsbsnqnxsk2z80000gq/t/pip-0r7hri-record/install-record.txt --single-version-externally-managed --compile: ...
Read more

arrays - C++ error: a brace-enclosed initializer is not allowed here before ‘{’ token -

-
simple question: i'm trying initialize array within c++ class declaration: using namespace std; #include <string> class myclass{ public: string myarray[] = {"a","b","c"}; }; and i'm getting error: error: brace-enclosed initializer not allowed here before ‘{’ token no, without complied c++11 compiler, cannot initialize member array in declaration. have initialize array member in constructor. , don't use open array if know number of elements initialize array.
Read more

cytoscape.js - How to add nodes to Dagre layout with Cytoscape -

-
i trying add nodes dagre graph through cityscape.js. problem nodes not painted. code have: cy.add({ group: "nodes", data: { id:"n0",name:"n0",weight: 75 }, }); $.ajax(url).done(function(jsontree){ var newnodes=[]; for(var i=1;i<6;i++){ //var child = jsontree.result[i]; newnodes.push( { group: "nodes", data: { id: "n"+i, name:"n"+i}}, { group: "edges", data: { id: "e"+i, source: "n0", target: "n"+i } } ); } cy.add(newnodes); any ideas? have used loaded function reload graph not work. not know if have use specific function reload graph. thank in advance. you need add positions new nodes. cytoscape.js docs : it important note positions of newly added nodes must defined when calling cy.add(). nodes can not placed in graph without valid position — otherwise not displayed. yo...
Read more