scala - Spark Mllib .toBlockMatrix results in matrix of 0.0 -

-

i trying create block matrix input data file. have managed data read data file , stored in indexedrowmatrix , coordinatematrix format correct.

when use .toblockmatrix on coordinatematrix result block matrix containing 0.0 same dimensions coordinatematrix.

i using version 1.5.0-cdh5.5.0

import org.apache.spark.sparkconf import org.apache.spark.sparkcontext import org.apache.spark.sparkcontext._ import org.apache.spark.mllib.linalg._ import org.apache.spark.mllib.linalg.vector import org.apache.spark.mllib.linalg.distributed.coordinatematrix import org.apache.spark.mllib.linalg.distributed.indexedrowmatrix import org.apache.spark.mllib.linalg.distributed.indexedrow import org.apache.spark.mllib.linalg.distributed.blockmatrix  val conf = new sparkconf().setmaster("local").setappname("transpose"); val sc = new sparkcontext(conf)  val datardd = sc.textfile("/user/cloudera/data/data.txt").map(line => vectors.dense(line.split(" ").map(_.todouble))).zipwithindex.map(_.swap) //format of datardd rdd[(long, vector)]  val rows = datardd.map{case(k,v) => indexedrow(k,v)} //format of rows rdd[indexedrow]  val mat: indexedrowmatrix = new indexedrowmatrix(rows) val coordmat: coordinatematrix = mat.tocoordinatematrix() val blockmat: blockmatrix = coordmat.toblockmatrix().cache()

the data file 2 columns sixty rows of integers.

140 123 141 310 310 381 480 321 ... ...

update:

i've done investigating , have discovered groupbykey function not working correctly, preventing blockmatrix being formed correctly. still not know why groupbykey, join, , groupby not working , returning empty result.

i have solved problem removing lines of code:

val conf = new sparkconf().setmaster("local").setappname("transpose") val sc = new sparkcontext(conf)

i found answer in below linked page in comment farzad nozarian,

unable count words using reducebykey((v1,v2) => v1 + v2) scala function in spark

as side-note might people getting empty results .groupbykey, .reducebykey, .join, etc.


Comments

Post a Comment

Popular posts from this blog

how to insert data php javascript mysql with multiple array session 2 -

-
i have method method discussed , successful data entered 1 when more 1 can not database this programs my php form : <form class="form-horizontal" id="form-produk"> <div class="control-group" > <label class="control-label" for="kelas">kategori</label> <div class="controls"> <select class='form-control' name='id_kategori'> <?php $tampil=mysql_query("select * kategori order nama_kategori"); if ($id_kat==0){ echo "<option value=0 selected>- pilih kategori -</option>"; } while($w=mysql_fetch_array($tampil)){ if ($id_kat==$w[id_kategori]){ echo "<option value=$w[id_kategori] selected>$w[nama_kategori]</option>"; } else{ echo "<option value=$w[id_kategori]>$...
Read more

multithreading - Exception in Application constructor -

-
i wanna write snake game javafx , there exception don't know , want know how fix it. ( know it's not complete yet ) and want know , class extends application ( start override ) main in other programs? see , here not static void main bc didn't need, if want add main shoud do? it's exeption... exception in application constructor exception in thread "main" java.lang.nosuchmethodexception: main_snake.main([ljava.lang.string;) @ java.lang.class.getmethod(class.java:1819) @ com.intellij.rt.execution.application.appmain.main(appmain.java:125) and code : import javafx.animation.animationtimer; import javafx.application.application; import javafx.event.eventhandler; import javafx.scene.scene; import javafx.scene.canvas.canvas; import javafx.scene.canvas.graphicscontext; import javafx.scene.input.keyevent; import javafx.scene.layout.borderpane; import javafx.scene.paint.color; import javafx.stage.stage; import java.util.arraylist; /** * creat...
Read more

windows - CertCreateCertificateContext returns CRYPT_E_ASN1_BADTAG / 8009310b -

-
i realize similar post others (e.g. this one ), there details missing posts might significant case. to start with, here's simplified program: #include "stdafx.h" #include <windows.h> #include <wincrypt.h> int _tmain(int argc, _tchar* argv[]) { // usage: certextract certpath char keyfile[] = "c:\\certificates\\public.crt"; byte lp[65536]; security_attributes sa; handle hkeyfile; dword bytes; pccert_context certcontext; sa.nlength = sizeof(sa); sa.lpsecuritydescriptor = null; sa.binherithandle = false; hkeyfile = createfile(keyfile, generic_read, file_share_read, &sa, open_existing, file_attribute_normal, null); if (hkeyfile) { if (readfile(hkeyfile, lp, getfilesize(hkeyfile, null), &bytes, null) && bytes > 0) { certcontext = certcreatecertificatecontext(x509_asn_encoding, lp, bytes); if (certcontext) { printf("yay...
Read more