I'm trying to save the results into a text file using:
results.collect().foreach { rule => println("[" + rule.antecedent.mkString(",") + "=>" + rule.consequent.mkString(",") + "]," + rule.confidence)}.saveAsTextFile("/user/cloudera/rules")
But I got this error:
<console>:43: error: value saveAsTextFile is not a member of Unit
results.collect().foreach { rule => println("[" + rule.antecedent.mkString(",") + "=>" + rule.consequent.mkString(",") + "]," + rule.confidence)}.saveAsTextFile("/user/cloudera/rules")
What I need to change? I don't understand the error...
If you want to save use:
results.map { rule =>
"[" + rule.antecedent.mkString(",") + "=>" + rule.consequent.mkString(",") + "]," + rule.confidence
}.saveAsTextFile("/user/cloudera/rules")
The foreach operation returns a Unit, which is a stand-in for an empty return value:
scala> val a = (0 to 5).foreach(println)
0
1
2
3
4
5
a: Unit = ()
If you want to save your original results you will need to use something like:
results.collect.saveAsTextFile(...)
Otherwise if you are trying to apply an operation to each element of results.collect and then save the result, then you will need to use a map and do something else to each element (because print also returns a Unit).
I got the same error message when I had initialized the variable with spark.read instead of sc. One of them returns a DataSet that doesn't have the saveAsTextFile method, the other one returns an RDD that does. Hope it helps someone.
Related
I´m using Spark 2.2.1 with Scala 2.11.12 version as a language to generate a recursive algorithm. First, I tried an implementation using RDD but the time when I used a lot of data was too much. I have made a new version using DataFrames but with very little data it takes too much, taking less data in each iteration than in the previous iteration.
I have tried to cache variables in different ways (types of persistence included), to use checkpoints in different moments, using the repartition method with different values and in different functions, and nothing works.
The code starts looking for the minimum distance between the points that make up the matrix (matrix is a DataFrame):
println("Finding minimum:")
val minDistRes = matrix.select(min("dist")).first().getFloat(0)
val clusterRes = matrix.where($"dist" === minDistRes)
println(s"New minimum:")
clusterRes.show(1)
Then, save the coordenates to the points for later calculations:
val point1 = clusterRes.first().getInt(0)
val point2 = clusterRes.first().getInt(1)
After, made several filters to use them in the new points generated in the next iteration (the creation of a broadcast variable is necessary to be able to access this data in a later map):
matrix = matrix.where("!(idW1 == " + point1 +" and idW2 ==" + point2 + " )").cache()
val dfPoints1 = matrix.where("idW1 == " + point1 + " or idW2 == " + point1).cache()
val dfPoints2 = matrix.where("idW1 == " + point2 + " or idW2 == " + point2).cache()
val dfPoints2Broadcast = spark.sparkContext.broadcast(dfPoints2)
val dfUnionPoints = dfPoints1.union(dfPoints2).cache()
val matrixSub = matrix.except(dfUnionPoints).cache()
Continued with the calculation of the new points and I return the matrix that will be used recursively by the algorithm:
val newPoints = dfPoints1.map{
r => val distAux = dfPoints2Broadcast.value.where("idW1 == " + r.getInt(0) +
" or idW1 == " + r.getInt(1) + " or idW2 == " + r.getInt(0) + " or idW2 == " +
r.getInt(1)).first().getFloat(2)
(newIndex.toInt, filterDF(r.getInt(0),r.getInt(1), point1, point2), math.min(r.getFloat(2), distAux))
}.asInstanceOf[Dataset[Row]]
matrix = matrixSub.union(newPoints)
Finalize each iteration caching the matrix variable and realized a checkpoint every so often:
matrix.cache()
if (a % 5 == 0)
matrix.checkpoint()
I am successfully loading files into Spark, from S3, through the following code. It's working, however I am noticing that there is a delay between 1 file and another, and they are loaded sequentially. I would like to improve this by loading in parallel.
// Load files that were loaded into firehose on this day
var s3Files = spark.sqlContext.read.schema(schema).json("s3n://" + job.awsaccessKey + ":" + job.awssecretKey + "#" + job.bucketName + "/" + job.awss3RawFileExpression + "/" + year + "/" + monthCheck + "/" + dayCheck + "/*/").rdd
// Apply the schema to the RDD, here we will have duplicates
val usersDataFrame = spark.createDataFrame(s3Files, schema)
usersDataFrame.createOrReplaceTempView("results")
// Clean and use partition by the keys to eliminate duplicates and get latest record
var results = spark.sql(buildCleaningQuery(job, "results"))
results.createOrReplaceTempView("filteredResults")
val records = spark.sql("select count(*) from filteredResults")
I have also tried loading through the textFile() method, however then I am having problems converting RDD[String] to RDD[Row] because afterwards I would need to move on to use Spark SQL. I am using it in the following manner;
var s3Files = sparkContext.textFile("s3n://" + job.awsaccessKey + ":" + job.awssecretKey + "#" + job.bucketName + "/" + job.awss3RawFileExpression + "/" + year + "/" + monthCheck + "/" + dayCheck + "/*/").toJavaRDD()
What is the ideal manner to load JSON files (Multiple files around 50MB each) into Spark? I would like to validate the properties against a schema, so I would later on be able to Spark SQL queries to clean data.
What's going on is that DataFrame is being converted into RDD and then into DataFrame again, which then loses the partitioning information.
var s3Files = spark
.sqlContext
.read.schema(schema)
.json(...)
.createOrReplaceTempView("results")
should be sufficient, and the partitioning information should still be present, allowing json files to be loaded concurrently.
This is first issue i am posting so apologies if i miss some info and mediocre formatting. I can update if required.
I will try to add as many details as possible. I have a not so optimized Spark Job which converts RDBMS data to graph nodes and relations in Neo4j.
To do this. Here is the steps i follow:
create a denormalized dataframe 'data' with spark sql and joins.
Foreach row in 'data' run a graphInsert function which does the following:
a. read contents of the row b. formulate a neo4j cypher query (We use Merge command so that we have have only one City e.g. Chicago created in Neo4j when Chicago will be present in multiple lines in RDBMS table) c. connect to neo4j d. execute the query e. disconnect from neo4j
Here is the list of problems i am facing.
Inserts are slow.
I know Merge query is slower than create but is there another way to do this instead of connecting and disconnecting for every record? This was my first draft code and maybe i am struggling how i will use one connection to insert from multiple threads on different spark worker nodes. Hence connecting and disconnecting for every record.
The job is not scalable. It only runs fine with 1 core. As soon as i run the job with 2 spark cores i suddenly get 2 cities with same name, even when i am running merge queries. e.g. There are 2 Chicago cities which violates the use of Merge. I am assuming that Merge functions something like "Create if not exist".
I dont know if my implementation is wrong in neo4j part or spark. If anyone can direct me to any documentation which helps me implement this on a better scale it will be helpful as i have a big spark cluster which i need to utilize at full potential for this job.
If you are interested to look at code instead of algorithm. Here is graphInsert implementation in scala:
class GraphInsert extends Serializable{
var case_attributes = new Array[String](4)
var city_attributes = new Array[String](2)
var location_attributes = new Array[String](20)
var incident_attributes = new Array[String](20)
val prop = new Properties()
prop.load(getClass().getResourceAsStream("/GraphInsertConnection.properties"))
// properties Neo4j
val url_neo4j = prop.getProperty("url_neo4j")
val neo4j_user = prop.getProperty("neo4j_user")
val neo4j_password = prop.getProperty("neo4j_password")
def graphInsert(data : Row){
val query = "MERGE (d:CITY {name:city_attributes(0)})\n" +"MERGE (a:CASE { " + case_attributes(0) + ":'" +data(11) + "'," +case_attributes(1) + ":'" +data(13) + "'," +case_attributes(2) + ":'" +data(14) +"'}) \n" +"MERGE (b:INCIDENT { " + incident_attributes(0) + ":" +data(0) + "," +incident_attributes(1) + ":" +data(2) + "," +incident_attributes(2) + ":'" +data(3) + "'," +incident_attributes(3) + ":'" +data(8)+ "'," +incident_attributes(4) + ":" +data(5) + "," +incident_attributes(5) + ":'" +data(4) + "'," +incident_attributes(6) + ":'" +data(6) + "'," +incident_attributes(7) + ":'" +data(1) + "'," +incident_attributes(8) + ":" +data(7)+"}) \n" +"MERGE (c:LOCATION { " + location_attributes(0) + ":" +data(9) + "," +location_attributes(1) + ":" +data(10) + "," +location_attributes(2) + ":'" +data(19) + "'," +location_attributes(3) + ":'" +data(20)+ "'," +location_attributes(4) + ":" +data(18) + "," +location_attributes(5) + ":" +data(21) + "," +location_attributes(6) + ":'" +data(17) + "'," +location_attributes(7) + ":" +data(22) + "," +location_attributes(8) + ":" +data(23)+"}) \n" +"MERGE (a) - [r1:"+relation_case_incident+"]->(b)-[r2:"+relation_incident_location+"]->(c)-[r3:belongs_to]->(d);"
println(query)
try{
var con = DriverManager.getConnection(url_neo4j, neo4j_user, neo4j_password)
var stmt = con.createStatement()
var rs = stmt.executeQuery(query)
con.close()
}catch{
case ex: SQLException =>{
println(ex.getMessage)
}
}
}
def operations(sqlContext: SQLContext){
....
#Get 'data' before this step
city_attributes = entity_metadata.filter(entity_metadata("source_name") === "tb_city").map(x =>x.getString(5)).collect()
case_attributes = entity_metadata.filter(entity_metadata("source_name") === "tb_case_number").map(x =>x.getString(5)).collect()
location_attributes = entity_metadata.filter(entity_metadata("source_name") === "tb_location").map(x =>x.getString(5)).collect()
incident_attributes= entity_metadata.filter(entity_metadata("source_name") === "tb_incident").map(x =>x.getString(5)).collect()
data.foreach(graphInsert)
}
object GraphObject {
def main(args: Array[String]) {
val conf = new SparkConf()
.setAppName("GraphNeo4j")
.setMaster("xyz")
.set("spark.cores.max","2")
.set("spark.executor.memory","10g")
Logger.getLogger("org").setLevel(Level.ERROR)
Logger.getLogger("akka").setLevel(Level.ERROR)
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val graph = new GraphInsert()
graph.operations(sqlContext)
}
}
Whatever you write inside the closure i.e it needs to be executed on Worker gets distributed.
You can read more about it here : http://spark.apache.org/docs/latest/programming-guide.html#understanding-closures-a-nameclosureslinka
And as you increase the number of cores, I think it must not effect the application because if you do not specify it ! then it takes the greedy approach ! I hope this document helps .
I am done improving the process but nothing could make it as fast as LOAD command in Cypher.
Hope this helps someone though:
use foreachPartition instead of foreach gives significant gain while doing such process. Also adding periodic commit using cypher.
//Post and Pre-Increment Test
func FindValueOFC() -> String
{
var a : Int = 10
var b : Int = 20
var c = a++ + a++ + b++ + b++ + ++a + ++b
return "The value Of variable C is \(c)"
}
let whatsTheValueOfC = FindValueOFC()
println(whatsTheValueOfC)
Why does this program prints out The value Of variable C is 98?
Logically it should be 96, as a++ + a++ + b++ + b++ + ++a + ++b can be translated to 10+11+20+21+12+22 = 96
Please don't ever do this in a real program. It leads to undefined behavior when you mix multiple mutators in the same expression. There is nothing in the language that guarantees that the pre-increment operator happens immediately before the value is accessed or that the post-increment operator happens immediately after the variable is accessed.
The compiler could have done the pre-increment of a and b first, added up all of the values, and then applied the post-increments. This would have given the result of
11 + 11 + 21 + 21 + 11 + 21 = 96
The point being that many answers are possible and valid, hence the name undefined behavior. It is possible that you would get different answers with different levels of compiler optimization, which could lead to very puzzling differences between your testing and shipping versions of your app.
I'm trying to understand what's going on in this recursive function. It reverses a String, but I don't quite get how these separate return calls get assembled into one string at the end.
def reverse(string: String): String = {
if (string.length() == 0)
return string
return reverse(string.substring(1)) + string.charAt(0)
}
I've analysed the function by adding in print statement, and while I kind of understand how it works (conceptually), I don't understand, well... how it works.
For instance, I know that each cycle of recursion pushes things into the stack.
So, I would expect reverse("hello"), to become a stack of
o
l
l
e
h
But it must be more complex than that, as the recursive call is return reverse(string.substring(1)) + string.charAt(0). So is the stack actually
o,
l, o
l, lo
e, llo
H, ello
?
How does that get turned into the single string we expect?
The stack contains all local variables, as well as any temporary result in an expression where the recursion appears (though those are pushed on the stack even without recursion, because JVM is a stack machine) and, of course, the point where the code execution should resume on return.
In this case, the recursive call is the whole expression (that is, nothing is computed before reverse on the expression it appears). So the only thing besides the code pointer is string. At the deepest level of recursion, the stack will look like this:
level string
5 (empty string)
4 o
3 lo
2 llo
1 ello
0 hello
So when the call to level 5 returns, level 4 will finish computing the expression that reverse is a part of, reverse(string.substring(1)) + string.charAt(0). The value of reverse(string.substring(1)) is the empty string, and the value of string.charAt(0) is o (since the value of string on level 4 is o). The result is o, which is returned.
On level 3, it concatenates the return value from level 4 (o) with string.charAt(0) for string equal to lo, which is l, resulting in ol.
On level 2, it concatenates ol with l, giving oll.
Level 1 concatenates oll with e, returning olle.
Level 0, finally, concatenates olle with h, returning olleh to its caller.
On a final note, when a call is made, what is pushed into the stack is the return point for the code and the parameters. So hello is the parameter to reverse, which is pushed on the stack by reverse's caller.
Use the substitution model to work through the problem:
reverse("hello") =
(reverse("ello") + 'h') =
((reverse("llo") + 'e') + 'h') =
(((reverse("lo") + 'l') + 'e') + 'h') =
((((reverse("o") + 'l') + 'l') + 'e') + 'h') =
(((((reverse("") + 'o') + 'l') + 'l') + 'e') + 'h') =
((((("" + 'o') + 'l') + 'l') + 'e') + 'h') =
(((("o" + 'l') + 'l') + 'e') + 'h') =
((("ol" + 'l') + 'e') + 'h') =
(("oll" + 'e') + 'h') =
("olle" + 'h') =
"olleh"
Аdd a couple of tips on how to make your code better:
Don't use return in functions because function automaticaly return result of last evaluated line
String is a List of Char's, and you can replace string.substring(1) => string.tai, and string.charAt(0) => string.head
If you call immutable method, like length, size or etc you can omit the parentheses string.length() === string.length
That last version of your single line function:
def reverse(s: String): String = if (s.size == 0) s else reverse(s.tail) + s.head