I'm setting up a new Meteor project and am having trouble working with collections on the client side. Right now what I'm building is an administrative page where a user can maintain a list of schools.
Running meteor mongo and db.schools.find() in Terminal returns a BSON object just as I would expect, however when I enter "Schools" in the Chrome console it returns Uncaught ReferenceError: Schools is not defined which is a bummer.
My project architecture (simplified to the bits that reference the School collection) is as follows:
client/
layouts/
admin/
schools.coffee
schools.jade
lib/
subscriptions.coffee
lib/
collections.coffee
server/
lib/
publications.coffee
The contents of each of these files (in the desirable load order) is:
1) lib/collections.coffee
1| Schools = new Mongo.Collection('schools')
2) server/lib/publications.coffee
1| Meteor.publish('schools'), ->
2| Schools.find()
3) client/lib/subscriptions.coffee
1| Meteor.subscribe('schools')
4) client/layouts/admin/schools.coffee
76| Template.admin_schools.helpers
77| schools: ->
78| Schools.find()
79| numSchools: ->
80| Schools.find().count()
81| hasSchools: ->
82| Schools.find().count() > 0
5) client/layouts/admin/schools.jade
4| h2.admin__header Schools
5| span.admin__record-count {{numSchools}} entries
...
22| table.admin__list
23| if hasSchools
24| each schools
25| tr.admin__list-item
26| td #{name}
I also have a form for new collection entries which calls Schools.insert, but the error there is the same.
When the page loads, I get the following error (likely because it is called first):
debug.js:41 Exception in template helper: ReferenceError: Schools is not defined
at Object.Template.admin_schools.helpers.numSchools
Those two errors, combined with the fact that I know there exists an entry in the collection, leads me to believe that the issue lies with the client side's awareness of the existence of the collection.
This discrepancy might possibly be due to load order (I am pretty sure I accounted for that by putting the important files in lib/ directories, though I would love a second opinion), or maybe due to a spelling/syntax mistake (though the absence of compile errors is puzzling). Maybe something completely different!
Thank you very much for your time and assistance, they are much appreciated.
Turns out that because this is CoffeeScript, placing an # before Schools = new Mongo.Collection('schools') in lib/collections.coffee makes Schools a global variable, thereby solving the problem! Pretty simple fix in the end :)
Special thanks to Kishor for helping troubleshoot.
Have you checked your subscriptions on created?
Related
For example, if I have a dataframe like this:
|sex| state_name| salary| www|
|---|------------------|-------|----|
| M| Ohio,California| 400|3000|
| M| Oakland| 70| 300|
| M|DF,Tbilisi,Calgary| 200|3500|
| M| Belice| 200|3000|
| m| Sofia,Helsinki| 800|7000|
I need to concatenate as a String the comma separated values in the "state_name" column with a delimiter specified by me. Also, I need to put a string at the end and the beginning of the generated string (the opposite of a strip() method or function).
For example, if I want an output like this:
|cool_city |
|--------------------------------|
|[***Ohio<-->California***] |
|[***Oakland***] |
|[***DF<-->Tbilisi<-->Calgary***]|
|[***Belice***] |
|[***Sofia<-->Helsinki***] |
The solution that I've already coded with Spark 3.1.1 is this:
df.select(concat(lit("[***"),
array_join(split(col("state_name"),","),"<-->"),lit("***]")).as("cool_city")).show()
The problem is that the computer where this will be running is using Spark 2.1.1 and the array_join() method isn't supported in this version (it's a pretty big project and upgrading the Spark version isn't over the table). Im pretty new using scala/spark and I don't know if there's another function that could help me emulating the array_join() use or if someone knows where to find the way to code a UDF with the same usefulness.
I would greatly appreciate your help!
I don't know Scala, but try this:
df.select(concat(lit("[***"),
concat_ws("<-->", split(col("state_name"), ",")),
lit("***]")).as("cool_city")).show()
UPDATE
Avoiding column split:
df.select(concat(lit("[***"),
regexp_replace(col("state_name"), ",", "<-->"),
lit("***]")).as("cool_city")).show()
I am using Apache Zeppelin (0.9.0) and Scala (2.11.12). I want to pull some data out of a dataframe and store it to InfluxDB, later to be visualized in Grafana, and cannot figure it out. I'm trying a naive approach with a foreach loop. The idea is to iterate through all rows, extract the columns I need, create a Point object (from this InfluxDB client library), and either send it to InfluxDB or add it to a list and then send all the points in bulk, after the loop.
The dataframe looks like this:
+---------+---------+-------------+-----+
|timestamp|sessionId|eventDuration|speed|
+---------+---------+-------------+-----+
| 1| ses1| 0.0| 50|
| 2| ses1| 1.0| 50|
| 3| ses1| 2.0| 50|
I've tried to do what is described above:
import scala.collection.mutable.ListBuffer
import spark.implicits._
import org.apache.spark.sql._
import com.paulgoldbaum.influxdbclient._
import scala.concurrent.ExecutionContext.Implicits.global
val influxdb = InfluxDB.connect("172.17.0.4", 8086)
val database = influxdb.selectDatabase("test")
var influxData = new ListBuffer[Point]()
dfAnalyseReport.foreach(row =>
{
val point = Point("acceleration")
.addTag("speedBin", row.getLong(3).toString)
.addField("eventDuration", row.getDouble(2))
influxData += point
}
)
val influxDataList = influxData.toList
database.bulkWrite(influxDataList)
The only thing I am getting here is a cryptic java.lang.ClassCastException with no additional info, neither in the notebook output nor in the logs of the Zeppelin Docker container. The error seems to be somewhere in the foreach, as it appears even when I comment out the last two lines.
I also tried adapting approach 1 from this answer, using a case class for columns, but to no avail. I got it to run without an error, but the resulting list was empty. Unfortunately I deleted that attempt. I could reconstruct it if necessary, but I've spent so much time on this I'm fairly certain I have some fundamental misunderstanding on how this should be done.
One further question: I also tried writing each Point to the DB as it was constructed (instead of in bulk). The only difference is that instead of appending to the ListBuffer I did a database.write(point) operation. When done outside of the loop with a dummy point, it goes through without a problem - the data ends up in InfluxDB - but inside the loop it results in org.apache.spark.SparkException: Task not serializable
Could someone point me in the right way? How should I tackle this?
I'd do it with the RDD map method and collect the results to a list:
val influxDataList = dfAnalyseReport.rdd.map(
row => Point("acceleration")
.addTag("speedBin", row.getInt(3).toString)
.addField("eventDuration", row.getDouble(2))
).collect.toList
I have a large text file which contains the page views of some Wikimedia projects. (You can find it here if you're really interested) Each line, delimited by a space, contains the statistics for one Wikimedia page. The schema looks as follows:
<project code> <page title> <num hits> <page size>
In Scala, using Spark RDDs or Dataframes, I wish to compute the total number of hits for each project, based on the project code.
So for example for projects with the code "zw", I would like to find all the rows that begin with project code "zw", and add up their hits. Obviously this should be done for all project codes simultaneously.
I have looked at functions like aggregateByKey etc, but the examples I found don't go into enough detail, especially for a file with 4 fields. I imagine it's some kind of MapReduce job, but how exactly to implement it is beyond me.
Any help would be greatly appreciated.
First, you have to read the file in as a Dataset[String]. Then, parse each string into a tuple, so that it can be easily converted to a Dataframe. Once you have a Dataframe, a simple .GroupBy().agg() is enough to finish the computation.
import org.apache.spark.sql.functions.sum
val df = spark.read.textFile("/tmp/pagecounts.gz").map(l => {
val a = l.split(" ")
(a(0), a(2).toLong)
}).toDF("project_code", "num_hits")
val agg_df = df.groupBy("project_code")
.agg(sum("num_hits").as("total_hits"))
.orderBy($"total_hits".desc)
agg_df.show(10)
The above snippet shows the top 10 project codes by total hits.
+------------+----------+
|project_code|total_hits|
+------------+----------+
| en.mw| 5466346|
| en| 5310694|
| es.mw| 695531|
| ja.mw| 611443|
| de.mw| 572119|
| fr.mw| 536978|
| ru.mw| 466742|
| ru| 463437|
| es| 400632|
| it.mw| 400297|
+------------+----------+
It is certainly also possible to do this with the older API as an RDD map/reduce, but you lose many of the optimizations that Dataset/Dataframe api brings.
i am playing a little bit with scala and i want to open a text file, read each line and save some of the fields in a hashset.
The input file will be something like this:
1 2 3
2 4 5
At first, i am just trying to store the first element of each column to a variable but nothing seems to happen.
My code is:
var id = 0
val textFile = sc.textFile(inputFile);
val nline = textFile.map(_.split(" ")).foreach(r => id = r(0))
I am using spark because i want to process bigger amount of data later, so i'm trying to get used to it. I am printing id but i get only 0.
Any ideas?
A couple of things:
First, inside map and foreach you are running code out on your executors. The id variable you defined is on the driver. You can pass variables to your executors using closures, but not the other way around. If you think about it, when you have 10 executors running through records simultaneously which value of ID would you expect to be returned?
Edit - foreach is an action
I mistakenly called foreach not an action below. It is an action that just lets your run arbitrary code against your rows. It is useful if you have your own code to save the result to a different data store for example. foreach does not bring any data back to the driver, so it does not help with your case.
End edit
Second, all of the spark methods you called are transformations, you haven't called an action yet. Spark doesn't actually run any code until an action is called. Instead it just builds a graph of the transformations you want to happen until you specify an action. Actions are things that require materializing a result either to provide data back to the driver or save them out somewhere like HDFS.
In your case, to get values back you will want to use an action like "collect" which returns all the values from the RDD back to the driver. However, you should only do this when you know there aren't going to be many values returned. If you are operating on 100 million records you do not want to try and pull them all back to the driver! Generally speaking you will want to only pull data back to the driver after you have processed and reduced it.
i am just trying to store the first element of each column to a
variable but nothing seems to happen.
val file_path = "file.txt"
val ds = ss.read.textFile(file_path)
val ar = ds.map(x => x.split(" ")).first()
val (x,y,z) = (ar(0),ar(1),ar(2))
You can access the first value of the columns with x,y,z as above.
With your file, x=1, y=2, z=3.
val ar1 = ds.map(x => x.split(" "))
val final_ds = ar.select($"value".getItem(0).as("col1") , $"value".getItem(1).as("col2") , $"value".getItem(2).as("col3")) // you can name the columns as like this
Output :
+----+----+----+
|col1|col2|col3|
+----+----+----+
| 1| 2| 3|
| 2| 4| 5|
+----+----+----+
You can run any kind of sql's on final_ds like a small sample below.
final_ds.select("col1","col2").where(final_ds.col("col1") > 1).show()
Output:
+----+----+
|col1|col2|
+----+----+
| 2| 4|
+----+----+
We are using Serenity with JBehave. We are facing issues when we have multiple scenario outlines with examples table in a single .feature file.
We have something like this in our Feature file:
Scenario Outline: title 1
Given
When
Then
Examples:
|data|
Scenario Outline: title 2
Given 2
When 2
Then 2
Examples:
|Data|
In this case after executing Examples of Scenario 1, it is considering Scenario outline 2 also as an input to scenario 1 example table, instead of treating it as a new scenario.
This is how the output looks like:
Scenario Outline: title 1
Given
When
Then
Examples:
|data|
|Scenario Outline: title 2|
|Given 2|
|When 2|
|Then 2|
|Examples:|
|Data|
Here are the versions of plugins:
jbehave.core.version - 3.9.5;
serenity.version - 1.0.47;
serenity.jbehave.version - 1.0.21
Can someone please help resolve this?
Note: I have seen posts where people had same issues when using scenario with examples table rather than using scenario outline with examples. Here I am using scenario outline only, but still have the same issue.