I have a cassandra table with 1000 rows. I am using the datastax java driver 2.1.8 with Cassandra 2.1.3
I have set the fetchSize to 10 for a prepared select statement. The code in Scala :
val stmt = preparedStatement.bind()
stmt.setFetchSize(10)
if(nextPage != null )
stmt.setPagingState(nextPage)
val rs = session.execute(stmt)
println( rs.all().count )
val nextPage = rs.getExecutionInfo().getPagingState()
I run this in a loop passing the nextPage value each time, starting with a null value.
But the result returned ignores the fetchSize although the PageState object is created according to the fetchSize. Result counts for each run of the loop are ...
1000
990
980
... so on.
what I want is the driver to return 10 results each time. What am I missing here ?
Calling all() will automatically consume all remaining pages. See javadoc on fetchMoreResults for details on how to manually fetch batches of the result.
Related
I'm trying to insert some data into a MariaDB database. I got two tables and I have to insert the rows (using a batch insert) into the first table and use the IDs of the newly-inserted rows to perform a second batch insert into the second table.
I'm doing so in Scala using Alpakka Slick. For the purpose of this question, let's call tests the main table and dependent the second one.
At the moment, my algorithm is as follows:
Insert the rows into tests
Fetch the ID of the first row in the batch using SELECT LAST_INSERT_ID();
Knowing the ID of the first row and the number of rows in the batch, compute by hand the other IDs and use them for the insertion in the second table
This works pretty well with only one connection at a time. However, I'm trying to simulate a scenario with multiple attempts to write simultaneously. To do that, I'm using Scala parallel collections and Akka Stream Source as follows:
// three sources of 10 random Strings each
val sources = Seq.fill(3)(Source(Seq.fill(10)(Random.alphanumeric.take(3).mkString))).zipWithIndex
val parallelSources: ParSeq[(Source[String, NotUsed], Int)] = sources.par
parallelSources.map { case (source, i) =>
source
.grouped(ChunkSize) // performs batch inserts of a given size
.via(insert(i))
.zipWithIndex
.runWith(Sink.foreach { case (_, chunkIndex) => println(s"Chunk $chunkIndex of source $i done") })
}
I'm adding an index to each Source just to use it a prefix in the data I write in the DB.
Here's the code of the insert Flow I've written so far:
def insert(srcIndex: Int): Flow[Seq[String], Unit, NotUsed] = {
implicit val insertSession: SlickSession = slickSession
system.registerOnTermination(() => insertSession.close())
Flow[Seq[String]]
.via(Slick.flowWithPassThrough { chunk =>
(for {
// insert data into `tests`
_ <- InsTests ++= chunk.map(v => TestProj(s"source$srcIndex-$v"))
// fetch last insert ID and connection ID
queryResult <- sql"SELECT CONNECTION_ID(), LAST_INSERT_ID();".as[(Long, Long)].headOption
_ <- queryResult match {
case Some((connId, firstIdInChunk)) =>
println(s"Source $srcIndex, last insert ID $firstIdInChunk, connection $connId")
// compute IDs by hand and write to `dependent`
val depValues = Seq.fill(ChunkSize)(s"source$srcIndex-${Random.alphanumeric.take(6).mkString}")
val depRows =
(firstIdInChunk to (firstIdInChunk + ChunkSize))
.zip(depValues)
.map { case (index, value) => DependentProj(index, value) }
InsDependent ++= depRows
case None => DBIO.failed(new Exception("..."))
}
} yield ()).transactionally
})
}
Where InsTests and InsDependent are Slick's TableQuery objects. slickSession creates a new session for each different insert and is defined as follows:
private def slickSession = {
val db = Database.forURL(
url = "jdbc:mariadb://localhost:3306/test",
user = "root",
password = "password",
executor = AsyncExecutor(
name = "executor",
minThreads = 20,
maxThreads = 20,
queueSize = 1000,
maxConnections = 20
)
)
val profile = slick.jdbc.MySQLProfile
SlickSession.forDbAndProfile(db, profile)
}
The problem is that the last insert IDs returned by the second step of the algorithm overlap. Every run of this app would print something like:
Source 2, last insert ID 6, connection 66
Source 1, last insert ID 5, connection 68
Source 0, last insert ID 7, connection 67
Chunk 0 of source 0 done
Chunk 0 of source 2 done
Chunk 0 of source 1 done
Source 2, last insert ID 40, connection 70
Source 0, last insert ID 26, connection 69
Source 1, last insert ID 27, connection 71
Chunk 1 of source 2 done
Chunk 1 of source 1 done
Chunk 1 of source 0 done
Where it looks like the connection is a different one for each Source, but the IDs overlap (Source 0 sees 7, source 1 sees 5, source 2 sees 2). It is correct that IDs start from 5, as I'm adding 4 dummy rows right after creating the tables (not shown in this question's code). Obviously, I see multiple rows in dependent with the same tests.id, which shouldn't happen.
It's my understanding that last insert IDs refer to a single connection. How is it possible that three different connections see overlapping IDs, considering that the entire flow is wrapped in a transaction (via Slick's transactionally)?
This happens with innodb_autoinc_lock_mode=1. As far as I've seen so far, it doesn't with innodb_autoinc_lock_mode=0, which makes sense, since InnoDB would lock tests until the whole batch insert terminates.
UPDATE after Georg's answer: For some other constraints in the project, I'd like the solution to be compatible with MariaDB 10.4, which, as far as I understand, doesn't feature INSERT...RETURNING. Additionally, Slick's ++= operator's support for returning is quite bad, as also reported here. I tested it on both MariaDB 10.4 and 10.5, and, according to the query logs, Slick does execute single INSERT INTO statements instead of a batch one. In my case, this is not quite acceptable, as I'm planning on writing several chunks of rows in a streaming fashion.
While I also understand that making assumptions about the auto-increment value being 1 is not ideal, we do have control over the Production setup and do not have multi-master replication.
You cannot generate subsequent values based on LAST_INSERT_ID():
There might be a second transaction which was rolled back running at the same time, so there will be a gap in your auto_incremented ID's.
Iterating over the number of rows by incrementing LAST_INSERT_ID value will not work, since it depends of value of session variable ##auto_increment_increment (which is especially in multi master replication not 1).
Instead, you should use RETURNING to get the ID's of inserted rows:
MariaDB [test]> create table t1 (a int not null auto_increment primary key);
Query OK, 0 rows affected (0,022 sec)
MariaDB [test]> insert into t1 (a) values (1),(3),(NULL), (NULL) returning a;
+---+
| a |
+---+
| 1 |
| 3 |
| 4 |
| 5 |
+---+
4 rows in set (0,006 sec)
I have data on mongodb. I want to reach that data from spark and make aggregations on executor not in driver.
But the thing that I couldnt understand is I m not sure that it is possible .
For example I have data on mongodb like this:
key value
---- -----
1 5
1 7
1 8
lets say I have 3 executor and when I select data in spark from mongo , each executor get 1 row from db.
So I want to get max value for each key. How is it possible without getting it to the driver?
As I know each executor has own value in itself and I cant get max value for a key. (Maybe I can write my own shuffle system and get same key in same executor but i m not sure is it efficient way)
my scala code is like this:
.....
df.coalesce(8).foreachPartition { iter =>
if (iter.hasNext) {
val con = ConnectionPool.getConnectionPool(dbName, url, user, pass, poolSize).getConnection
val batchSelectValue= con.prepareStatement(sql)
while (iter.hasNext) {
/*my logic*/
}
}
/* I want some aggregations in here*/
}
.....
Table structure -
create table test
(month integer,
year integer,
thresholds decimal(18,2)
);
Static insert for simulation -
insert into test(month,year,threshold) values(4,2021,100),(5,2021,98),(6,2021,99);
If I query postgres database using anorm, it works for regular queries. However on adding aggregate functions like max, the RowParser is not able to find the alias column.
val queryString =
"""select max(month) as monthyear from test
| where (month || '-' || year)
| = {inQuery}""".stripMargin
val inQuery1 = "'5-2021'"
The below on method causes the issue -
val latestInBenchmark = SQL(queryString).on("inQuery" -> inQuery1) // removing the on resolves the problem
logger.info("query for latest period ---> " + latestInBenchmark)
val latestYearMonthInInterval = database.withConnection(implicit conn => {
latestInBenchmark.as(SqlParser.int("monthyear").*)
})
Removing the on rectifies the problem and SqlParser.int(column-name) works as expected.
This also does not affect queries that use count aggregate function.
Error encountered :
(Validated intervals with sorting -> ,Failure(anorm.AnormException: 'monthyear' not found, available columns: monthyear, monthyear))
[error] c.b.ThresholdController - 'monthyear' not found, available columns: monthyear, monthyear
The error you have is a bit misleading but it means the query either returns a row with a null value or no row.
In your case I think the issue is the WHERE clause: you have put single quotes around the value but Anorm will do it by itself when using .on(...) or Anorm interpolation.
Thus, replace:
val inQuery1 = "'5-2021'"
By:
val inQuery1 = "5-2021"
Spark 2.2.0
I have the following code converted from SQL script. It has been running for two hours and it's still running. Even slower than SQL Server. Is anything not done correctly?
The following is the plan,
Push table2 to all executors
Partition table1 and distribute the partitions to executors.
And each row in table2/t2 joins (cross join) each partition of table1.
So the calculation on the result of the cross-join can be run distributed/parallelly. (I wanted to, for example suppose I have 16 executors, keep a copy of t2 on all the 16 executors. Then divide table 1 into 16 partitions, one for each executor. Then each executor do the calculation on one partition of table 1 and t2.)
case class Cols (Id: Int, F2: String, F3: BigDecimal, F4: Date, F5: String,
F6: String, F7: BigDecimal, F8: String, F9: String, F10: String )
case class Result (Id1: Int, ID2: Int, Point: Int)
def getDataFromDB(source: String) = {
import sqlContext.sparkSession.implicits._
sqlContext.read.format("jdbc").options(Map(
"driver" -> "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url" -> jdbcSqlConn,
"dbtable" -> s"$source"
)).load()
.select("Id", "F2", "F3", "F4", "F5", "F6", "F7", "F8", "F9", "F10")
.as[Cols]
}
val sc = new SparkContext(conf)
val table1:DataSet[Cols] = getDataFromDB("table1").repartition(32).cache()
println(table1.count()) // about 300K rows
val table2:DataSet[Cols] = getDataFromDB("table2") // ~20K rows
table2.take(1)
println(table2.count())
val t2 = sc.broadcast(table2)
import org.apache.spark.sql.{functions => func}
val j = table1.joinWith(t2.value, func.lit(true))
j.map(x => {
val (l, r) = x
Result(l.Id, r.Id,
(if (l.F1!= null && r.F1!= null && l.F1== r.F1) 3 else 0)
+(if (l.F2!= null && r.F2!= null && l.F2== r.F2) 2 else 0)
+ ..... // All kind of the similiar expression
+(if (l.F8!= null && r.F8!= null && l.F8== r.F8) 1 else 0)
)
}).filter(x => x.Value >= 10)
println("Total count %d", j.count()) // This takes forever, the count will be about 100
How to rewrite it with Spark idiomatic way?
Ref: https://forums.databricks.com/questions/6747/how-do-i-get-a-cartesian-product-of-a-huge-dataset.html
(Somehow I feel as if I have seen the code already)
The code is slow because you use just a single task to load the entire dataset from the database using JDBC and despite cache it does not benefit from it.
Start by checking out the physical plan and Executors tab in web UI to find out about the single executor and the single task to do the work.
You should use one of the following to fine-tune the number of tasks for loading:
Use partitionColumn, lowerBound, upperBound options for the JDBC data source
Use predicates option
See JDBC To Other Databases in Spark's official documentation.
After you're fine with the loading, you should work on improving the last count action and add...another count action right after the following line:
val table1: DataSet[Cols] = getDataFromDB("table1").repartition(32).cache()
// trigger caching as it's lazy in Dataset API
table1.count
The reason why the entire query is slow is that you only mark table1 to be cached when an action gets executed which is exactly at the end (!) In other words, cache does nothing useful and more importantly makes the query performance even worse.
Performance will increase after you table2.cache.count too.
If you want to do cross join, use crossJoin operator.
crossJoin(right: Dataset[_]): DataFrame Explicit cartesian join with another DataFrame.
Please note the note from the scaladoc of crossJoin (no pun intended).
Cartesian joins are very expensive without an extra filter that can be pushed down.
The following requirement is already handled by Spark given all the optimizations available.
So the calculation on the result of the cross-join can be run distributed/parallelly.
That's Spark's job (again, no pun intended).
The following requirement begs for broadcast.
I wanted to, for example suppose I have 16 executors, keep a copy of t2 on all the 16 executors. Then divide table 1 into 16 partitions, one for each executor. Then each executor do the calculation on one partition of table 1 and t2.)
Use broadcast function to hint Spark SQL's engine to use table2 in broadcast mode.
broadcast[T](df: Dataset[T]): Dataset[T] Marks a DataFrame as small enough for use in broadcast joins.
I am trying to figure out how to work with slick streaming. I use slick 3.0.0 with postgres driver
The situation is following: server have to give client sequences of data split into chunks limited by size(in bytes). So, I wrote following slick query:
val sequences = TableQuery[Sequences]
def find(userId: Long, timestamp: Long) = sequences.filter(s ⇒ s.userId === userId && s.timestamp > timestamp).sortBy(_.timestamp.asc).result
val seq = db.stream(find(0L, 0L))
I combined seq with akka-streams Source, wrote custom PushPullStage, that limits size of data(in bytes) and finishes upstream when it reaches size limit. It works just fine. The problem is - when I look into postgres logs, I see query like that
select * from sequences where user_id = 0 and timestamp > 0 order by timestamp;
So, at first glance it appears to be much (and unnecessary) database querying going on, only to use a few bytes in each query. What is the right way to do streaming with Slick so as to minimize database querying and to make best use of the data transferred in each query?
The "right way" to do streaming with Slick and Postgres includes three things:
Must use db.stream()
Must disable autoCommit in JDBC-driver. One way is to make the query run in a transaction by suffixing .transactionally.
Must set fetchSize to be something else than 0 or else postgres will push the whole resultSet to the client in one go.
Ex:
DB.stream(
find(0L, 0L)
.transactionally
.withStatementParameters(fetchSize = 1000)
).foreach(println)
Useful links:
https://github.com/slick/slick/issues/1038
https://github.com/slick/slick/issues/809
The correct way to stream in Slick is as provided in documentation is
val q = for (c <- coffees) yield c.image
val a = q.result
val p1: DatabasePublisher[Blob] = db.stream(a.withStatementParameters(
rsType = ResultSetType.ForwardOnly,
rsConcurrency = ResultSetConcurrency.ReadOnly,
fetchSize = 1000 /*your fetching size*/
).transactionally)