Transactions With JSONCollection instead of BSONCollection - scala

I have a problem to make Transaction via JSONCollection, I getting the following error:
JsResultException(errors:List((,List(JsonValidationError(List(CommandError[code=14, errmsg=BSON field 'OperationSessionInfo.txnNumber' is the wrong type 'int', expected type 'long', doc: {"operationTime":{"$time":1596894245,"$i":5,"$timestamp":{"t":1596894245,"i":5}},"ok":0,"errmsg":"BSON field 'OperationSessionInfo.txnNumber' is the wrong type 'int', expected type 'long'","code":14,"codeName":"TypeMismatch","$clusterTime":{"clusterTime":{"$time":1596894245,"$i":5,"$timestamp":{"t":1596894245,"i":5}},"signature":{"hash":{"$binary":"0000000000000000000000000000000000000000","$type":"00"},"keyId":0}}}]),WrappedArray())))))
I tried to change my project to BSONCollection but got some troubles, maybe there solution to overcome the above error with JSONCollection.
Also the exceptions occurs on testing update method, but checking the insertOneViaTransaction and setRuleAsInactiveViaTransaction is finished with success
This is my code for Transaction:
Update:
def update(oldRule: ExistRuleDto): Future[UpdateResult] = {
val transaction = (collection: JSONCollection) => for {
newRule <- dao.insertOneViaTransaction(collection,oldRule.toUpdatedRule) // insert new with ref to old
oldRule <- dao.setRuleAsInactiveViaTransaction(collection,oldRule.id)
} yield UpdateResult(oldRule, newRule)
makeTransaction[UpdateResult](transaction)
}
makeTransaction:
def makeTransaction[Out](block: JSONCollection => Future[Out]): Future[Out] = for {
dbWithSession <- dao.collection.db.startSession()
dbWithTx <- dbWithSession.startTransaction(None)
coll = dbWithTx.collection[JSONCollection](dao.collection.name)
// Operations:
res <- block(coll)
_ <- dbWithTx.commitTransaction()
_ <- dbWithSession.endSession()
} yield res
insertOneViaTransaction:
def insertOneViaTransaction(collection: JSONCollection, rule: Rule): Future[Rule] = {
collection.insert.one(rule).map {
case DefaultWriteResult(true, 1, _, _, _, _) => rule
case err => throw GeneralDBError(s"$rule was not inserted, something went wrong: $err")
}.recover {
case WriteResult.Code(11000) => throw DuplicationError(s"$rule exist on DB")
case err => throw GeneralDBError(err.getMessage)
}
}
setRuleAsInactiveViaTransaction:
def setRuleAsInactiveViaTransaction(collection: JSONCollection, ruleId: BSONObjectID): Future[Rule] = {
collection.findAndUpdate(
Json.obj(s"${Rule.ID}" -> ruleId),
Json.obj(
"$set" -> Json.obj(s"${Rule.Metadata}.${Metadata.Active}" -> false),
"$unset" -> Json.obj(s"${Rule.Metadata}.${Metadata.LastVersionExists}" -> "")),
fetchNewObject = true, upsert = false, sort = None, fields = None, bypassDocumentValidation = false, writeConcern = WriteConcern.Acknowledged, maxTime = None, collation = None, arrayFilters = Nil
).map(el => el.result[Rule].getOrElse {
val msg = s"Operation fail for updating ruleId = $ruleId"
logger.error(msg)
throw GeneralUpdateError(msg)
})
}
I'm using the following dependencies:
Play:
"com.typesafe.play" % "sbt-plugin" % "2.7.2
Reactivemongo:
"org.reactivemongo" %% "play2-reactivemongo" % "0.18.8-play27"

Solve it. (Not with compact)
Serializers:
implicit object JsValueHandler extends BSONHandler[BSONValue, JsValue] {
implicit override def read(bson: BSONValue): JsValue = BSONFormats.toJSON(bson)
implicit override def write(j: JsValue): BSONValue = BSONFormats.toBSON(j).get
}
asTransaction:
def asTransaction[Out](block: BSONCollection => Future[Out]): Future[Out] = {
for {
dbWithSession <- collection.db.startSession()
dbWithTx <- dbWithSession.startTransaction(None)
collectionWithTx = dbWithTx.collection[BSONCollection](collection.name)
out <- block(collectionWithTx)
_ <- dbWithTx.commitTransaction()
_ <- dbWithSession.endSession()
} yield out
}.recover {
case ex: Exception =>
logger.warn(s"asTransaction failed with ex: ${ex.getMessage}, rollback to previous state...")
throw GeneralDBErrorOnTx(ex.getMessage)
}
transaction example:
def `change visibility of ExistsRules and insert UpdateEvents`(oldRules: List[Rule], active: Boolean): Future[Unit] = {
ruleDao.asTransaction { collectionTx =>
for {
// (1) - $active old Rules
_ <- ruleDao.updateManyWithBsonCollection(
collectionTx,
filter = BSONDocument(s"${Rule.ID}" -> BSONDocument("$in" -> oldRules.map(_._id))),
update = BSONDocument("$set" -> BSONDocument(s"${Rule.Metadata}.${Metadata.Active}" -> active)))
// (2) - Sync Cache with Update Events
_ <- eventsService.addEvents(oldRules.map(rule => RuleEvent(rule.metadata.cacheKey, Update)))
} yield ()
}
}
Enjoy!

Related

Scala App doesn't run, just finishes with exit code 0

Full application below - why does this not run with the menu ready for user inputs? Simply returns "Process finished with exit code 0" - I am unsure what I am doing wrong here, the menu should be displayed ready for user input to then return the data from the specified file.
import scala.util.{Try, Success, Failure, Using}
case object MyApp5 {
case class State(name : String
,code : Int
,parties : Array[(String,Int)])
val mapdata = readFile("filename.txt")
def readFile(filename: String): Try[List[State]] = {
val dataRE = "([^(]+) \\((\\d+)\\),(.+)".r
val pVotes = "([^:]+):(\\d+)".r
Using(io.Source.fromFile(filename)) {
_.getLines()
.toList
.collect{ case dataRE(name, code, votes) =>
State(name.trim
,code.toInt
,votes.split(",")
.collect{case pVotes(p,v) => (p,v.toInt)})
}
}
}
class Menu(states: List[State]) {
def apply(key: String): Boolean = {
val (_, op, continue) = lookup(key)
op()
continue
}
private val lookup: Map[String,(String,()=>Unit,Boolean)] =
Map("?" -> ("show this menu", menu _, true)
,"menu" -> ("show this menu", menu _, true)
,"all" -> ("display all voting data", all _, true)
,"st" -> ("vote totals by state", stVotes _, true)
,"x" -> ("exit", done _, false)
,"quit" -> ("exit", done _, false)
).withDefaultValue(("",unknown _, true))
private def done(): Unit = println("bye")
private def unknown(): Unit = println("unknown selection ('?' for main menu)")
private def menu(): Unit =
lookup.keys.toVector.sorted
.map(k => s"$k\t: ${lookup(k)._1}")
.foreach(println)
private def all(): Unit =
states.sortBy(_.name) //alphabetical
.foreach{ st =>
println(st.name) //state name
st.parties
.sortBy(-_._2) //votes in decreasing order
.map{case (p,v) => f"\t$p%-12s:$v%9d"}
.foreach(println)
}
private def stVotes(): Unit =
states.map(st => (st.name, st.parties.map(_._2).sum))
.sortBy(-_._2) //votes in decreasing order
.map{case (state,total) => f"$state%-9s:$total%8d"}
.foreach(println)
}
def main(args: Array[String]): Unit =
args.headOption.map(readFile) match {
case None =>
println(s"usage: ${this.productPrefix} <data_file>")
case Some(Failure(exc)) =>
println(s"Error reading data file: $exc")
case Some(Success(stateData)) =>
val menu = new Menu(stateData)
menu("menu")
Iterator.continually(menu(io.StdIn.readLine(">> ").toLowerCase))
.dropWhile(identity)
.next()
}
}
This is a copy and paste from my scala case object MyApp5 (with the exception of "filename.txt" where I have put the absolute path - what am I missing?
Hard-coding the data file into the program would be a very unfortunate design decision. That way you wouldn't be able to run the program on different data sets, e.g. the results from different elections.
But ... oh well.
def main(args: Array[String]): Unit =
readFile("filename.txt") match {
case Failure(exc) =>
println(s"Error reading data file: $exc")
case Success(stateData) =>
val menu = new Menu(stateData)
menu("menu")
Iterator.continually(menu(io.StdIn.readLine(">> ").toLowerCase))
.dropWhile(identity)
.next()
}
I really don't see any point or advantage in doing it this way.

How to write documents to MongoDB using ReactiveMongo transactions?

I am trying to run this simple code, which insert a document to "reports" collection under transaction.
This is the code:
def runWithTransaction(): Future[Unit] = {
for {
dbWithSession <- db.db.startSession()
dbWithTx <- dbWithSession.startTransaction(None)
reportsCollection = dbWithTx.collection[JSONCollection]("reports")
_ <- reportsCollection.insert.one(BSONDocument("id" -> "123123"))
_ <- dbWithTx.commitTransaction()
_ <- dbWithSession.endSession()
} yield ()
}
I am getting the following error:
play.api.libs.json.JsResultException: JsResultException(errors:List((,List(JsonValidationError(List(CommandError[code=14, errmsg=BSON field 'OperationSessionInfo.txnNumber' is the wrong type 'int', expected type 'long', doc: {"operationTime":{"$time":1573135213,"$i":1,"$timestamp":{"t":1573135213,"i":1}},"ok":0,"errmsg":"BSON field 'OperationSessionInfo.txnNumber' is the wrong type 'int', expected type 'long'","code":14,"codeName":"TypeMismatch","$clusterTime":{"clusterTime":{"$time":1573135213,"$i":1,"$timestamp":{"t":1573135213,"i":1}},"signature":{"hash":{"$binary":"0000000000000000000000000000000000000000","$type":"00"},"keyId":0}}}]),WrappedArray())))))
at reactivemongo.play.json.JSONSerializationPack$.deserialize(JSONSerializationPack.scala:61)
at reactivemongo.play.json.JSONSerializationPack$.deserialize(JSONSerializationPack.scala:33)
at reactivemongo.api.commands.Command$$anon$2.$anonfun$one$6(commands.scala:141)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:658)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I am using:
scala 2.12
"org.reactivemongo" % "play2-reactivemongo_2.12" % "0.18.8-play27"
MongoDB version 4.2.1 (its actually a replica set with 1 primary)
Any help is appreciated, thanks.
Use this code to Insert:
def insert(coll: BSONCollection, document: BSONDocument): Future[Unit] = {
val writeRes: Future[WriteResult] = coll.insert.one(document)
writeRes.onComplete { // Dummy callbacks
case Failure(e) => e.printStackTrace()
case Success(writeResult) =>
println(s"successfully inserted document with result: $writeResult")
}
writeRes.map(_ => {}) // in this example, do nothing with the success
}
And If you want to update, use this:
def update(collection: BSONCollection, item_id: Long, modifier: BSONDocument) = {
val selector = BSONDocument("_id" -> item_id)
println("selector ===" + selector)
val futureUpdate = collection.update.one(
q = selector, u = modifier,
upsert = true, multi = false)
futureUpdate.onComplete { // Dummy callbacks
case Failure(e) => e.printStackTrace()
case Success(writeResult) =>
println(s"successfully updated document with result: $writeResult")
}
}

managed (ARM) in scala for nested resources

So, I have this code in scala which I am converting to managed.
val file_out = new FileOutputStream(new java.io.File(filePath, resultFile + ".tar.gz"));
val buffer_out = new BufferedOutputStream(file_out);
val gzip_out = new GzipCompressorOutputStream(buffer_out);
val tar_out = new TarArchiveOutputStream(gzip_out);
try {
addFileToTarGz(tar_out, filePath + "/" + resultFolder, "");
} finally {
tar_out.finish();
tar_out.close();
gzip_out.close();
buffer_out.close();
file_out.close();
}
First attempt is:
val file = new java.io.File(filePath, s"$resultFile.tar.gz")
managed(new FileOutputStream(file))
.acquireAndGet(stream => managed(new BufferedOutputStream(stream))
.acquireAndGet(buffer => managed(new GzipCompressorOutputStream(buffer))
.acquireAndGet(gzip => managed(new TarArchiveOutputStream(gzip))
.acquireAndGet(tar => {
try {
addFileToTarGz(tar, filePath + "/" + resultFolder, "")
} finally {
tar.finish()
}
}))))
However, it doesn't look very readable. Is there a better way to make it managed but also readable?
Have you considered load pattern?
def withResource[T](block: Resource => T): T = {
val resource = new Resource
try {
block(resource)
} finally {
resource.close()
}
}
then you would use it like:
withResourse { resource =>
// do something with resource
}
If you used separate load for each of those files you would end up with nested blocks... (which under some circumstances might be the most reasonable choice), but here I guess it will be enough to do:
def withTarStream(filePath: String, resultFile: String)(block: TarArchiveOutputStream => T): T = {
val fileOut = new FileOutputStream(new java.io.File(filePath, resultFile))
val bufferOut = new BufferedOutputStream(fileOut)
val gzipOut = new GzipCompressorOutputStream(bufferOut)
val tarOut = new TarArchiveOutputStream(gzipOut)
try {
block(tarOut)
} finally {
tarOut.finish()
tarOut.close()
gzipOut.close()
bufferOut.close()
fileOut.close()
}
}
used like:
withTarStream(filePath, s"$resultFile.tar.gz") { tarStream =>
addFileToTarGz(tarStream, filePath + "/" + resultFolder, "")
}
Based on #Mateusz Kubuszok's suggestion, I tried these variations:
private def withResource[T: Resource : Manifest, X](t: T, block: T => X): X = managed(t).acquireAndGet(x => block(x))
withResource(new FileOutputStream(file),
(x:FileOutputStream) => withResource(new BufferedOutputStream(x),
(y: BufferedOutputStream) => withResource(new GzipCompressorOutputStream(y),
(z: GzipCompressorOutputStream) => withResource(new TarArchiveOutputStream(z),
(tar: TarArchiveOutputStream) => writeTar(tar, filePath, resultFolder)))));
And then also refactored the above to following form:
private def withResource[T: Resource : Manifest, X](t: T, block: T => X): X = managed(t).acquireAndGet(x => block(x))
def writeInFile(x: FileOutputStream): Try[Unit] = withResource(new BufferedOutputStream(x), writeInBuffer)
def writeInBuffer(y: BufferedOutputStream): Try[Unit] = withResource(new GzipCompressorOutputStream(y), writeGzipStream)
def writeGzipStream(z: GzipCompressorOutputStream): Try[Unit] = withResource(new TarArchiveOutputStream(z), writeTarStream)
val file = new File(filePath, s"$resultFile.tar.gz")
withResource(new FileOutputStream(file), writeInFile);
The next day, a colleague mentioned this, which looks better than both of above: I am still exploring how to propagate result/error out of this block.
val file = new File(filePath, s"$resultFile.tar.gz")
for {
stream <- managed(new FileOutputStream(file))
buffer <- managed(new BufferedOutputStream(stream))
gzip <- managed(new GzipCompressorOutputStream(buffer))
tar <- managed(new TarArchiveOutputStream(gzip))
} {
writeTar(tar)
}
Similar to what #cchantep suggested, I ended up doing this:
val tarOutputStream: ManagedResource[TarArchiveOutputStream] = (for {
stream <- managed(new FileOutputStream(file))
buffer <- managed(new BufferedOutputStream(stream))
gzip <- managed(new GzipCompressorOutputStream(buffer))
tar <- managed(new TarArchiveOutputStream(gzip))
} yield tar)
Try (tarOutputStream.acquireAndGet(writeTarStream(_))) match {
case Failure(e) => Failure(e)
case Success(_) => Success(new File(s"$filePath/$runLabel.tar.gz"))
}

Slick3 with SQLite - autocommit seems to not be working

I'm trying to write some basic queries with Slick for SQLite database
Here is my code:
class MigrationLog(name: String) {
val migrationEvents = TableQuery[MigrationEventTable]
lazy val db: Future[SQLiteDriver.backend.DatabaseDef] = {
val db = Database.forURL(s"jdbc:sqlite:$name.db", driver = "org.sqlite.JDBC")
val setup = DBIO.seq(migrationEvents.schema.create)
val createFuture = for {
tables <- db.run(MTable.getTables)
createResult <- if (tables.length == 0) db.run(setup) else Future.successful()
} yield createResult
createFuture.map(_ => db)
}
val addEvent: (String, String) => Future[String] = (aggregateId, eventType) => {
val id = java.util.UUID.randomUUID().toString
val command = DBIO.seq(migrationEvents += (id, aggregateId, None, eventType, "CREATED", System.currentTimeMillis, None))
db.flatMap(_.run(command).map(_ => id))
}
val eventSubmitted: (String, String) => Future[Unit] = (id, batchId) => {
val q = for { e <- migrationEvents if e.id === id } yield (e.batchId, e.status, e.updatedAt)
val updateAction = q.update(Some(batchId), "SUBMITTED", Some(System.currentTimeMillis))
db.map(_.run(updateAction))
}
val eventMigrationCompleted: (String, String, String) => Future[Unit] = (batchId, id, status) => {
val q = for { e <- migrationEvents if e.batchId === batchId && e.id === id} yield (e.status, e.updatedAt)
val updateAction = q.update(status, Some(System.currentTimeMillis))
db.map(_.run(updateAction))
}
val allEvents = () => {
db.flatMap(_.run(migrationEvents.result))
}
}
Here is how I'm using it:
val migrationLog = MigrationLog("test")
val events = for {
id <- migrationLog.addEvent("aggregateUserId", "userAccessControl")
_ <- migrationLog.eventSubmitted(id, "batchID_generated_from_idam")
_ <- migrationLog.eventMigrationCompleted("batchID_generated_from_idam", id, "Successful")
events <- migrationLog.allEvents()
} yield events
events.map(_.foreach(event => event match {
case (id, aggregateId, batchId, eventType, status, submitted, updatedAt) => println(s"$id $aggregateId $batchId $eventType $status $submitted $updatedAt")
}))
The idea is to add event first, then update it with batchId (which also updates status) and then update the status when the job is done. events should contain events with status Successful.
What happens is that after running this code it prints events with status SUBMITTED. If I wait a while and do the same allEvents query or just go and check the db from command line using sqlite3 then it's updated correctly.
I'm properly waiting for futures to be resolved before starting the next operation, auto-commit should be enabled by default.
Am I missing something?
Turns out the problem was with db.map(_.run(updateAction)) which returns Future[Future[Int]] which means that the command was not finished by the time I tried to run another query.
Replacing it with db.flatMap(_.run(updateAction)) solved the issue.

Testing reactivemongo with scalatest raise an exception

I am trying to test some functionality in the app. I have other tests (with scalatest for reactivemongo) and are working, but with this I am getting this exception
[info] - should Persist and find a token * FAILED *
[info] The future returned an exception of type: reactivemongo.api.commands.bson.DefaultBSONCommandError, with message: CommandError[code=26, errmsg=ns not found, doc: {
[info] ok: BSONDouble(0.0),
[info] errmsg: "ns not found",
[info] code: BSONInteger(26)
[info] }]. (DaosApplicationSpecOneAppPerTest.scala:74)
This is the code for the tests (both are throwing the same error)
class UserTokenDaoMongoSpec extends DaosApplicationSpecOneAppPerTest {
"UserTokenDao" should {
"Persist and find a token" in withUserTokenDao { userTokenDao =>
val future = for {
_ <- userTokenDao.save(token)
maybeToken <- userTokenDao.find(token.id)
} yield {
maybeToken.map(_ == token)
}
whenReady (future) { result =>
result.get must be (true)
}
}
"Remove a token" in withUserTokenDao { userTokenDao =>
val future = for {
_ <- userTokenDao.save(token)
_ <- userTokenDao.remove(token.id)
maybeToken <- userTokenDao.find(token.id)
} yield {
maybeToken
}
whenReady (future) { result =>
result must be (None)
}
}
}
}
and for brevity, this is the method that inherits
def withUserTokenDao[T](t: UserTokenDao => T):T = running(app) {
val userTokenDao = new UserTokenDaoMongo
whenReady (userTokenDao.tokens.drop()) { result =>
t(userTokenDao)
}
}
The UserTokenDao implementation
class UserTokenDaoMongo extends UserTokenDao {
lazy val reactiveMongoApi = current.injector.instanceOf[ReactiveMongoApi]
val tokens = reactiveMongoApi.db.collection[JSONCollection]("tokens")
def find(id:UUID):Future[Option[UserToken]] =
tokens.find(Json.obj("id" -> id)).one[UserToken]
def save(token:UserToken):Future[UserToken] = for {
_ <- tokens.insert(token)
} yield token
def remove(id:UUID):Future[Unit] = for {
_ <- tokens.remove(Json.obj("id" -> id))
} yield ()
}
and this is the model of UserToken
class UserTokenDaoMongo extends UserTokenDao {
lazy val reactiveMongoApi = current.injector.instanceOf[ReactiveMongoApi]
val tokens = reactiveMongoApi.db.collection[JSONCollection]("tokens")
def find(id:UUID):Future[Option[UserToken]] =
tokens.find(Json.obj("id" -> id)).one[UserToken]
def save(token:UserToken):Future[UserToken] = for {
_ <- tokens.insert(token)
} yield token
def remove(id:UUID):Future[Unit] = for {
_ <- tokens.remove(Json.obj("id" -> id))
} yield ()
}
I am not sure what could be causing the error
Thank you
It turned out that the problem was that the collection tokens did not exist. I got into the mongo console, then I created the collection and the tests started working.