In the below code, I'm trying to do two operations. One, to create a customer in a db, and the other, to create an event in the db. The creation of the event, is dependent on the creation of the user.
I'm new to Scala, and confused on the role of Futures here. I'm trying to query a db and see if the user is there, and if not, create the user. The below code is supposed to check if the user exists with the customerByPhone() function, and if it doesn't, then go into the createUserAndEvent() function.
What it's actually doing, is skipping the response from customerByPhone and going straight into createUserAndEvent(). I thought that by using a flatmap, the program would automatically wait for the response and that I wouldn't have to use Await.result is that not the case? Is there a way to avoid using Await.result to not block the thread on production code?
override def findOrCreate(phoneNumber: String, creationReason: String): Future[AvroCustomer] = {
//query for customer in db
//TODO this goes into createUserAndEvent before checking that response comes back empty from querying for user
customerByPhone(phoneNumber)
.flatMap(_ => createUserAndEvent(phoneNumber, creationReason, 1.0))
}
You don't need to use Await.result or any other blocking. You do in fact have the result from customerByPhone, you're just ignoring it with the _ . I think what you want is something like this:
customerByPhone(phoneNumber)
.flatMap(customer => {
if(customer == null)
createUserAndEvent(phoneNumber, creationReason, 1.0)
else
Future(customer)
})
You need to code the logic to do something only if the customer isn't there.
Related
I am new to Scala and Gatling and I am trying to figure out what would be the best way to define a user story and pass it a ChainBuilder to Gatling Scenario.
When I say user Story In my case I mean a flow that will consist of Login, many different calls and then a loop over another list of calls for the whole duration of the test.
I have created the following function to create a scenario:
def createScenario(name: String, feed: FeederBuilder, chains: ChainBuilder*): ScenarioBuilder = {
scenario(name).feed(feed).forever() {
exec(chains).pause(Config.pauseBetweenRequests)
}
}
And here is how I execute this function:
val scenario = createScenario(Config.testName, feeder.random,
setSessionParams(PARAM1, Config.param1),
setSessionParams(PARAM2, Config.param2),
login,
executeSomeCall1,
executeSomeCall2,
executeSomeCall3,
executeSomeCall4,
executeSomeCall5,
executeSomeCall6,
executeSomeCall7,
executeSomeCall8,
executeSomeCall9,
)
Here is an example of what executeSomeCall function looks like:
def executeSomeCall = {
exec(http("ET Call Home")
.post("/et/call/home")
.body(ElFileBody("/redFingerBody.json")).asJson
.check(status is 200))
}
My first question:
Is that the correct way to define a chain of rest calls and feed it to the scenario? I am asking that because what I see when I define a flow like that is that for some reason not all the my REST calls are actually executed. Weirdly enough, if I change the order of the calls it does work and all functions are called. (So I am definitely doing something wrong)
My second question:
How can I define an infinite loop within this flow? (Infinite for as long as the test is running)
So for example, I'd like the above flow to start and when it reaches executeSomeCall8, it will then loop executeSomeCall8 and executeSomeCall9 for the whole duration of the test.
I don't see why your calls would not be executed, however the way you're constructing your scenario is not that flexible. You can make use of chaining without requiring a createScenario() method.
That leads to your second question, when you have the scenario chained like:
val scn = scenario("something")
...
.exec(someCall7)
.forever(){
exec(sommeCall8)
.exec(someCall9)
}
...
where someCallN in my case look like:
val someCall = http("request name")
.get("/some/uri")
...
Note: foerever() is just an example, you can use other loop statements that suits your needs.
I hope it helps.
is there a way to make synchronous queries to MongoDB?
I'd like to run some code only after I've retrieved all my data from the DB.
Here is a sample snipped.
Code Snippet A
const brandExists = Brands.find({name: trxn.name}).count();
Code Snippet B
if(brandExists == 0){
Brands.insert({
name:trxn.name,
logo:"default.png",
});
Trxs.insert({
userId,
merchant_name,
amt,
});
}
I'd like Code snippet B to run only after Code Snippet A has completed its data retrieval from the DB. How would one go about doing that?
You can use simple async function async function always returns a promise.
const brandExists;
async function brandExist() {
brandExists = Brands.find({
name: trxn.name
}).count();
}
brandExist().then(
// Your Code comes here
if (brandExists == 0) {
Brands.insert({
name: trxn.name,
logo: "default.png",
})
Trxs.insert({
userId,
merchant_name,
amt,
});
});
I don't think using an if statement like the one you have makes sense: the queries are sent after each other; it is possible someone else creates a brand with the same name as the one you are working with between your queries to the database.
MongoDB has something called unique indexes you can use to enforce values being unique. You should be able to use name as a unique index. Then when you insert a new document into the collection, it will fail if there already exists a document with that name.
https://docs.mongodb.com/manual/core/index-unique/
In Meteor, MongoDB queries are synchronous, so it already delivers what you need. No need to make any changes, snippet B code will only run after snippet A code.
When we call a function asynchronous we mean that when that function is called it is non-blocking, which means our program will call the function and keep going, or, not wait for the response we need.
If our function is synchronous, it means that our program will call that function and wait until it's received a response from that function to continue with the rest of the program.
Meteor is based in Node, which is asynchronous by nature, but coding with only asynchronous functions can origin what developers call "callback hell".
On the server side, Meteor decided to go with Fibers, which allows functions to wait for the result, resulting in synchronous-style code.
There's no Fibers in the client side, so every time your client calls a server method, that call will be asynchronous (you'll have to worry about callbacks).
Your code is server-side code, and thanks to Fibers you can be assure that snippet B code will only run after snippet A code.
I have a flow with data associated to users. I also have a state for each user, that I can get asynchronously from DB.
I want to separate my flow with one subflow per user, and load the state for each user when materializing the subflow, so that the elements of the subflow can be treated with respect to this state.
If I don't want to merge the subflows downstream, I can do something with groupBy and Sink.lazyInit :
def getState(userId: UserId): Future[UserState] = ...
def getUserId(element: Element): UserId = ...
def treatUser(state: UserState): Sink[Element, _] = ...
val treatByUser: Sink[Element] = Flow[Element].groupBy(
Int.MaxValue,
getUserId
).to(
Sink.lazyInit(
elt => getState(getUserId(elt)).map(treatUser),
??? // this is never called, since the subflow is created when an element comes
)
)
However, this does not work if treatUser becomes a Flow, since there is no equivalent for Sink.lazyInit.
Since subflows of groupBy are materialized only when a new element is pushed, it should be possible to use this element to materialize the subflow, but I wasn't able to adapt the source code for groupBy so that this work consistently. Likewise, Sink.lazyInitdoesn't seem to be easily translatable to the Flow case.
Any idea on how to solve this issue ?
The relevant Akka issue you have to look at is #20129: add Sink.dynamic and Flow.dynamic.
In the associated PR #20579 they actually implemented LazySink stuffs.
They are planning to do LazyFlow next:
Will do next lazyFlow with similar signature.
Unfortunately you have to wait for that functionality to be implemented in Akka or write it yourself (then consider a PR to Akka).
When I create a query in squeryl, it returns a Query[T] object. The query was not yet executed and will be, when I iterate over the Query object (Query[T] extends Iterable[T]).
Around the execution of a query there has to be either a transaction{} or a inTransaction{} block.
I'm just speaking of SELECT queries and transactions wouldn't be necessary, but the squeryl framework needs them.
I'd like to create a query in the model of my application and pass it directly to the view where a view helper in the template iterates over it and presents the data.
This is only possible when putting the transaction{} block in the controller (the controller includes the call of the template, so the template which does the iteration is also inside). It's not possible to put the transaction{} block in the model, because the model doesn't really execute the query.
But in my understanding the transaction has nothing to do with the controller. It's a decision of the model which database framework to use, how to use it and where to use transactions. So I want the transaction{} block to be in the model.
I know that I can - instead of returning the Query[T] instance - call Iterable[T].toList on this Query[T] object and then return the created list. Then the whole query is executed in the model and everything is fine. But I don't like this approach, because all the data requested from the database has to be cached in this list. I'd prefer a way where this data is directly passed to the view. I like the MySql feature of streaming the result set when it's large.
Is there any possibility? Maybe something like a function Query[T].executeNow() which sends the request to the database, is able to close the transaction, but still uses the MySQL streaming feature and receives the rest of the (selected and therefore fixed) result set when it's accessed? Because the result set is fixed in the moment of the query, closing the transaction shouldn't be a problem.
The general problem that I see here is that you try to combine the following two ideas:
lazy computation of data; here: database results
hiding the need for a post-processing action that must be triggered when the computation is done; here: hiding from your controller or view that the database session must be closed
Since your computation is lazy and since you are not obliged to perform it to the very end (here: to iterate over the whole result set), there is no obvious hook that could trigger the post-processing step.
Your suggestion of invoking Query[T].toList does not exhibit this problem, since the computation is performed to the very end, and requesting the last element of the result set can be used as a trigger for closing the session.
That said, the best I could come up with is the following, which is an adaptation of the code inside org.squeryl.dsl.QueryDsl._using:
class IterableQuery[T](val q: Query[T]) extends Iterable[T] {
private var lifeCycleState: Int = 0
private var session: Session = null
private var prevSession: Option[Session] = None
def start() {
assert(lifeCycleState == 0, "Queries may not be restarted.")
lifeCycleState = 1
/* Create a new session for this query. */
session = SessionFactory.newSession
/* Store and unbind a possibly existing session. */
val prevSession = Session.currentSessionOption
if(prevSession != None) prevSession.get.unbindFromCurrentThread
/* Bind newly created session. */
session.bindToCurrentThread
}
def iterator = {
assert(lifeCycleState == 1, "Query is not active.")
q.toStream.iterator
}
def stop() {
assert(lifeCycleState == 1, "Query is not active.")
lifeCycleState = 2
/* Unbind session and close it. */
session.unbindFromCurrentThread
session.close
/* Re-bind previous session, if it existed. */
if(prevSession != None) prevSession.get.bindToCurrentThread
}
}
Clients can use the query wrapper as follows:
var manualIt = new IterableQuery(booksQuery)
manualIt.start()
manualIt.foreach(println)
manualIt.stop()
// manualIt.foreach(println) /* Fails, as expected */
manualIt = new IterableQuery(booksQuery) /* Queries can be reused */
manualIt.start()
manualIt.foreach(b => println("Book: " + b))
manualIt.stop()
The invocation of manualIt.start() could already be done when the object is created, i.e., inside the constructor of IterableQuery, or before the object is passed to the controller.
However, working with resources (files, database connections, etc.) in such a way is very fragile, because the post-processing is not triggered in case of exceptions. If you look at the implementation of org.squeryl.dsl.QueryDsl._using you will see a couple of try ... finally blocks that are missing from IterableQuery.
I have a really long Excel file wich I read using EPPlus. For each line I test if it meets certain criteria and if so I add the line (an object representing the line) to a collection. When the file is read, I store those objects to the database. Would it be possible to do both things at the same time? My idea is to have a collection of objects that somehow would be consumed by thread that would save the objects to the DB. At the same time the excel reader method would populate the collection... Could this be done using Rx or is there a better method?
Thanks.
An alternate answer - based on comments to my first.
Create a function returning an IEnumberable<Records> from EPPlus/Xls - use yield return
then convert the seqence to an observable on the threadpool and you've got the Rx way of having a producer/consumer and BlockingCollection.
function IEnumberable<Records> epplusRecords()
{
while (...)
yield return nextRecord;
}
var myRecords = epplusRecords
.ToObservable(Scheduler.ThreadPool)
.Where(rec => meetsCritera(rec))
.Select(rec => newShape(rec))
.Do(newRec => writeToDb(newRec))
.ToArray();
Your case seems to be of pulling data (IEnumerable) and not pushing data (IObservable/Rx).
Hence I would suggest LINQ to objects is something that can be used to model the solution.
Something like shown in below code.
publis static IEnumerable<Records> ReadRecords(string excelFile)
{
//Read from excel file and yield values
}
//use linq operators to do filtering
var filtered = ReadRecords("fileName").Where(r => /*ur condition*/)
foreach(var r in filtered)
WriteToDb(r);
NOTE: In using IEnumerable you don't create intermediate collections in this case and the whole process looks like a pipeline.
It doesn't seem like a good fit, as there's no inherent concurrency, timing, or eventing in the use case.
That said, it may be a use case for plinq. If EEPlus supports concurrent reads. Something like
epplusRecords
.AsParallel()
.Where(rec => meetsCritera(rec))
.Select(rec => newShape(rec))
.Do(newRec => writeToDb(newRec))
.ToArray();