So this is my celery Tasks code where I am trying to group the results of a few tasks:
#tasks.task(name='tasks.sample_group')
def sample_group(self, **kwargs):
tasks = [ add.s(**kwargs).set(queue = "vTestQueue")),
sub.s(**kwargs).set(queue = "vTestQueue"))]
return (group(tasks)).apply_async()
#tasks.task(name='tasks.add')
def add(**kwargs) :
print("add task started at " + str(datetime.now(tz=None)))
writefile("add" , kwargs['x'] + kwargs['y'])
print("add task ended at " + str(datetime.now(tz=None)))
return kwargs['x'] + kwargs['y']
#tasks.task(name='tasks.subtract')
def subtract(**kwargs) :
print("subtract started task at " + str(datetime.now(tz=None)))
writefile("sub" , kwargs['x'] + kwargs['y'])
print("subtract ended task at " + str(datetime.now(tz=None)))
return kwargs['x'] - kwargs['y']
The problem with the group is that it never returns the result of the operation. The result is always none. What do I need to set to get the result?
This is the config I am using:
accept_content = ["application/json"]
enable_utc = True
timezone = "Australia/Melbourne"
broker_url = "pyamqp://********************:8080/******"
result_backend = "rpc://"
For Group you should use GroupResult. Also do not try to call .get() inside a task as that is wrong way to do it.
Related
I'm trying to loop through an array of products (Strings) in order to execute a call for each of them but I'm mismatching some types during scenario building, here is an example of what I'm trying to do:
val createAddToCart: ScenarioBuilder = scenario("Add to cart").exec(CartRequest.addToCart())
This is my CartRequest object:
def addToCart(): Unit (???) = {
for (product <- products) {
addToCart(product)
}
}
def addToCart(productId:String): ChainBuilder =
{
exec(http("addToCart")
.post(addToCartUrl)
.headers(authHeaders)
.body(generateRequestBodyForProduct(productId)).asJson
.check(bodyString.saveAs("body"))
).exec(
session => {
body = session("body").as[String]
println("response body " + body)
session
}
)
}
def generateRequestBodyForProduct(productId: String): Body with (Expression[String]) =
{
StringBody("{ \"productId\": \"" + productId + "\"," +
" \"qty\": " + 1 + " , " +
"\"storeId\": \"" + storeId + "\"}")
}
Obviously, I'm having problems calling CartRequest.addToCart() due to type mismatching (Unit to ChainBuilder).
Is there a way to execute and return the addToCart method as a list of scenarios?
I hope I explained myself well.
Thanks for your time.
First, you don't look like a Scala developper. You should probably use the new Java DSL available since Gatling 3.7.
Then, you're not using the Gatling constructs properly.
your for loop should be a Gatling foreach instead
the dynamic productId in your request body should be resolved with Gatling Expression Language => "{ \"productId\": \"#{productId}\", \"qty\": " + 1 + " , " + "\"storeId\": \"" + storeId + "\"}"
I've set the timeout duration to "2 minutes" as follows:
def updateAcrossEvents (tuple3: Tuple3[String, String, String], inputs: Iterator[R00tJsonObject],
oldState: GroupState[MyState]): OutputRow = {
println("$$$$ Inside updateAcrossEvents with : " + tuple3._1 + ", " + tuple3._2 + ", " + tuple3._3)
var state: MyState = if (oldState.exists) oldState.get else MyState(tuple3._1, tuple3._2, tuple3._3)
if (oldState.hasTimedOut) {
println("##### oldState has timed out ####")
// Logic to Write OutputRow
OutputRow("some values here...")
} else {
for (input <- inputs) {
state = updateWithEvent(state, input)
oldState.update(state)
oldState.setTimeoutDuration("2 minutes")
}
OutputRow(null, null, null)
}
}
I have also specified ProcessingTimeTimeout in 'mapGroupsWithState' as follows...
.mapGroupsWithState(GroupStateTimeout.ProcessingTimeTimeout())(updateAcrossEvents)
But 'hasTimedOut' is never true so I don't get any output! What am I doing wrong?
It seems it only works if input data is continuously flowing. I had stopped the input job because I had enough data but it seems timeouts work only if the data is continuously fed. Not sure why it's designed that way. Makes it a bit harder to write unit/integration tests BUT I am sure there's a reason why it's designed this way. Thanks.
I'm new Jbehave. Im trying to find a way to achieve world in cucumber with Jbehave.
I generated project with jbehave with pico archetype. Im trying to run two stories in parallel. I updated threads in pom.xml to 2.
Consider a sample stepdef class as below
public class PojoSteps {
public Pojo pojo;
public PojoSteps(Pojo pojo){
this.pojo = pojo;
System.out.println(" ##" + new Timestamp(System.currentTimeMillis()) + " * this is " + this + ". In PojoSteps constructor. Pojo created is:" + this.pojo + ". Thread = " + Thread.currentThread().getId());
}
#Given("I have a pojo with $i")
public void iHaveAPojoWith1(int i){
pojo.setI(i);
System.out.println(" ##" + new Timestamp(System.currentTimeMillis()) + " * this is " + this + ".In iHaveAPojoWith1. pojo = " + this.pojo +". Value of To be set = " + i +". Value set Pojo.getI() = " + this.pojo.getI() + ". Thread = " + Thread.currentThread().getId());
}
#When("I wait for some time")
public void randomWait() throws InterruptedException {
Thread.sleep(new Random().nextInt(200));
}
#Then("my pojo should still have $expectedi")
public void myPojoShouldStillHave1(int expectedi){
System.out.println(" ##" + new Timestamp(System.currentTimeMillis()) + " * this is " + this +". In myPojoShouldStillHave1. pojo = " + this.pojo + ". expected = " + expectedi + ". actual = " + this.pojo.getI() + ". Thread = " + Thread.currentThread().getId());
Assert.assertEquals(this.pojo.getI(), expectedi);
}
}
I have Pojo model as below
public class Pojo {
private int i;
public void setI(int i){
this.i = i;
}
public int getI(){
return this.i;
}
}
My two stories are as below
PojoOne.story
Scenario: scenario description
Given I have a pojo with 1
When I wait for some time
Then my pojo should still have 1
PojoTwo.story
Scenario: random description
Given I have a pojo with 2
When I wait for some time
Then my pojo should still have 2
I have MyStories class which extends JUnitStories as below
public class MyStories extends JUnitStories {
....
private PicoContainer createPicoContainer() {
MutablePicoContainer container = new DefaultPicoContainer(new Caching().wrap(new ConstructorInjection()));
container.addComponent(PojoSteps.class);
container.addComponent(Pojo.class);
return container;
}
}
When I don't run stories in parallel, they succeed. When I run stories in parallel, they are failing.
##2020-01-02 20:35:36.53 * this is learning.steps.PojoSteps#49f3f232. In PojoSteps constructor. Pojo created is:learning.Support.Pojo#4aa9e824. Thread = 12
##2020-01-02 20:35:36.531 * this is learning.steps.PojoSteps#49f3f232.In iHaveAPojoWith1. pojo = learning.Support.Pojo#4aa9e824. Value of To be set = 1. Value set Pojo.getI() = 1. Thread = 12
##2020-01-02 20:35:36.532 * this is learning.steps.PojoSteps#6136e035. In PojoSteps constructor. Pojo created is:learning.Support.Pojo#1f4412c8. Thread = 13
##2020-01-02 20:35:36.533 * this is learning.steps.PojoSteps#6136e035.In iHaveAPojoWith1. pojo = learning.Support.Pojo#1f4412c8. Value of To be set = 2. Value set Pojo.getI() = 2. Thread = 13
##2020-01-02 20:35:36.558 * this is learning.steps.PojoSteps#6136e035. In myPojoShouldStillHave1. pojo = learning.Support.Pojo#1f4412c8. expected = 1. actual = 2. Thread = 12
##2020-01-02 20:35:36.668 * this is learning.steps.PojoSteps#6136e035. In myPojoShouldStillHave1. pojo = learning.Support.Pojo#1f4412c8. expected = 2. actual = 2. Thread = 13
...
Then my pojo should still have 1 (FAILED)
(java.lang.AssertionError: expected:<2> but was:<1>)
I'm not able to find documentation on how to run tests in parallel with Jbehave and pico. Also I need to have a model outside my stepdef class as I will be having multiple stefdef classes which will share the same model instance.
After reading a bit more and trying out Jbehave with spring,
1. I think there is no way to get different instances of stepdef created per thread.
2. In Jbehave, there is no equivalent of cucumber's world object which is threadsafe.
As #VaL mentioned in the comments of the questions, we have to make thinks threadsafe explicitly.
This is my understanding as of now. If there is any better way - kindly let me know.
Trying to process a list of long running jobs in a vertx way
One would hope one could do something like:
use the executeBlocking to process the long running job in an async manner
use the composite future to wait for the futures to complete
I'm aware this approach does not work .. the list of Futures is not complete before the code drops into the CompositeFuture.
Is there a executeBlocking approach or does one have to use either the eventbus, vertx utils that support lists?
java.util.ArrayList futureList = new ArrayList()
for (i = 0; i < 100; i ++){
vertx.executeBlocking({ future ->
int id = i
println "Running " + id
java.lang.Thread.sleep(1000)
println "Thread done " + id
future.complete()
}, true , { res ->
if (res.succeeded()) {
print "."
} else {
print "x"
}
})
}
CompositeFuture.join(futureList).setHandler({ ar ->
if (ar.succeeded()) {
System.err.println "all threads should be done.."
}
})
Results in .. "all threads should be done" printing early
Running 84
Running 87
Running 87
Running 95
all threads should be done..
done.
Thread done 3
Thread done 36
Thread done 3
Thread done 0
In your example, futureList is empty so CompositeFuture.join(futureList) is completed immediately.
Change your example like this:
java.util.ArrayList futureList = new ArrayList()
for (i = 0; i < 100; i ++){
Future jobFuture = Future.future()
futureList.add(jobFuture)
vertx.executeBlocking({ future ->
int id = i
println "Running " + id
java.lang.Thread.sleep(1000)
println "Thread done " + id
future.complete()
}, true , { res ->
if (res.succeeded()) {
print "."
} else {
print "x"
}
jobFuture.complete()
})
}
Notice the jobFuture creation:
Future jobFuture = Future.future()
futureList.add(jobFuture)
As well as completion:
jobFuture.complete()
Now the CompositeFuture.join(futureList) handler will be executed only after all jobs complete.
I'm using Spark with cassandra. And i'm reading some rows from my table in order to delete theme using the PrimaryKey. This is my code :
val lines = sc.cassandraTable[(String, String, String, String)](CASSANDRA_SCHEMA, table).
select("a","b","c","d").
where("d=?", d).cache()
lines.foreach(r => {
val session: Session = connector.openSession
val delete = s"DELETE FROM "+CASSANDRA_SCHEMA+"."+table+" where channel='"+r._1 +"' and ctid='"+r._2+"'and cvid='"+r._3+"';"
session.execute(delete)
session.close()
})
But this method create an session for each row and it takes lot of time. So is it possible to delete my rows using sc.CassandraTable or another solution better then the mine.
Thank you
I don't think there's a support for delete at the moment on the Cassandra Connector. To amortize the cost of connection setup, the recommended approach is to apply the operation to each partition.
So your code will look like this:
lines.foreachPartition(partition => {
val session: Session = connector.openSession //once per partition
partition.foreach{elem =>
val delete = s"DELETE FROM "+CASSANDRA_SCHEMA+"."+table+" where channel='"+elem._1 +"' and ctid='"+elem._2+"'and cvid='"+elem._3+"';"
session.execute(delete)
}
session.close()
})
You could also look into using the DELETE FROM ... WHERE pk IN (list) and use a similar approach to build up the list for each partition. This will be even more performant, but might break with very large partitions as the list will become consequentially long. Repartitioning your target RDD before applying this function will help.
You asked the question a long time ago so you probably found the answer already. :P Just to share, here is what I did in Java. This code works against my local Cassandra instance beautifully. But it does not work against our BETA or PRODUCTION instances, because I suspect there are multiple instances of the Cassandra database there and the delete only worked against 1 instance and the data got replicated right back. :(
Please do share if you were able to get it to work against your Cassandra production environment, with multiple instances of it running!
public static void deleteFromCassandraTable(Dataset myData, SparkConf sparkConf){
CassandraConnector connector = CassandraConnector.apply(sparkConf);
myData.foreachPartition(partition -> {
Session session = connector.openSession();
while(partition.hasNext()) {
Row row = (Row) partition.next();
boolean isTested = (boolean) row.get(0);
String product = (String) row.get(1);
long reportDateInMillSeconds = ((Timestamp) row.get(2)).getTime();
String id = (String) row.get(3);
String deleteMyData = "DELETE FROM test.my_table"
+ " WHERE is_tested=" + isTested
+ " AND product='" + product + "'"
+ " AND report_date=" + reportDateInMillSeconds
+ " AND id=" + id + ";";
System.out.println("%%% " + deleteMyData);
ResultSet deleteResult = session.execute(deleteMyData);
boolean result = deleteResult.wasApplied();
System.out.println("%%% deleteResult =" + result);
}
session.close();
});
}