I want to execute the Spark program from Workflow and store the results on Minio. What do I do?
Here is an example Pi that I executed using workflow choreography
apiVersion: argoproj.io/v1alpha1
kind: Workflow
...
manifest: |
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: spark-pi
namespace: spark
spec:
type: Scala
mode: cluster
image: "gcr.io/spark-operator/spark:v3.1.1"
nodeSelector:
kubernetes.io/hostname: node01
imagePullPolicy: Never
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
...
driver:
...
executor:
...
- name: sparkling-operator
dag:
tasks:
- name: SparkGroupBY
template: spark-groupby
Related
Recently all the cronjobs on my GKE cluster started showing some weird behaviour. With all the configurations remaining same for the cronjobs, out of nowhere, now my cronjobs are getting triggered, executed & completed but the Pod is left behind in a Not Ready state. I've been using successfulJobsHistoryLimit and failedJobsHistoryLimit as 0. Also, tried concurrencyPolicy: "Replace" and spec.ttlSecondsAfterFinished: 1 but still nothing changed. I want to get rid of these as there are some crons running at every 5 minutes and so, generating huge list of not ready completed pods here when I run kubectl get pods
NAME READY STATUS RESTARTS AGE
my-cron-27711665-gjdk4 0/1 Completed 0 46m
Here's the cronjob.yaml file:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: my-cron
spec:
schedule: "0 */1 * * *"
successfulJobsHistoryLimit: 0
failedJobsHistoryLimit: 0
jobTemplate:
spec:
template:
spec:
imagePullSecrets:
- name: cronjob-secret
containers:
- name: my-cron
image: kubernetes_cronjob_new:dev
command: ["/bin/bash","-c"]
args:
- po1=$(kubectl get pods | awk '/mybackend/ {printf($1);exit}');
kubectl exec -i $po1 -- python manage.py run_cron my_cron False;
exit 0;
restartPolicy: OnFailure
Currently working on Kubernetes Version: 1.22
Can anyone help me here? How to get rid of these or why this started happening all of sudden?
I am trying and trying to make my Entity move towards some previously described destination.
The entity moves on a microlevel over and over again but never reaches its destination. DestinationSystem checks with its update function if the Entity is close enough so it can choose a new destination for it.
I kindly ask you for help as I am absolutely out of ideas.
Here is my DestinationSystem's update function.
This system will only chose the current destination for the entity and then checks if the entity reached it.
func update(context: SceneUpdateContext) {
let walkers = context.scene.performQuery(Self.query)
walkers.forEach({ entity in
guard var walker = entity.components[DestinationComponent.self] as? DestinationComponent,
let _ = entity.components[MotionComponent.self] as? MotionComponent else { return }
defer {
entity.components[DestinationComponent.self] = walker
}
// set the default destination to 0
var distanceFromDestination: Float = 0
// check if walker component for this specific entity has any destination
if let destination = walker.destination {
// distract distance from it
distanceFromDestination = entity.distance(from: destination)
}
// check if the distance still is. If its tiny, then lets choose another distance
if distanceFromDestination < 0.02 {
var fixedDestination = SIMD3<Float>.spawnPoint(from: entity.transform.translation, radius: 0.7)
fixedDestination.y = Float(0)
// If there is any obstacle on the way to the destination, set that obstacle as the destination so it wont pass it, perhaps through the wall
let obstacles = context.scene.raycast(
from: entity.position,
to: fixedDestination,
query: .nearest,
mask: .sceneUnderstanding,
relativeTo: nil
)
if let nearest = obstacles.first {
fixedDestination = nearest.position
fixedDestination.y = Float(0)
}
walker.destination = fixedDestination
}
})
}
Here is my MotionSystem. Constant variable s calculates current translation to cross and multiplies it by the deltaTime of the context and 0.03 which I choose as a speed.
func update(context: SceneUpdateContext) {
context.scene.performQuery(Self.query).forEach { entity in
let fixedDelta: Float = Float(context.deltaTime)
guard let _ = entity.components[MotionComponent.self] as? MotionComponent,
let walker = entity.components[DestinationComponent.self] as? DestinationComponent
else { return }
var newTransform = entity.transform
let s = (walker.destination! - entity.transform.translation) * 0.03 * fixedDelta
newTransform.translation = s
newTransform.translation.y = Float(0)
entity.move(to: newTransform, relativeTo: entity.parent)
}
}
I believe it should take my Entity frame by frame to the destination point, then choose another one. It never will again.
Unless my math is terribly off?
Thank you for any help.
I am unable to update content in xml tag using xml Twig module.
Here is my xml file :-
<?xml version="1.0"?>
<test customerProprietary="false">
<classes>
<class name="new" />
</classes>
<chars>
<log>false</log>
</chars>
</test>
Code snippet:-
my $twig = XML::Twig->new(
pretty_print => 'indented',
twig_handlers => {
log => sub {
if ($_->text eq 'true'){
exit;
}
else{
print $_->text;
subp4open($xmlfile);
$_->set_text( 'true' );
exit;
}
}
);
$twig->parsefile($script_path);
$twig->print_to_file($script_path);
i have to update <log>false</log> to <log>true</log>. Am i missing something here?
That's because exit exits the program, so print_to_file is never reached. Remove the exits. You can replace them with returns, but they aren't needed.
When I run into these sorts of things, I like to put a statement like say "This far" to see that I've made it to a particular part of the code:
say "About to parse file";
$twig->parsefile($script_path);
say "About to print file";
$twig->print_to_file($script_path);
In your handler, there are some things that you can improve. Since you don't want to do anything in the case that the value is already true, don't even think about that case:
log => sub {
if ($_->text ne 'true'){
print $_->text;
subp4open($xmlfile);
$_->set_text( 'true' );
}
},
I might be inclined to always update the value if you want them all to be true. I don't care what it was before as long as I get what I want at the end (although this way tweak your source control a bit):
log => sub { $_->set_text( 'true' ) },
Also, I might consider doing any Perforce magic outside of the twig handler. Get the lock on the file before you give it to Twig. If you can't get that lock, you might as well not process it. Then, remember to release it later. If you do all of those operations close together, you're less likely to forget a step.
According to this scenario.
Used to test it using sample message - text. And it works fine.
I should implement that by using buttons (callbacks).
The real problem is :
Sample text :
context.Activity.Id
"mid.$cAAOmNGtaSJ9i1_f223cprKylFOpb"
on button click (callback) :
context.Activity.Id
"9Yw5TAcq1qr"
This is the code i used to post human support button :
msg.AttachmentLayout = AttachmentLayoutTypes.Carousel;
msg.Attachments = new List<Attachment>();
ThumbnailCard thumbnailCard2 = new ThumbnailCard()
{
Buttons = new List<CardAction>
{
new CardAction ()
{
Value = "Method_Support",
Type = "postBack",
Title = "Human Support"
},
},
};
msg.Attachments.Add(thumbnailCard2.ToAttachment());
await context.PostAsync(msg);
It returns activity.Id : 9Yw5TAcq1qr - as i already mentioned. I can't use that id to query Facebook graph api.
What's the best way to handle scenario like this ?
Is it possible to render all incoming messages as native Facebook (sample-text) messages ?
Is it possible to make call to graph api using callback's activity.id ?
Following my previous answer about how to link the Activity from botFramework to the messages in Conversation in Facebook's Graph API, I had a look to your problem.
Analysis
From my point of view, there is a strange thing in this Activity.Id value because on Facebook's side, all messages have the same id format, even for callbacks.
Here is a capture of 2 messages (1st one is the latest one) from Facebook Graph API:
{
"data": [{
"messages": {
"data": [
{
"id": "m_mid.$cAAC3Jml0RcBi2JK3Clcp01uu8FFe",
"message": "Human support"
},
{
"id": "m_mid.$cAAC3Jml0RcBi2I5bXlcp0kScM7af",
"message": ""
}
]
},
"id": "t_mid.$..."
}]
}
Message with empty text is a ThumbnailCard with a CardAction inside (type = postback, Value = Method_HumanSupport)
Human support is what is sent on the click on the CardAction
What is surprising is that on the bot side, for those 2 messages we have the following values for Activity.Id:
For "id": "m_mid.$cAAC3Jml0RcBi2I5bXlcp0kScM7af" (my Thumbnail) => mid.$cAAC3Jml0RcBi2I5bXlcp0kScM7af, so it's matching
For "id": "m_mid.$cAAC3Jml0RcBi2JK3Clcp01uu8FFe" (the callback) => DWhE2C0VO79, no match at all
For information my ThumbnailCard is made like the OP said:
var msg = context.MakeMessage();
msg.AttachmentLayout = AttachmentLayoutTypes.Carousel;
msg.Attachments = new List<Attachment>();
msg.Text = "updates & support";
ThumbnailCard thumbnailCard2 = new ThumbnailCard()
{
Buttons = new List<CardAction>
{
new CardAction ()
{
Value = "Method_HumanSupport",
Type = "postBack",
Title = "Human support"
}
}
};
I will add a question on Bot Framework GitHub Project.
Edit: issue logged: https://github.com/Microsoft/BotBuilder/issues/2950
Workaround
There are several workaround that will depend on your global use cases: you could try to log the Activity.Id of a "normal" message in the discussion and use this value later when your callback is clicked, or you can also try to change your flow to avoid making this kind of link between Bot Framework and Facebook Graph API.
I have the following query:
SELECT QuoteReference,
CreatedDate,
StartDate,
EndDate,
Operation,
TableName,
OccurredAt,
PerformedBy,
FieldName,
OldValue,
NewValue,
Quotes.CreatedByID,
CompletedDate,
EmailAddress = (SELECT ContactDetails.EmailAddress,
QuoteReference,
ContactDetails.MobilePhoneNumber
FROM ContactDetails,
Quotes,
QuoteCustomers
WHERE ContactDetails.ID = Quotes.ID
AND QuoteCustomers.QuoteID = ContactDetails.ID)
FROM Quotes
JOIN Audit
ON Quotes.ID = Audit.RowId
WHERE Quotes.CreatedDate BETWEEN '20100401' AND '20120830'
AND PaymentReference IS NOT NULL
AND Audit.OccurredAt > Quotes.CompletedDate
AND Quotes.EmailAddress < > NULL
AND TableName = 'Quotes'
AND Quotes.PolicyReference = NULL
AND Quotes.CreatedByID < > 2
ORDER BY Audit.OccurredAt desc
I get this error when executing:
Only one expression can be specified in the select list when the subquery is not introduced with EXISTS.
After adding the second where clause in the subquery. What can I do to fix this?
You are selecting three columns values into one variable.
Change
EmailAddress = (SELECT EmailAddress,
QuoteReference,
ContactDetails.MobilePhoneNumber
FROM ContactDetails,
Quotes,
QuoteCustomers
WHERE ContactDetails.ID = Quotes.ID
AND QuoteCustomers.QuoteID = ContactDetails.ID)
to
EmailAddress = (SELECT EmailAddress
FROM ContactDetails,
Quotes,
QuoteCustomers
WHERE ContactDetails.ID = Quotes.ID
AND QuoteCustomers.QuoteID = ContactDetails.ID)
Your subquery have to return ONE row and ONE column so it should looks like:
EmailAddress = (SELECT TOP 1 ContactDetails.EmailAddress
FROM ContactDetails,
Quotes,
QuoteCustomers
WHERE ContactDetails.ID = Quotes.ID
AND QuoteCustomers.QuoteID = ContactDetails.ID)