I cannot think of how to query a splunk. I have log "Waiting for changelog lock.." and now I need to select all occurrences if in 10 minutes time period, starting from after this is printed, there is no log saying "Successfully acquired change log lock". This does not work, as it checks if same log contains it: index="[there goes my index]" | spath log | search NOT log="*Successfully acquired change log lock*" AND log='Waiting for changelog lock..' earliest=-10m#m latest=now
You're right in that the current query expects one field to contain two different values, which will never work. Instead, use OR and then pick off the most recent event. If it's "Successfully..." then all is well; otherwise, it's a problem.
index="[there goes my index]" earliest=-10m#m latest=now
| spath log
| search NOT log="*Successfully acquired change log lock*" OR log='Waiting for changelog lock..'
| dedup log
| where log="Waiting for changelog lock"
Related
I've been work on Loki for centralized logging with Grafana. I want to 'Explore' log by query without using time control on top of the Grafana. I wonder if its possible to add range time manually by query (not the time control provided by grafana)?
It's probably like
{job=docker-container} |~ "error" | startsAt = formatTime | endsAt = formatTime
I didn't found any variables that can describe control time range though, also for the labels
I'm working with DataStage 11.3 with parallel jobs.
My input is a date and I use the "DateOffsetByComponents" function in a -transformer- stage to obtain 4 dates with different rules, these 4 results ends in different -sequential file- stages.
The next step of my transformation is to make a query in Sybase but, the conditional clause "Where" uses the 4 dates mentioned before as parameters to get the proper information from the DB.
Does anybody have an idea of how can I read the date in each sequential file and put these as a parameter in the next step?
I read a similar question of it in which, he suggested use the -execute command- stage in a Sequence job but I'm new using DataStage and it isn't clear for me on how can I achieve this, although, I can see that, in this type of job (Sequence) you can select parameters that are contained in others -job activities- stages.
Thanks a lot in advance
Yes, an Execute Command activity to read the date from each file is the way to go. Figure out an operating system command to extract the information you need, perhaps some combination of grep and sed, and use this as the command. The command output is available in an activity variable called $CommandOutput.
For example, if your file contains the line "Date: 2021-11-17" then a suitable command might be grep Date: filename | cut -f2
You might like to add a tr command to the pipeline to remove the trailing newline character.
You'll need one Execute Command activity per file. Access the required date from the $CommandOutput activity variable for each.
When I run the ingest utility with the delete statement it gives the number of rows inserted as 0 and doesn't show the number of rows deleted. Is there any option to show the number of rows deleted?
I have included the output message of the ingest utility and the code
output
------
Number of rows read = 255
Number of rows inserted = 0
Number of rows rejected = 0
code
----
db2 "ingest from file mypipe format delimited(
$field1 CHAR(9),
$field2 DATE 'yyyy-mm-dd'
)
Delete from mytable where dob = $field2"
The documentation specifies initially that the summary report only contains the number of rows read, inserted , rejected. This is perhaps what you are seeing.
Quote from documentation:
Messages from the INGEST command If the utility read at least one
record from the input source, the utility issues a summary of the
number of rows read, inserted, and rejected (similar to the import and
load utilities) and a successful completion message.
However, on the same page a later statement is:
Number of rows inserted (updated, deleted, merged)
The number of rows affected by the execution of the SQL statement against the target table and committed to the database. The message
says "inserted", "updated", "deleted", or "merged", depending on the
SQL statement.
So the behaviour for your case seems unhelpful, and IBM could make it better by additionally including the count of rows deleted, and count of rows updated when the sole SQL statement is DELETE. I tested this behaviour with Db2-LUW v11.5.6.0".
Even when the delete statement is replaced by a MERGE with WHEN MATCHED THEN DELETE the summary report excludes the count of deleted rows. Undesirable behaviour.
If you have a support contract, you could open a ticket with IBM asking for a workaround or fix as there may be some regression here.
I am creating a Data Comparison/Verification script using SQL and Spoon PDI. We're moving data between two servers, and to make sure we've got all the data we have SQL queries showing a date then the quantity of rows transferred.
Example:
Serv1: 20150522 | 100
Serv2: 20150522 | 100
The script will then try to union these values, and if it fails we'll get a fail email. However, we wish to change this setup to write the outcome to a text file, and based on that text file send either a pass or fail email.
The idea behind this is we have multiple tables we're comparing, so we wish to write all the outcomes of each comparison (eight) to a text file and based off the final text file, send the outcome - rather than spamming our email inbox if multiple steps fail.
The format of the text file we wish to have is either match -> send email or mismatch [step-name] [date] -> send email.
Usually I wouldn't ask a question if I haven't tried anything first, but I've searched everywhere on Google, tried the knowledge I currently have and nothing is going the way I wish it to. I believe this is due to the logic I am using.
I am not asking for a solution to this, or for someone to do it for me. I am simply asking for guidance along the correct path.
I would do this in a transformation where there are steps for each union where the result of each step is the comparison_name and the result. This would result in a data set at the end that looks something like this:
comparison_name | result
Union A | true
Union B | false
Union C | true
You would then be able to output those results to a text file in another step to get your result file to sent out regardless of whether the job passed or failed.
Lastly you would loop through the result row in the stream, and if all are true, you could do an email step to send out a "pass" email, and if one is false, send out a "fail" email.
EDIT:
To get the date of the pass or fail you could either get the date from each individual union query result by adding it to the query like so:
SELECT CURRENT_DATE
Or you could use the Get System Info step in spoon which has multiple ways of injecting the current date into the data stream. (system date fixed, start date range of the transformation, today 00:00:00, etc.)
I have an BPM application where I am polling some rows from DB2 database at every 5 mins with a scheduler R1 with below query -
- select * from Table where STATUS = 'New'
based on rows returned I do some processing and then change the status of these rows to 'Read'.
But while this processing is being completed, its takes more than 5 mins and in meanwhile scheduler R1 runs and picks up some of the cases already picked up in last run.
How can I ensure that every scheduler picks up the rows which were not selected in last run. What changes do i need to do in my select statement? Please hep.
How can I ensure that every scheduler picks up the rows which were not selected in last run
You will need to make every scheduler aware of what was selected by other schedulers. You can do this, for example, by locking the selected rows (SELECT ... FOR UPDATE). Of course, you will then need to handle lock timeouts.
Another option, allowing for better concurrency, would be to update the record status before processing the records. You might introduce an intermediary status, something like 'In progress', and include the status in the query condition.