I'm investigating a problem on my PC (more exactly a sharing violation during the xcopy of a bunch of files), and I'm thinking of verifying the event log, but I'd like to investigate all events which occured between the beginning of that xcopy and the end of it, something like:
wevtutil qe * /q:"*[System[TimeCreated[#SystemTime>='2017-04-11T03:30:00' and #SystemTime<'2017-04-11T03:33:00']]]" /f:text
(the timestamps are retrieved from the commands echo [!TIME!], one just before and one just behind the xcopy command)
This command is not accepted, as the usage of * is not permitted while working with wevtutil qe. I can have a look inside the event viewer but then I'd need to investigate all possible logs (and I'm not very familiar with this).
Is there a way to interrogate all event logs and filter them on timestamps?
While Microsoft and others say the format is UTC it is actually a variation, if you query the values you will see the difference, no "T" for starters.
The format is the correct time for the BIAS at the end of the string, so for me in a +600 TZ with a bias of "+600" on end of the WMI time string the values can be read as local time (as many Microsoft samples assume is ALWAYS the case).
If however the bias is "-000" for example, in my case the values are all 10 hours (600 minutes) older as you'd expect.
Related
I have an issue with capturing exchange logs from a customer production environment. The logs exist in a set of directories, and are labeled such as:
-- .../dir1/http_2021101002-1.log
-- .../dir1/http_2021101003-1.log
-- .../dir1/http_2021101004-1.log
-- .../dir1/http_2021101004-2.log (if previous log reached max size for example)
These logs would cover October 10 at hour 02, 03, and two sets at 04.
Now, I can add an ossec entry in the config such as:
<localfile>
<log_format>syslog</log_format>
<location>C:\Program Files\microsoft\exchange server\v15\logging\httpproxy\Ecp\HttpProxy_%Y%m%d%H-2.log</location>
</localfile>
The problem though, is this hour field, as with the rest of the date field strftime replacements only gets evaluated on agent start/restart. So, it might catch the first one of the day if I add an entry for hour zero, but it will not catch any logs after that unless I restart each hour. Is there any way around this to capture logs from each hour of the day? I cannot mix wildcard and strftime so that is out, and if I wildcard the entire file name wazuh will crash under load because of a know issue where it can only handle so much traffic before dying HARD. Any thoughts?
Logcollector location field only allows year, month and day strftime format strings (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#location). The use of hourly files will not work due to the update of the filename to read is done daily.
As you have said the only solution is to use a wildcard or select the whole directory as a location value. Regarding the overloading issue, there is a few of possible solutions, that can be used together or separately:
Use the age field with a 1h value. This will ignore all files that have not been modified for 1 hour.
Change the logcollector.max_lines in your local_internal_option file (https://documentation.wazuh.com/current/user-manual/reference/internal-options.html?highlight=local_internal_option) to change the maximum number of logs read from the same file in each iteration. Change it according to your environment (10000 by default)
I am working on a complex Modelica model that contains a large set of data, and I need the simulation to keep going until I terminate the simulation process, maybe even for days, so the .mat file could get very large, I got trouble with how to do data processing. So I'd like to ask if there are any methods that allow me to
output the data I need after a fixed time step during simulation, but not using the .mat file after simulation. I am considering using Modelica.Utilities.Stream.Print` function to print the data I need into a CSV file, but I have to write a huge amount of code that prints every variable I need, so I think there should be a better solution.
delete the .mat file during a fixed time step, so the .mat file stored on my PC wouldn't get too large, and don't affect the normal simulation of Dymola.
Long time ago I wrote a small C-program that runs the executable of Dymola with two threads. One of them is responsible for terminating the whole simulation after exceeding an input time limit. I used the executable of this C-program within the standard given mfiles from Dymola. I think with some hacking capabilities, one would be able to conduct the mentioned requirements.
Have a look at https://github.com/Mathemodica/dymmat however I need to warn that the associated mfiles were for particular type of models and the software is not maintained since long time. However, the idea of the C-program would be reproducible.
I didn't fully test this, so please think of this more like "source of inspiration" than a full answer:
In Section "4.3.6 Saving periodic snapshots during simulation" of the Dymola 2021 Release Notes you'll find a description to do the following:
The simulator can be instructed to print the simulation result file “dsfinal.txt” snapshots during simulation.
This can be done periodically using the Simulation Setup options "Complete result snapshots", but I think for your case it could be more useful to trigger it from the model using the function Dymola.Simulation.TriggerResultSnapshot(). A simple example is given as well:
when x > 0 then
Dymola.Simulation.TriggerResultSnapshot();
end when;
Also one property of this function could help, as it by default creates multiple files without overwriting them:
By default, a time stamp is added to the snapshot file name, e.g.: “dsfinal_0.1.txt”.
The format of the created dsfinal_[TIMESTAMP].txt is a bit overwhelming at first, as it contains all information for initializing the model, but there should be everything you need...
So some effort is shifted to the post processing, as you will likely need to read multiple files, but I think this is an acceptable trade-off.
I've stumbled on this page in PostgreSQL wiki, where it's advised to not use BETWEEN with timestamps:
Why not?
BETWEEN uses a closed-interval comparison: the values of both ends of
the specified range are included in the result.
This is a particular problem with queries of the form
SELECT * FROM blah WHERE timestampcol BETWEEN '2018-06-01' AND
'2018-06-08'
This will include results where the timestamp is exactly 2018-06-08
00:00:00.000000, but not timestamps later in that same day. So the
query might seem to work, but as soon as you get an entry exactly on
midnight, you'll end up double-counting it.
Can anyone explain how this "double-counting" can occur?
Often when using BETWEEN, you want to use multiple ranges (in separate queries) which cumulatively cover all the data.
If the next invocation of your query uses BETWEEN '2018-06-08' AND '2018-06-015', then exact midnight of 2018-06-08 will be included in both invocations. On the other hand, if the next one uses BETWEEN '2018-06-09' AND '2018-06-015', then all of 2018-06-08 except for exact midnight has been overlooked. There is no easy way (using BETWEEN) to construct ranges which cover every data point once and only once.
Is there any way to see how many transaction logs a process (agent_id) currently spans? Or list the transaction logs it's currently using/spanning? I.e. is it possible to check if NUM_LOG_SPAN is about to be reached?
We've had an issue recently whereby a long running transaction breached NUM_LOG_SPAN. This was set to 70, when we had 105 logs. We've increased this now but potentially it may still not be enough. We could set NUM_LOG_SPAN to 0, but that's a last resort... what we'd like to be able to do is at least monitor the situation (and not just wait until it hits and causes issues) - to be able to run a command to see if, for example, a process was now using/spanning, say, 90 of the logs? And then we could decide whether to cancel it or not.
We're after something similar to the following statement where you can see the percentage of transaction log usage:
select log_utilization_percent,dbpartitionnum from sysibmadm.log_utilization
-is there anything similar for monitoring processes to ensure they don't cross the NUM_LOG_SPAN threshold?
NB: This is in a SAP system (NW7.3)... perhaps there's something in DBACOCKPIT to view this too?
As far as I can tell you can't calculate this from the monitor functions only, because none of the monitoring functions expose the Start LSN for a unit of work.
You can do this with db2pd, though. Use db2pd -db <dbname> -logs to find the Current LSN, and use db2pd -db <dbname> -transactions to find Firstlsn for the particular unit of work.
With these two numbers, you can use the formula
(currentLSN - firstLSN)
Logs Files Spanned = -------------------------
logfilsiz * 4096
(You should convert the hex values for current LSN and firstLSN returned by db2pd to decimal values).
I wrote a simple powershell script that recursively walks a file tree and returns the paths of each node along with the time of its creation in tab-separated form, so that I can write it out to a text file and use it to do statistical analysis:
echo "PATH CREATEDATE"
get-childitem -recurse | foreach-object {
$filepath = $_.FullName
$datecreated = $_.CreationTime
echo "$filepath $datecreated"
}
Once I had done this, however, I noticed that the CreationDate times that get produced by the script are exactly one hour ahead of what Windows Explorer says when I look at the same attribute of the same files. Based on inspecting the rest of my dataset (which recorded surrounding events in a different format), it's clear that the results I get from explorer are the only ones that fit the overall narrative, which leads me to believe that there's something wrong with the Powershell script that makes it write out the incorrect time. Does anyone have a sense for why that might be?
Problem background:
I'm trying to correct for a problem in the design of some XML log files, which logged when the users started and stopped using an application when it was actually supposed to log how long it took the users to get through different stages of the workflow. I found a possible way to overcome this problem, by pulling date information from some backup files that the users sent along with the XML logs. The backups are generated by our end-user application at the exact moment when a user transitions between stages in the workflow, so I'm trying to bring information from those files' timestamps together with the contents of the original XML log to figure out what I wanted to know about the workflow steps.
Summary of points that have come out in comment discussion:
The files are located on the same machine as the script I'm running (not a network store)
Correcting for daylight savings and time zones has improved the data quality, but not for the specific issue posed in the original question.
I never found the ultimate technical reason for the discrepancy between the timestamps from powershell vs. explorer, but I was able to correct for it by just subtracting an hour off all the timestamps I got from the powershell script. After doing that, however, there was still a large amount of disagreement between the time stamps I got from out of my XML log files and the ones I pulled from the filesystem using the powershell script. Reasoning that the end-users probably stayed in the same time zone when they were generating the files, I wrote a little algorithm to estimate the time zone of each user by evaluating the median amount of time between steps 1 and 2 in the workflow and steps 2 and 3. If there was a problem with the user's time zone, one of those two timespans would be negative (since the time of the step 2 event was estimated and the times of the steps 1 and 3 events were known from the XML logs.) I then rounded the positive value down to the nearest hour and applied that number of hours as an offset to that user's step 2 times. Overall, this took the amount of bad data in my dataset from 20% down to 0.01%, so I'm happy with the results.
In case anyone needs it, here's the code I used to make the hour offset in the timestamps (not powershell code, this was in a C# script that handled another part of data processing):
DateTime step2time = DateTime.Parse(LastModifyDate);
TimeSpan shenanigansCorrection = new TimeSpan(step2time.Hour-1,step2time.Minute,step2time.Second);
step2time= step2time.Date + shenanigansCorrection;
The reason for redefining the step2time variable is that DateTimes aren't mutable in .NET.