I have an issue with capturing exchange logs from a customer production environment. The logs exist in a set of directories, and are labeled such as:
-- .../dir1/http_2021101002-1.log
-- .../dir1/http_2021101003-1.log
-- .../dir1/http_2021101004-1.log
-- .../dir1/http_2021101004-2.log (if previous log reached max size for example)
These logs would cover October 10 at hour 02, 03, and two sets at 04.
Now, I can add an ossec entry in the config such as:
<localfile>
<log_format>syslog</log_format>
<location>C:\Program Files\microsoft\exchange server\v15\logging\httpproxy\Ecp\HttpProxy_%Y%m%d%H-2.log</location>
</localfile>
The problem though, is this hour field, as with the rest of the date field strftime replacements only gets evaluated on agent start/restart. So, it might catch the first one of the day if I add an entry for hour zero, but it will not catch any logs after that unless I restart each hour. Is there any way around this to capture logs from each hour of the day? I cannot mix wildcard and strftime so that is out, and if I wildcard the entire file name wazuh will crash under load because of a know issue where it can only handle so much traffic before dying HARD. Any thoughts?
Logcollector location field only allows year, month and day strftime format strings (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#location). The use of hourly files will not work due to the update of the filename to read is done daily.
As you have said the only solution is to use a wildcard or select the whole directory as a location value. Regarding the overloading issue, there is a few of possible solutions, that can be used together or separately:
Use the age field with a 1h value. This will ignore all files that have not been modified for 1 hour.
Change the logcollector.max_lines in your local_internal_option file (https://documentation.wazuh.com/current/user-manual/reference/internal-options.html?highlight=local_internal_option) to change the maximum number of logs read from the same file in each iteration. Change it according to your environment (10000 by default)
Related
Some events don't take place at any specific time and instead are meant to be valid for the whole day irrespective of the time zone the user is at.
For the sake of argument, let's say a system sitting on a server (up in the cloud) runs a job at 5 am and imports data from a different system between this run and the last (24 hours ago). The actual user sitting at his desk doesn't know when the job runs, the user only knows that they go to sleep at night, the server crunches all the entries for the day.
The next morning the user wants to see all the entries from yesterday (what ever the job produced) and they go to the app, pull up a calendar input selector and they pick the 5/26/2022 (today being 5/27/2022).
Assuming the developers followed best practices, the client will transform the date into it's UTC version and send it up through an API. Chances are, depending on where the user is located and the server is, there might be a mismatch.
I could send the date up without it being UTC or I could send a UTC date and try to adjust it back to local time so that I could then compare with the date on record (that exists without an actual time zone).
What I am asking is:
What's the more conventional answer to this particular problem?
Is the idea of a date without time or time zone just ridiculous?
Use UNIX Time. It will give you a timestamp that is universal no matter what timezone the user is in. You can then convert it into whatever timezone you want to.
The concern you describe is well solved/addressed by the ISO 8601 dates/time presentation protocol.
All modern software can read/write dates in ISO 8601.
In Unix machines, the correct command is date with option -I
-I[FMT], --iso-8601[=FMT]
output date/time in ISO 8601 format. FMT='date' for date
only (the default), 'hours', 'minutes', 'seconds', or 'ns'
for date and time to the indicated precision. Example:
2006-08-14T02:34:56-06:00
I've currently got this Cron expression that I'm using to trigger a process in UiPath Orchestrator:
0 0 15 21W * ? *
Runs on the closest working day to the 21st of each month at 3pm.
However I need it to run on the next working day at 3pm if the 21st is a non working day.
Tried searching for an answer and nothing quite fit the brief.
I used this website to build my expression (which is a great tool) but it only had an option for 'nearest day' and not next working day given a specific day of month: https://www.freeformatter.com/cron-expression-generator-quartz.html
As you don't need the nearest day, you can't use the functionality of Orchestrator cronjob. I would recommend creating a wrapper process as follows:
Create a new process, let's call it StartJobByCheckingDate
Now create a trigger that starts StartJobByCheckingDate each day at 3pm
So that process is now your manager of your desired process
Now we need to check if it is the 21th day
Here you have different ways to solve it
You could create a DataTable or even a file in the StartJobByCheckingDate process, that contains all the different days where your desired process should be fired (but this is very manual, you might not want to update this every year, so this might not be the smartest but the easiest solution)
The other idea is to check if the current day is the 21th day. If so check if it is Saturday/Sunday (non-working day).
If true: you could now create a empty dummy file somewhere that tracks that the 21th was a non-working day, and the next day you check that file existing, if it exists you check the current day to be a working day, and if so you delete the file again and start your desired process
If false: just start your desired process directly
I think 2. idea would be that best. Sure you have 365 jobs runs/year. But when you keep that helper process smart this will just be seconds.
Another idea instead of using the dummy file, would be to use Entities. Smarter but need some more time to get familiar with.
We have (had) the exact same issue. Since UiPath doesn't offer a feasible solution out of the box, we will work around the restriction using the following strategy: We trigger the actual job daily, considering a custom-built, static NonWorkingDay-list that will just suppress the execution of the robot every day we don't want it to run.
These steps are needed:
Get a list with of all known bank holidays, saturdays and sundays until 2053 or so...
Build a the static exclusion-list using a script that does something like this (pseudocode. I will update the answer once we have actually implemented the solution):
1. get all valid execution dates
loop through every 28th of the month until end of 2053
if the date is in the bankHolidayList then
loop until the next bankDay is found
add it to the list of valid ExecutionDates
else
add the date to the validExecutionDate-list
2. build exclusion-list
loop through every day until end of 2053
if the date is not in the validExecutionDate-list
add it to the exclusionDate-list
Format the csv accordingly and upload it to the orchestrator tenant as a NonWorkingDay-List
Update your trigger to run daily at your desired time, using the uploaded NonWorkDay-Calendar
While the accepted answer will surely work as well, we prefered to go with this approach because having a separate robot that does nothing but executing a UiPath trigger just doesn't seem right to me. With this approach we have no additional code that we potentially need to maintain.
In my oppinion not having a solution for this concern out of the box is a lack of feature that UiPath will (hopefully) fix until end of 2053 ;-)
Cheers
You can configure your trigger to launch oftener, then manage dates at init of your process, but you must set up a list of "holydays" or check in some way.
Also you can use the calendar option of orchestrator (+info)
I'm attempting to create a channel designed to take a binary file from an sFTP site and transfer it to a second sFTP site. The source site updates their files on Friday, so I'm creating the channel to run every Sunday. The main issue is that I want to leave processed files in the source directory, and only process files that have been deposited between the weekly channel runs.
Is something like this doable in Mirth Connect, and what is the most efficient way to accomplish it?
In the Source Map is a variable named fileLastModified, which holds the "last modified date of the file, as an epoch time in milliseconds". You should be able to use this as a condition for Destination Set filtering.
In the Source Transformer, add a Destination Set Filter step. Remove all destinations on the condition of fileLastModified being older than a week, or older than the time elapsed since the last time the channel ran. You can track the last time the channel ran by storing that value in the globalChannelMap, if you need to.
I'm investigating a problem on my PC (more exactly a sharing violation during the xcopy of a bunch of files), and I'm thinking of verifying the event log, but I'd like to investigate all events which occured between the beginning of that xcopy and the end of it, something like:
wevtutil qe * /q:"*[System[TimeCreated[#SystemTime>='2017-04-11T03:30:00' and #SystemTime<'2017-04-11T03:33:00']]]" /f:text
(the timestamps are retrieved from the commands echo [!TIME!], one just before and one just behind the xcopy command)
This command is not accepted, as the usage of * is not permitted while working with wevtutil qe. I can have a look inside the event viewer but then I'd need to investigate all possible logs (and I'm not very familiar with this).
Is there a way to interrogate all event logs and filter them on timestamps?
While Microsoft and others say the format is UTC it is actually a variation, if you query the values you will see the difference, no "T" for starters.
The format is the correct time for the BIAS at the end of the string, so for me in a +600 TZ with a bias of "+600" on end of the WMI time string the values can be read as local time (as many Microsoft samples assume is ALWAYS the case).
If however the bias is "-000" for example, in my case the values are all 10 hours (600 minutes) older as you'd expect.
I wrote a simple powershell script that recursively walks a file tree and returns the paths of each node along with the time of its creation in tab-separated form, so that I can write it out to a text file and use it to do statistical analysis:
echo "PATH CREATEDATE"
get-childitem -recurse | foreach-object {
$filepath = $_.FullName
$datecreated = $_.CreationTime
echo "$filepath $datecreated"
}
Once I had done this, however, I noticed that the CreationDate times that get produced by the script are exactly one hour ahead of what Windows Explorer says when I look at the same attribute of the same files. Based on inspecting the rest of my dataset (which recorded surrounding events in a different format), it's clear that the results I get from explorer are the only ones that fit the overall narrative, which leads me to believe that there's something wrong with the Powershell script that makes it write out the incorrect time. Does anyone have a sense for why that might be?
Problem background:
I'm trying to correct for a problem in the design of some XML log files, which logged when the users started and stopped using an application when it was actually supposed to log how long it took the users to get through different stages of the workflow. I found a possible way to overcome this problem, by pulling date information from some backup files that the users sent along with the XML logs. The backups are generated by our end-user application at the exact moment when a user transitions between stages in the workflow, so I'm trying to bring information from those files' timestamps together with the contents of the original XML log to figure out what I wanted to know about the workflow steps.
Summary of points that have come out in comment discussion:
The files are located on the same machine as the script I'm running (not a network store)
Correcting for daylight savings and time zones has improved the data quality, but not for the specific issue posed in the original question.
I never found the ultimate technical reason for the discrepancy between the timestamps from powershell vs. explorer, but I was able to correct for it by just subtracting an hour off all the timestamps I got from the powershell script. After doing that, however, there was still a large amount of disagreement between the time stamps I got from out of my XML log files and the ones I pulled from the filesystem using the powershell script. Reasoning that the end-users probably stayed in the same time zone when they were generating the files, I wrote a little algorithm to estimate the time zone of each user by evaluating the median amount of time between steps 1 and 2 in the workflow and steps 2 and 3. If there was a problem with the user's time zone, one of those two timespans would be negative (since the time of the step 2 event was estimated and the times of the steps 1 and 3 events were known from the XML logs.) I then rounded the positive value down to the nearest hour and applied that number of hours as an offset to that user's step 2 times. Overall, this took the amount of bad data in my dataset from 20% down to 0.01%, so I'm happy with the results.
In case anyone needs it, here's the code I used to make the hour offset in the timestamps (not powershell code, this was in a C# script that handled another part of data processing):
DateTime step2time = DateTime.Parse(LastModifyDate);
TimeSpan shenanigansCorrection = new TimeSpan(step2time.Hour-1,step2time.Minute,step2time.Second);
step2time= step2time.Date + shenanigansCorrection;
The reason for redefining the step2time variable is that DateTimes aren't mutable in .NET.