I'm having an issue trying to accurately parse log files which time stamps may (or may not parse multiple files.
Example:
3 log files that get rotated so oldest logs are in time.log.3 and newest logs are in time.log.1
snapshot of the 3 log files
time.log.1
line 17 | 2021-03-17 21:08
line 18 | test5
line 19 | 2021-03-17 21:09
line 20 | test6
line 21 | 2021-03-18 22:10
line 22 | test7
line 23 | 2021-03-18 22:11
line 24 | test8
time.log.2
line 09 | 2021-03-16 21:04
line 10 | test9
line 11 | 2021-03-16 22:05
line 12 | test10
line 13 | 2021-03-17 19:06
line 14 | test11
line 15 | 2021-03-17 19:07
line 16 | test12
time.log.3
line 01 | 2021-03-16 19:00
line 02 | test1
line 03 | 2021-03-16 19:01
line 04 | test2
line 05 | 2021-03-16 20:02
line 06 | test3
line 07 | 2021-03-16 20:03
line 08 | test4
and so I parse them in reverse order using for loop and sed pattern match
for (( i = 3; i >= 1; --i )); do sudo cat time.log.$i | sed -nre '/2021-03-16 19:00/,/2021-03-16 19:01/p;'; done
all is good if you know exact timestamp:
line 01 | 2021-03-16 19:00
line 02 | test1
line 03 | 2021-03-16 19:01
but if I only know approx time range (like changeing it to 19:02) then because that exact pattern is not found it prints the rest of file...
for (( i = 3; i >= 1; --i )); do sudo cat time.log.$i | sed -nre '/2021-03-16 19:00/,/2021-03-16 19:02/p;'; done
yields this, but the 20:XX timestamps are outside the desire window of 19:00 thru 19:02... this is undesirable.
line 01 | 2021-03-16 19:00
line 02 | test1
line 03 | 2021-03-16 19:01
line 04 | test2
line 05 | 2021-03-16 20:02
line 06 | test3
line 07 | 2021-03-16 20:03
line 08 | test4
I need it to print only timestamps within a range, but am having trouble finding the answer... I've tried many google searches for awk, sed, perl, yet cannot land an answer... I feel I need to get past this hurdle before expecting larger time windows spanning multiple files to work...
Expounding on some of the criteria. The parse cannot happen upon the pipe (|) character, those Line number exists only to give context to the reverse order nature log file rotations and log info within the files. The parse can only happen upon the timestamps alone, and needs to contain all the "test#" info too in between the the timestamps, for not every line will have a timestamp.
This type of timestamp can be compared as a string. sed doesn't do comparisons, but awk does:
awk -F' [|] ' '$NF>="2021-03-16 19:00" && $NF<"2021-03-16 19:02"' time.log.{3,2,1}
Note that {3,2,1} is not a filename generating pattern (aka glob), and awk will quit if it encounters a non-existent file argument. It would be better if the files were named in such a way that log.* would work (i.e. sorted in ascending order lexicographically).
When not every line contains a timestamp, detecting if there is a timestamp might complicate things:
awk -F ' [|] ' '
$NF~/^[0-9]{4}(-[0-9]{2}){2} [0-9]{2}:[0-9]{2}$/ {x=$NF} x>=a; x>=b{exit}
' a='2021-03-16 21:00' b='2021-03-17 19:00' time.log.{3,2,1}
Or maybe the lines are grouped in pairs:
awk -F ' [|] ' '
NR%2==1 {x=$NF} x>=a; x>=b{exit}
' a='2021-03-16 21:00' b='2021-03-17 19:00' time.log.{3,2,1}
Maybe the input doesn't really contain ... | :
awk '
NR%2==1 {x=$0} x>=a; x>=b{exit}
' a='2021-03-16 21:00' b='2021-03-17 19:00' time.log.{3,2,1}
Related
I need to read multiple files and count the no. of lines in each file. Then write all this information in a single CSV file with the file path information.
I could write a code (with help from forums) that will help to read all files and print the information as required.
Script:
Get-ChildItem -Path C:\Logs\test *.log | Select-Object -Property FullName,#{Name="LineCount";$Expression={#(Get-Content -Path $_.FullName).Length}}
Output of script:
FullName LineCount
-------- ---------
C:\Logs\test\ANCD065.log 296
C:\Logs\test\ANCE066.log 287
Problems:
The problem is not able to write the above information in a CSV file.
Write "what parameter need pass not clear to me" | Out-File C:\Log\OP.CSV
The script reads all lines, is there a way to start reading line after certain lines? An example in the given file need is count lines from line 9 (start) to the end(could be 300).
1. 3.03 MET DATA RINEX VERSION / TYPE
2. cnvtToRINEX 3.14.0 convertToRINEX OPR 20220511 072939 UTC PGM / RUN BY / DATE
3. ----------------------------------------------------------- COMMENT
4. XYXTY3 MARKER NAME
5. UBWTZ3 MARKER NUMBER
6. 3 PR TD HR # / TYPES OF OBSERV
7. 0083019.4060 0025010.0967 0253356.6176 0.0000 PR SENSOR POS XYZ/H
8. END OF HEADER
9. 19 03 02 00 00 00 946.0 8.5 93.0
10. 19 03 02 00 05 00 946.0 8.4 93.4
11. 19 03 02 00 10 00 946.0 8.4 93.4
12. 19 03 02 00 15 00 946.0 8.4 94.2
13. 19 03 02 00 20 00 946.0 8.5 93.1
14. 19 03 02 00 25 00 946.1 8.7 90.2
15. 19 03 02 00 30 00 946.2 8.4 92.0
16. 19 03 02 00 35 00 946.3 8.3 93.2
The problem is not able to write the above information in a CSV file.
This should be as simple as appending an Export-Csv command to the pipeline:
Get-ChildItem -Path C:\Logs\test *.log |
Select-Object -Property #(
'FullName'
#{ Name = "LineCount"; Expression = {
#(Get-Content -Path $_.FullName).Length
}}
) |
Export-Csv output.csv -NoTypeInformation
Note that my Select-Object changes are just for code readability.
The script reads all lines, is there a way to start reading line after
certain lines? An example in the given file need is count lines from
line 9 (start) to the end(could be 300).
You may use Select-Object -Skip n to skip given number of lines from the beginning. Another improvement is to use Measure-Object to count number of items. In your current code, you have to read the whole log file into an array in memory, before being able to count lines. With Measure-Object, the file will be streamed, requiring much less memory for processing.
(Get-Content -Path $_.FullName | Select-Object -Skip 8 | Measure-Object).Count
Complete solution:
Get-ChildItem -Path C:\Logs\test *.log |
Select-Object -Property #(
'FullName'
#{ Name = "LineCount"; Expression = {
(Get-Content -Path $_.FullName | Select-Object -Skip 8 | Measure-Object).Count
}}
) |
Export-Csv output.csv -NoTypeInformation
Delete the only occurrence of "(123456789)" anywhere in a text file including both open and close parenthesis
get-content "D:\testing.txt" | % {$_ -replace "(\d{9})",""} | Out-File D:\temp\out.txt
Your regex is incorrect -> try "\(\d{9}\)".
You can check your code via this statement:
'12 abc"(123456789)" 90 efg' -replace '"\(\d{9}\)"'
Output:
12 abc 90 efg
Above code is available only under this tio.run link.
In my powershell profile I have a shortcut to docker ps, with the format set to improve readability
function dps {
docker ps --format "table {{.Names}}\t{{.ID}}\t{{.Status}}\t{{.Ports}}" | Sort-Object
# ????
# Tee-Object -Variable lines |
# select -first 1 |
# Write-Host;
# $lines | Sort-Object | Write-Host # Prints nothing
}
Sorting the lines in this way works nicely and groups my containers by name, but the table header ends up in the middle of the output.
You can see that I tried to separate out the first line of output (the header), but I couldn't figure out how to get the table to continue to print. How can I keep the table header at the top while sorting and printing the table?
Sample Output
database_A 27b33272e64c Up 15 hours 3306/tcp, 33060/tcp
database_B 1b1662223f17 Up 15 hours 33060/tcp, 0.0.0.0:33640->3306/tcp
database_C 8f98fc0890cc Up 15 hours 3306/tcp, 33060/tcp
framework_A a0d829729c8e Up 15 hours (unhealthy) 5050/tcp, 0.0.0.0:5170->80/tcp
NAMES CONTAINER ID STATUS PORTS
service_A 8708aec85ea7 Up 15 hours (healthy) 0.0.0.0:5100->8080/tcp
service_B 7931365f450e Up 15 hours (unhealthy) 5160/tcp, 0.0.0.0:5160->80/tcp
service_C e9b9272011d8 Up 15 hours (unhealthy) 5160/tcp, 0.0.0.0:5110->80/tcp
I suggest to convert tab delimited output to an object first, then sort.
function dps {
docker ps --format "table {{.Names}}\t{{.ID}}\t{{.Status}}\t{{.Ports}}" |
ConvertFrom-Csv -delimiter "`t" |Sort-Object Names
}
Should return this:
NAMES CONTAINER ID STATUS PORTS
----- ------------ ------ -----
database_A 27b33272e64c Up 15 hours 3306/tcp, 33060/tcp
database_B 1b1662223f17 Up 15 hours 33060/tcp, 0.0.0.0:33640->3306/tcp
database_C 8f98fc0890cc Up 15 hours 3306/tcp, 33060/tcp
framework_A a0d829729c8e Up 15 hours (unhealthy) 5050/tcp, 0.0.0.0:5170->80/tcp
service_A 8708aec85ea7 Up 15 hours (healthy) 0.0.0.0:5100->8080/tcp
service_B 7931365f450e Up 15 hours (unhealthy) 5160/tcp, 0.0.0.0:5160->80/tcp
service_C e9b9272011d8 Up 15 hours (unhealthy) 5160/tcp, 0.0.0.0:5110->80/tcp
Try using same output without headers:
docker ps --format "{{.Names}}\t{{.ID}}\t{{.Status}}\t{{.Ports}}" | Sort-Object
LotPing's answer was close but duplicated the header information. Combining it with Volodymyr's answer gives us this:
function dps {
docker ps --format "{{.Names}}\t{{.ID}}\t{{.Status}}\t{{.Ports}}" |
ConvertFrom-CSV -Delimiter "`t" -Header ("Names","Id","Status","Ports") |
Sort-Object Names
}
This has the desired output! Thanks everyone.
How to delete lines containing a specific character in a text file with PowerShell?
Input file example:
12314+000000 00000000
20 300 1238238948023
A+A+000000 00000000
20 500 1238238948023
Output file example:
20 300 1238238948023
20 500 1238238948023
So it will delete lines containing + or specific characters or words.
The simplest way is to use new file to write data in it:
Get-Content "D:\test.txt" | Where-Object {$_ -notmatch '\+'} | Set-Content "D:\out.txt"
That will give:
20 300 1238238948023
20 500 1238238948023
In out.txt.
This is used in this answer and this
I'm writing a perl script which needs to parse a file (or more specifically, a known sub in the file) find a specific function call, and retrieve all the arguments for that call. For example:
sub MySub {
# some code here...
# ...
MyFunction ([q{I}, q{want}], [qw(to get)],
["these", "arguments"]);
# ...
# more code...
}
I've been able to parse through the file grab the appropriate line/lines for the function call, and then I wrote my own local "MyFunction" which can grab the args. Then I use a stringy eval to call that function and get the arguments. Example:
sub MyFunction {
return #_;
}
# Not actually doing this, but this is what I might get from parsing:
my $lines = 'MyFunction ({arg1 => "hashref"}, ["arg2", "arrayref"], "etc...");';
my #arguments = eval $lines;
This works great, except for the fact that I'm using a stringy eval. The problem with parsing the argument directly out of the string is there are many different formats (including multiple lines) that are possible. So I guess is there an easy way to parse the arguments directly out of the string, or to call the function without a stringy eval?
Thanks a lot for any help!
You don't have to eval if you don't want to. You can use the name to get the symbol and then use that to get the code ref.
my $func = do {
no strict 'refs';
*{'main::MyFunction'}{CODE};
};
my #arguments = $func->( {arg1 => "hashref"}, ["arg2", "arrayref"], "etc..." );
Parsing Perl is hard (and there's a folk theorem that only Perl can parse Perl). Parsing perl with your own hand-rolled string hack is likely to end in disaster in all but the simplest cases. What you really want is a tool that can parse Perl and give you the arguments.
Our DMS Software Reegineering Toolkit can do this. See parse tree below,
obtained from a file that contains exactly OP's MySub example text:
C:\DMS\Domains\Perl\Tools\Parser>run ..\domainparser ++AST C:\temp\test.pl
Perl~Perl5 Domain Parser Version 2.6.15 DEBUG BUILD
Copyright (C) 1996-2015 Semantic Designs, Inc; All Rights Reserved; SD Confidential
Powered by DMS (R) Software Reengineering Toolkit
Using encoding Unicode-UTF-8?ANSI +CRLF +1 /^I
432 tree nodes in tree.
(Perl#Perl~Perl5=1#5c21d20^0 Line 1 Column 1 File C:/temp/test.pl
(statements#Perl~Perl5=10#5c21c80^1#5c21d20:1 Line 1 Column 1 File C:/temp/test.pl
(subroutine_head#Perl~Perl5=551#5c73680^1#5c21c80:1 Line 1 Column 1 File C:/temp/test.pl
(IDENTIFIER#Perl~Perl5=573#5c735e0^1#5c73680:1[`MySub'] Line 1 Column 5 File C:/temp/test.pl)IDENTIFIER
(prototype#Perl~Perl5=552#5c73640^1#5c73680:2 Line 1 Column 11 File C:/temp/test.pl)prototype
(attributes#Perl~Perl5=554#5c73660^1#5c73680:3 Line 1 Column 11 File C:/temp/test.pl)attributes
)subroutine_head#5c73680
(block#Perl~Perl5=4#5c21640^1#5c21c80:2 Line 1 Column 11 File C:/temp/test.pl
precomment 3:1 `# ...'
precomment 3:2 `# more code...'
(statement#Perl~Perl5=21#5c7cf40^1#5c21640:1 Line 5 Column 5 File C:/temp/test.pl
|(call_statement#Perl~Perl5=561#5c209a0^1#5c7cf40:1 Line 5 Column 5 File C:/temp/test.pl
| (IDENTIFIER#Perl~Perl5=573#5c73920^13#5c210a0:1#5c73ec0:1#5c778c0:1#5c21080:1#5c73e00:1#5c77860:1#5c209a0:1#5c7f860:1#5c73d60:1#5c77840:1#5c7f840:1#5c73560:1#5c777e0:1[`MyFunction'] Line 5 Column 5 File C:/temp/test.pl
| precomment 0:1 `# some code here...'
| precomment 0:2 `# ...')IDENTIFIER
| (L20#Perl~Perl5=514#5c7f240^3#5c206c0:2#5c209a0:2#5c21020:1 Line 5 Column 17 File C:/temp/test.pl
| (L20#Perl~Perl5=514#5c7b540^3#5c7f240:1#5c7c640:1#5c738e0:1 Line 5 Column 17 File C:/temp/test.pl
| (L1_term#Perl~Perl5=187#5c7a700^11#5c7abc0:1#5c7b540:1#5c7c860:1#5c7d280:1#5c7b620:1#5c7ad60:1#5c7a9a0:1#5c20840:1#5c7c1c0:1#5c7b8a0:1#5c7aaa0:1 Line 5 Column 17 File C:/temp/test.pl
| |(L20#Perl~Perl5=514#5c77da0^2#5c7a700:1#5c7a380:2 Line 5 Column 18 File C:/temp/test.pl
| | (constant#Perl~Perl5=101#5c73180^6#5c77bc0:1#5c77da0:1#5c77d60:1#5c77a00:1#5c7a420:1#5c77ae0:1 Line 5 Column 18 File C:/temp/test.pl
| | (string_prefix#Perl~Perl5=120#5c731a0^1#5c73180:1 Line 5 Column 18 File C:/temp/test.pl)string_prefix
| | (ORDINARY_STRING#Perl~Perl5=604#5c77400^1#5c73180:2[`I'] Line 5 Column 19 File C:/temp/test.pl)ORDINARY_STRING
| | )constant#5c73180
| | (constant#Perl~Perl5=101#5c77e00^6#5c7a6a0:2#5c7a740:2#5c7a2a0:2#5c7a160:2#5c77da0:2#5c7a140:2 Line 5 Column 24 File C:/temp/test.pl
| | (string_prefix#Perl~Perl5=120#5c7a020^1#5c77e00:1 Line 5 Column 24 File C:/temp/test.pl)string_prefix
| | (ORDINARY_STRING#Perl~Perl5=604#5c7a000^1#5c77e00:2[`want'] Line 5 Column 25 File C:/temp/test.pl)ORDINARY_STRING
| | )constant#5c77e00
| |)L20#5c77da0
| )L1_term#5c7a700
| (L1_term#Perl~Perl5=187#5c7b420^14#5c7b600:2#5c7b780:2#5c7b620:2#5c7c7c0:1#5c7e880:1#5c20600:1#5c7d160:1#5c7c0c0:1#5c7bf00:2#5c7bf80:2#5c20c20:1#5c7c360:1#5c7b540:2#5c7b560:2 Line 5 Column 34 File C:/temp/test.pl
| |(qw_constant#Perl~Perl5=113#5c7b260^1#5c7b420:1 Line 5 Column 35 File C:/temp/test.pl
| | (WORD_LIST_STRING_START#Perl~Perl5=630#5c7b180^1#5c7b260:1["("] Line 5 Column 37 File C:/temp/test.pl)WORD_LIST_STRING_START
| | (word_list#Perl~Perl5=117#5c7b240^1#5c7b260:2 Line 5 Column 38 File C:/temp/test.pl
| | (word_list#Perl~Perl5=117#5c7b200^1#5c7b240:1 Line 5 Column 38 File C:/temp/test.pl
| | (word_list#Perl~Perl5=116#5c7b1c0^1#5c7b200:1 Line 5 Column 38 File C:/temp/test.pl)word_list
| | (WORD#Perl~Perl5=633#5c7b1a0^1#5c7b200:2[`to'] Line 5 Column 38 File C:/temp/test.pl)WORD
| | )word_list#5c7b200
| | (WORD#Perl~Perl5=633#5c7b1e0^1#5c7b240:2[`get'] Line 5 Column 41 File C:/temp/test.pl)WORD
| | )word_list#5c7b240
| | (STRING_END#Perl~Perl5=620#5c7b220^1#5c7b260:3[")"] Line 5 Column 44 File C:/temp/test.pl)STRING_END
| |)qw_constant#5c7b260
| )L1_term#5c7b420
| )L20#5c7b540
| (L1_term#Perl~Perl5=187#5c20260^12#5c20e80:2#5c20ea0:2#5c20f20:2#5c20f60:2#5c205c0:2#5c20600:2#5c20620:2#5c20380:2#5c7f240:2#5c7f260:2#5c7e880:2#5c7e8a0:2 Line 6 Column 9 File C:/temp/test.pl
| (L20#Perl~Perl5=514#5c7e7e0^1#5c20260:1 Line 6 Column 10 File C:/temp/test.pl
| |(constant#Perl~Perl5=103#5c7d820^5#5c7de60:1#5c7e7e0:1#5c7e140:1#5c7f940:1#5c7dd60:1 Line 6 Column 10 File C:/temp/test.pl
| | (INTERPOLATED_STRING_START#Perl~Perl5=621#5c7d5e0^2#5c7d840:1#5c7d820:1["""] Line 6 Column 10 File C:/temp/test.pl)INTERPOLATED_STRING_START
| | (interpolated_string_content#Perl~Perl5=130#5c7d800^2#5c7d840:2#5c7d820:2 Line 6 Column 11 File C:/temp/test.pl
| | (interpolated_string_content#Perl~Perl5=129#5c7d7c0^1#5c7d800:1 Line 6 Column 11 File C:/temp/test.pl)interpolated_string_content
| | (ORDINARY_INTERPOLATED_STRING_CONTENT#Perl~Perl5=642#5c7c6e0^1#5c7d800:2[`these'] Line 6 Column 11 File C:/temp/test.pl)ORDINARY_INTERPOLATED_STRING_CONTENT
| | )interpolated_string_content#5c7d800
| | (STRING_END#Perl~Perl5=620#5c7d7e0^2#5c7d840:3#5c7d820:3["""] Line 6 Column 16 File C:/temp/test.pl)STRING_END
| |)constant#5c7d820
| |(constant#Perl~Perl5=103#5c7dc40^3#5c7e7e0:2#5c7e840:2#5c7e8e0:2 Line 6 Column 19 File C:/temp/test.pl
| | (INTERPOLATED_STRING_START#Perl~Perl5=621#5c76320^4#5c7dec0:1#5c7e420:1#5c7e400:1#5c7dc40:1["""] Line 6 Column 19 File C:/temp/test.pl)INTERPOLATED_STRING_START
| | (interpolated_string_content#Perl~Perl5=130#5c76800^4#5c7dec0:2#5c7e420:2#5c7e400:2#5c7dc40:2 Line 6 Column 20 File C:/temp/test.pl
| | (interpolated_string_content#Perl~Perl5=129#5c76140^1#5c76800:1 Line 6 Column 20 File C:/temp/test.pl)interpolated_string_content
| | (ORDINARY_INTERPOLATED_STRING_CONTENT#Perl~Perl5=642#5c76120^1#5c76800:2[`arguments'] Line 6 Column 20 File C:/temp/test.pl)ORDINARY_INTERPOLATED_STRING_CONTENT
| | )interpolated_string_content#5c76800
| | (STRING_END#Perl~Perl5=620#5c76180^4#5c7dec0:3#5c7e420:3#5c7e400:3#5c7dc40:3["""] Line 6 Column 29 File C:/temp/test.pl)STRING_END
| |)constant#5c7dc40
| )L20#5c7e7e0
| )L1_term#5c20260
| )L20#5c7f240
|)call_statement#5c209a0
)statement#5c7cf40
)block#5c21640
)statements#5c21c80
)Perl#5c21d20
You still have to walk over the parse tree and pick out the arguments. If you do pick up subtrees, they can be prettyprinted to generate the surface syntax string.