i have a log file in hand, looks like this:
0226 111641 (1911) 0 some space separated message containing whatever letters and marks
I need to import it to database, to use filters on it, when troubleshooting is needed. Currently i think powershell is best selection to achieve this, but i'm too green to know specifically how to do it so it can perform actually. I tried to do it like this:
$file = Get-Content "test.txt"
foreach ($line in $file)
{
#Write-Host $line
$a = $line
$month1 = $a[0..1]
$month2 = "$month1"
$month2 = $month2.ToString()
$month = $month2.Replace(" ", "")
$day1 = $a[2..3]
$day2 = "$day1"
$day2 = $day2.ToString()
$day = $day2.Replace(" ", "")
}
... and so on. after that inserting it to database. However, log file is quite big (currently 15MB in 3 weeks, expected to be hundreds of megabytes within months), and already the script takes about 4-5min to process it.
So what i need is method to split four space separated columns from beginning of the line, convert first and second to date and time and add them with message part of the line to database. Separately processing each block of text seems too time consuming, excel for example can process this file within seconds. Is there around some position aware csv-import command?
Thanks.
Found this:
Replace first two whitespace occurrences with a comma using sed
would help, if i would use linux... :(
I'm not sure if the ConvertFrom-Csv or Import-Csv cmdlets can help you since your field delimiter can appear in the message field. Without knowing what these different fields are, I came up with this:
$file = Get-Content "test.txt"
foreach ($line in $file)
{
# Split $line into at most 5 fields
$fields = $line -split ' ', 5;
# fields[0] is a two-digit month followed by a two-digit day
$date = [DateTime]::ParseExact($fields[0], 'MMdd', $null);
$field2 = $fields[1];
$field3 = $fields[2];
$field4 = $fields[3];
$message = $fields[4];
# Process variables here...
}
Using the sample text you provided for $line, the above variables look like this after execution:
PS> Get-Variable -Name #('date', 'field*', 'line', 'message')
Name Value
---- -----
date 2/26/2012 12:00:00 AM
field2 111641
field3 (1911)
field4 0
fields {0226, 111641, (1911), 0...}
line 0226 111641 (1911) 0 some space separated message
message some space separated message
More information will be needed on the format of your data in order to give you a more specific answer.
Related
I'm trying to get text from a Get-WinEvent messageblock, normally I simply use this to get the first sentence:
($_.Message).split(".")[0]
This works fine if the first part is seperated by a .
The Message-Block looks like this:
Remotedesktopdienste: Die Sitzungsanmeldung war erfolgreich:
Benutzer: testlab\testuser.local
Sitzungs-ID: 1
Quellnetzwerkadresse: LOKAL.
I try to extract only the part Remotedesktopdienste: Die Sitzungsanmeldung war erfolgreich: (including the : would be perfect, but not mandatory)
Simply using the : as delimiter did not work (the second part is missing).
After that, I tried to use
($_.Message).split(':',2)[0]
Should be the second : in the message-block, this gives Remotedesktopdienste
Another idea was to use
($_.Message).split('[\r\n]',2)[0]
to use the second new line, also no success with that, the result is Remotedesktopdie
so you want to split by newline:
$MessageArray = $Message.Split("`n")
and then you can iterate through it
$Counter = 0
$MessageBody = ""
foreach($Line in $MessageArray) {
if($Counter -eq 0) {
$FirstLine = $Line
} else {
$MessageBody += $Line + "`n"
}
$Counter++
}
Result is: $FirstLine has your first line .. $MessageBody has the rest of the message
Update: you can even do it with this onliner
$FirstLine,$MessageBody = $Message.Split("`n")
My text file contains G-Code with the code "G94" appearing 5 times at different line numbers.
G94
G94
G94
G94
G94
I need to change the last occurrence of "G94" to
G94
/M16
but I keep getting no edit at all.
I'm trying this:
$text = get-content C:\cncm\g94.txt
$i = 1
$replace = 5 #Replace the 5th match
ForEach ( $match in ($text | select-String "G94" -AllMatches).matches)
{
$index = $match.Index
if ( $i -eq $replace )
{
$text.Remove($index,"G94".length).Insert($index,"G94 n /M16")
}
$i++
}
What am I missing?
$text is an array of strings, how are you calling Remove() without getting an exception? First because Remove() only takes one parameter, second because you can't remove from a fixed length array.
I'm thinking:
$text = get-content C:\cncm\g94.txt
$fifthMatch = ($text | select-string "G94" -AllMatches)[4]
$line = $text[$fifthMatch.LineNumber]
$line = $line.Remove($fifthMatch.index,"G94".length).Insert($fifthMatch.index,"G94 `n /M16")
$text[$fifthMatch.LineNumber] = $line
$text | out-file c:\cncm\g942.txt
Use regexp with negative lookahead on a string that contains the entire file.
Replacing last occurrence in the entire file - (?s) DOTALL mode allows .* to span across new line characters:
$text = [IO.File]::ReadAllText('C:\cncm\g94.txt')
$text = $text -replace '(?s)G94(?!.*?G94)', "G94`n/M16"
Replacing last occurrence in every line - (?m) MULTILINE mode:
$text = [IO.File]::ReadAllText('C:\cncm\g94.txt')
$text = $text -replace '(?m)G94(?!.*?G94)', "G94`n/M16"
I have a text file which contains all historical data. From next day i start receiving the incremental files which can have new records as well as the updated records from the historical file.
lets say I have historical data:
ID|NAME|COUNTRY
----------
1|BRAD PITT|USA
2|GEORGE CLOONEY|USA
3|RAJESH KHANNA|INDIA
Next day i recieve a file
ID|NAME|COUNTRY
----------
1|BRAD PITT|CANADA
4|DANIEL RADCLIFFE|UK
I want to create a new file with
ID|NAME|COUNTRY
----------
1|BRAD PITT|CANADA
2|GEORGE CLOONEY|USA
3|RAJESH KHANNA|INDIA
4|DANIEL RADCLIFFE|UK
ID is the key here, delimiter is constant thruoghout the file.
It can be easily achieved with any database tool but unfortunately my new Job needs it to be achieved with Scripting(Perl/Powershell), which I am not very good at.
I appreciate any help.
If you want to accomplish this using PowerShell - when you receive a new file, you can parse the new data files easily with Import-Csv:
$NewData = Import-Csv -Path '.\incomingFile' -Delimiter '|' -Header ID,Name,Country
$NewData is now an array of objects all having 3 properties named ID, Name and Country with the values from the file.
Assuming that the original file (the file you want to keep updating) is saved in the same format, you'll want to read that file into memory as well, using the same technique.
The fastest way to locate the rows that you need to update, is put all the rows from the persistent file into a hashtable, using the ID property as the key:
$OriginalDataTable = #{}
$OriginalData = Import-Csv -Path '.\originalFile' -Delimiter '|'
$OriginalData |ForEach-Object {
$OriginalDataTable[$_.ID] = $_
}
Now, you can use the same approach to loop through the $NewData array and assign the values to the correct ID in the hashtable with the original data - if the row already exists, it gets overwritten, otherwise it gets added:
$NewData |ForEach-Object {
$OriginalDataTable[$_.ID] = $_
}
Then finally sort the values by ID and export them back to a file:
$OriginalDataTable.Keys |Sort-Object |ForEach-Object {
$OriginalDataTable[$_]
} |Export-Csv -Path '.\updatedFile' -Delimiter '|' -NoTypeInformation
It's been a long time since I wrote perl but it's a very easy language to learn.
The basics of what you want to do are:
load original data
update original data with new data
write out new file
The code to do that is below but should definitely be refactored to suite your direct needs and this isn't a good example if we're looking at large files.
#!/usr/bin/perl
use strict;
use warnings;
my $handle;
my #lines;
my $line;
my #file_data = ();
my #columns = ();
open $handle, '<', "/tmp/sample.txt";
chomp(#lines = <$handle>);
close $handle;
# remove header
shift #lines;
shift #lines;
foreach $line (#lines){
#columns = split("|",$line);
#file_data[$columns[0]] = $line;
}
open $handle, '<', "/tmp/updated.txt";
chomp(#lines = <$handle>);
close $handle;
# remove header
shift #lines;
shift #lines;
foreach $line (#lines){
#columns = split("|",$line);
#file_data[$columns[0]] = $line;
}
# remove blank line
shift #file_data;
open $handle, '>', "/tmp/combined.txt";
print $handle join("\n",#file_data);
close $handle;
I have a string like this $data = .|abc|bcd|cde|.
I need the string like this : abc|bcd|cde.
So I do :
$data =~ s/\|$//; # trim the last '|' out...
$data =~ s/^\.| +//gm ; #trim '.' in the begining
$data =~ s/^\|//; # trim '|' in the begining
But the problem I am facing is, the script is taking too long to execute. Is there any way to complete the whole operation with a single command ??
(Also tried chop($data) but that takes out only the last |)
Please suggest...
$data =~ s/(^[.|]*)|([.|]*$)//g;
That said, I don't assume that this will speed up your script significantly.
Another way: $data =~ s/^\.\|(.*)\|/$1/
But as Rene said, your speed bottleneck is probably somewhere else in your script.
I have a file in a different folder I want to rename in perl, I was looking at a solution earlier that showed something like this:
#rename
for (<C:\\backup\\backup.rar>) {
my $file = $_;
my $new = $file . 'backup' . $ts . '.rar';
rename $file, $new or die "Error, can not rename $file as $new: $!";
}
however backup.rar is in a different folder, I did try putting "C:\backup\backup.rar" in the <> above, however I got the same error.
C:\Program Files\WinRAR>perl backup.pl
String found where operator expected at backup.pl line 35, near "$_ 'backup'"
(Missing operator before 'backup'?)
syntax error at backup.pl line 35, near "$_ 'backup'"
Execution of backup.pl aborted due to compilation errors.
I was using
# Get time
my #test = POSIX::strftime("%m-%d-%Y--%H-%M-%S\n", localtime);
print #test;
To get the current time, however I couldn't seem to get it to rename correctly.
What can I do to fix this? Please note I am doing this on a windows box.
Pay attention to the actual error message. Look at the line:
my $new = $_ 'backup'. #test .'.rar';
If you want to interpolate the contents of $_ and the array #test into a string like that, you need to use:
my $new = "${_}backup#test.rar";
but I have a hard time making sense of that.
Now, strftime returns a scalar. Why not use:
my $ts = POSIX::strftime("%m-%d-%Y--%H-%M-%S", localtime);
my $new = sprintf '%s%s%s.rar', $_, backup => $ts;
Incidentally, you might end up making your life much simpler if you put the time stamp first and formatted it as YYYYMMDD-HHMMSS so that there is no confusion about to which date 04-05-2010 refers.
The line
my $new = $_ 'backup'. #test .'.rar';
probably should read
my $new = $file . 'backup' . #test . '.rar';
(You were missing a concatenation operator, and it is clearer to use the named variable from the line before than reusing $_ there...)
I think you missed the string concat symbol . (the period) :
my $new = $_ 'backup'. #test .'.rar';
should be
my $new = $_ . 'backup' . #test . '.rar';
A slight side issue but you don't need
for (<C:\\backup\\backup.rar>) {
my $file = $_;
.....
}
The < > construct would be useful if you were expanding a wildcard but you are not.
Be thoughtful of future readers of this code (you in a year!) and write
my $file = 'C:\backup\backup.rar' ;
Note the single quotes which doen't expand backslashes.