I have a log file that has a lot of fields listed in it. I want to extract the fields out of the file, but I don't want to search through the file line by line.
I have my pattern:
$pattern="Hostname \(Alias\):(.+)\(.+Service: (.+)"
This will give me the two values that I need. I know that if I have a string, and I'm looking for one match I can use the $matches array to find the fields. In other words, If I'm looking at a single line in the file using the string variable $line, I can extract the fields using this code.
if($line -matches $pattern){
$var1=$matches[1]
$var2=$matches[2]
}
But how can I get these values without searching line by line? I want to pass the whole file as a single string, and add the values that I am extracting to two different arrays.
I'm looking for something like
while($filetext -match $pattern){
$array1+=$matches[1]
$array2+=$matches[2]
}
But this code puts me in an infinite loop if there is even one match. So is there a nextMatch function I can use?
PowerShell 2.0 addressed this limitation by adding the -AllMatches parameter to the Select-String cmdlet e.g.:
$filetext | Select-String $pattern -AllMatches |
Foreach {$_.Matches | Foreach {$_.Groups[1] | Foreach {$_.Value}}}
Related
I have a csv file, with no headlines, that looks like this:
"88212526";"Starter";"PowerMax";"4543";"5713852369748";"146,79";"EUR";"6"
"88212527";"Starter";"PowerMax";"4543";"5713852369755";"66,88";"EUR";"20"
"88212530";"Starter";"PowerMax";"4543";"5713852369786";"143,27";"EUR";"0"
"88212532";"Starter";"PowerMax";"4543";"5713852369809";"80,98";"EUR";"6"
"88212536";"Starter";"PowerMax";"4543";"5713852369847";"";"EUR";"0"
"88212542";"Starter";"PowerMax";"4543";"5713852369908";"77,16";"EUR";"9"
"88212543";"Starter";"PowerMax";"4543";"5713852369915";"77,46";"EUR";"52"
I need a script in PowerShell that deletes the entire row if column 6 is empty.
I have tried this
Foreach ($line in Get-Content .\POWERMAX_DK_1.csv) {
$linearray = $line.split(";")
if($linearray[6] -ne "") {
Add-Content .\myTempFile.csv $line
}
}
But it don't work. The line with empty column is not removed.
Please help
/Kim
Your immediate problem is twofold:
As Mauro Takeda's answer points out, to access the 6th element, you must use index 5, given that array indices are 0-based.
Since you're reading your CSV file as plain text, the field you're looking for has verbatim content "", i.e. including the double quotes, so you'd have to use -ne '""' instead of -ne "" ($linearray[5])
However, it's worth changing your approach:
Use Import-Csv to import your CSV file, which in your case requires manually supplying headers (column names) with the -Header parameter.
This outputs objects whose properties are named for the columns, and whose property values have the syntactic " delimiters removed.
These properties can then be used to robustly filter the input with the Where-Object cmdlet.
In order to convert the results back to a CSV file, use a - single -call to Export-Csv, as shown below (see next point).
Using Add-Content in a loop body is ill-advised for performance reasons, because the file has to be opened and closed in every iteration; instead, pipe to a single call of a file-writing cmdlet - see this answer for background information.
Therefore:
# Note: The assumption is that there are 8 columns, as shown in the sample data.
# Adjust as needed.
Import-Csv .\POWERMAX_DK_1.csv -Delimiter ';' -Header (1..8) |
Where-Object 6 -ne '' |
Export-Csv -NoTypeInformation \myTempFile.csv
Character-encoding caveat: In Windows PowerShell, Export-Csv uses ASCII(!) by default; PowerShell (Core) 7+ commendably uses BOM-less UTF-8. Use the -Encoding parameter as needed.
If you need check column 6, you have to use $linearray[5], because arrays starts counting on zero ($linearray[0] should be the first element)
I have a Commands.csv file like:
| Command |
| -----------------------------------------------|
|(Get-FileHash C:\Users\UserA\Desktop\File1).Hash|
|(Get-FileHash C:\Users\UserA\Desktop\File2).Hash|
|(Get-FileHash C:\Users\UserA\Desktop\File3).Hash|
Header name is "Command"
My idea is to:
Use ForEach ($line in Get-Content C:\Users\UserA\Desktop\Commands.csv ) {echo $line}
Execute $line one by one via powershell.exe, then output a result to a new .csv file - "result.csv"
Can you give me some directions and suggestions to implement this idea? Thanks!
Important:
Only use the technique below with input files you either fully control or implicitly trust to not contain malicious commands.
To execute arbitrary PowerShell statements stored in strings, you can use Invoke-Expression, but note that it should typically be avoided, as there are usually better alternatives - see this answer.
There are advanced techniques that let you analyze the statements before executing them and/or let you use a separate runspace with a restrictive language mode that limits what kinds of statements are allowed to execute, but that is beyond the scope of this answer.
Given that your input file is a .csv file with a Commands column, import it with Import-Csv and access the .Commands property on the resulting objects.
Use Get-Content only if your input file is a plain-text file without a header row, in which case the extension should really be .txt. (If it has a header row but there's only one column, you could get away with Get-Content Commands.csv | Select-Object -Skip 1 | ...). If that is the case, use $_ instead of $_.Commands below.
To also use the CSV format for the output file, all commands must produce objects of the same type or at least with the same set of properties. The sample commands in your question output strings (the value of the .Hash property), which cannot meaningfully be passed to Export-Csv directly, so a [pscustomobject] wrapper with a Result property is used, which will result in a CSV file with a single column named Result.
Import-Csv Commands.csv |
ForEach-Object {
[pscustomobject] #{
# !! SEE CAVEAT AT THE TOP.
Result = Invoke-Expression $_.Commands
}
} |
Export-Csv -NoTypeInformation Results.csv
I wrote a PowerShell script to compare two text files. In file1 the data is organised. But in file2 the data is not organised. I usually organize data manually. But now the data is increased. I need to automate organising using PowerShell.
PowerShell has to read data between two special characters. For example: <****# is my data. It has to read **** only. this pattern repeats 'n' number of times.
Use a regular expression <(.*?)# to match the relevant substring. .*? matches all characters between < and the next occurrence of # (non-greedy/shortest match). The parentheses put the match in a capturing group, so it can be referenced later.
Select-String -Path 'C:\path\to\file2.txt' -Pattern '<(.*?)#' -AllMatches |
Select-Object -Expand Matches |
ForEach-Object { $_.Groups[1].Value }
$_.Groups[1] refers to the first capturing group in a match.
Question
How do I reformat an unknown CSV column name according to a formula or subroutine (e.g. rename column " Arbitrary Column Name " to "Arbitrary Column Name" by running a trim or regex or something) while maintaining data?
Goal
I'm trying to more or less sanitize columns (the names) in a hand-produced (or at least hand-edited) csv file that needs to be processed by an existing PowerShell script. In this specific case, the columns have spaces that would be removed by a call to [String]::Trim(), or which could be ignored with an appropriate regex, but I can't figure a way to call or use those techniques when importing or processing a CSV.
Short Background
Most files and columns have historically been entered into the CSV properly, but recently a few columns were being dropped during processing; I determined it was because the files contained a space (e.g., Select-Object was being told to get "RFC", but Import-CSV retrieved "RFC ", so no matchy-matchy). Telling the customer to enter it correctly by hand (though preferred and much simpler) is not an option in this case.
Options considered
I could manually process the text of the file, but that is a messy and error prone way to re-invent the wheel. I wonder if there's a syntax with Select-Object that would allow a softer match for column names, but I can't find that info.
The closest I have come conceptually is using a calculated property in the call to Select-Object to rename the column, but I can only find ways to rename a known column to another known column. So, this would require enumerating the columns and matching them exactly (preferred) or a softer match (like comparing after trimming or matching via regex as a fallback) with expected column names, then creating a collection of name mappings to use in constructing calculated properties from that information to select into a new object.
That seems like it would work, but more it's work than I'd prefer, and I can't help but hope that there's a simpler way I haven't been able to find via Google. Maybe I should try Bing?
Sample File
Let's say you have a file.csv like this:
" RFC "
"1"
"2"
"3"
Code
Now try to run the following:
$CSV = Get-Content file.csv -First 2 | ConvertFrom-Csv
$FixedHeaders = $CSV.PSObject.Properties.Name.Trim(' ')
Import-Csv file.csv -Header $FixedHeaders |
Select-Object -Skip 1 -Property RFC
Output
You will get this output:
RFC
---
1
2
3
Explanation
First we use Get-Content with parameter -First 2 to get the first two lines. Piping to ConvertFrom-Csv will allow us to access the headers with PSObject.Properties.Name. Use Import-Csv with the -Header parameter to use the trimmed headers. Pipe to Select-Object and use -Skip 1 to skip the original headers.
I'm not sure about comparisons in terms of efficiency, but I think this is a little more hardened, and imports the CSV only once. You might be able to use #lahell's approach and Get-Content -raw, but this was done and it works, so I'm gonna leave it to the community to determine which is better...
#import the CSV
$rawCSV = Import-Csv $Path
#get actual header names and map to their reformatted versions
$CSVColumns = #{}
$rawCSV |
Get-Member |
Where-Object {$_.MemberType -eq "NoteProperty"} |
Select-Object -ExpandProperty Name |
Foreach-Object {
#add a mapping to the original from a trimmed and whitespace-reduced version of the original
$CSVColumns.Add(($_.Trim() -replace '(\s)\s+', '$1'), "$_")
}
#Create the array of names and calculated properties to pass to Select-Object
$SelectColumns = #()
$CSVColumns.GetEnumerator() |
Foreach-Object {
$SelectColumns += {
if ($CSVColumns.values -contains $_.key) {$_.key}
else { #{Name = $_.key; Expression = $CSVColumns[$_.key]} }
}
}
$FormattedCSV = $rawCSV |
Select-Object $SelectColumns
This was hand-copied to a computer where I don't have the rights to run it, so there might be an error - I tried to copy it correctly
You can use gocsv https://github.com/DataFoxCo/gocsv to see the headers of the csv, you can then rename the headers, behead the file, swap columns, join, merge, any number of transformations you want
I am writing a script which will basically do the following:
Read from a text file some arguments:
DriveLetter ThreeLetterCode ServerName VolumeLetter Integer
Eg. W MSS SERVER01 C 1
These values happen to form a folder destination W:\MSS\, and a filename which works in the following naming convention:
SERVERNAME_VOLUMELETTER_VOL-b00X-iYYY.spi - Where The X is the Integer above
The value Y I need to work out later, as this happens to be the value of the incremental image (backups) and I need to work out the latest incremental.
So at the moment --> Count lines in file, and loop for this many lines.
$lines = Get-Content -Path PostBackupCheck-Textfile.txt | Measure-Object -Line
for ($i=0; $i -le $lines.Lines; $i++)
Within this loop I need to do a Get-Content to read off the line I am currently looking at i.e. line 0, line 1, line 2, as there will be multiple lines in the format I wrote at the beginning and split the line into an array, whereby each part of the file, as seen above naming convention, is in a[0], a[1], a[2]. etc
The reason for this is because, I need to then sort the folder that contains these, find the latest file, by date, and take the _iXXX.spi part and place this into the array value a[X] so I then have a complete filename to mount. This value will replace iYYY.spi
It's a little complex because I also have to make sure when I do a Get-ChildItem with -Include before I sort it all by date, I am only including the filename that matches the arguments fed to it from the text file :
So,
SERVER01_C_VOL-b001-iYYY.spi and not anything else.
i.e. not SERVER01_D_VOL-b001-iYYY.spi
Then take the iYYY value from the sort on the Get-ChildItem -Include and place that into the appropriate array item.
I've literally no idea where to start, so any ideas are appreciated!
Hopefully I've explained in enough detail. I have also placed the code on Pastebin: http://pastebin.com/vtFifTW6
This doesn't need to be that complex. You can start by operating over lines in your file with a simple pipeline:
Get-Content PostBackupCheck-Textfile.txt |
Foreach-Object {
$drive, $folder, $server, $volume, [int]$i = -split $_
...
}
The line inside the loop splits the current input line at spaces and assigns appropriate variables. This saves you the trouble of handling an array there. Everything that follows needs to be in said loop as well.
You can then construct the file name pattern:
$filename = "$server_$drive_VOL-b$($i.ToString('000'))-i*.spi"
which you can use to find all fitting files and sort them by date:
$lastFile = Get-ChildItem $filename | sort LastWriteTime | select -last 1