powershell filter csv - powershell

Looking for help on best practice as I am Powershell newbie,
I have a csv file and I would like to filter out every row in the csv except for the rows containing "Not Installed"
I would then like to filter those results against a separate csv files housing a list of computers and also exclude any rows containing a match.
Any help or starting points would be appreciated !

For the first part, you can use Select-String -Notmatch ... e.g.:
$file = "data.csv"
$csv = Get-Content $file
$csv | Select-String -Notmatch 'Not Installed' | Out-File $file -Enc ascii
If your CSV file happens to be Unicode, then remove the "-Enc ascii" part since the default for Out-File is Unicode. As for the second part of your question, you're going to need to be bit more specific. However, you should look at the help on Import-Csv and Export-Csv:
man Import-Csv -full
man Export-Csv -full

get-help *csv*
get-help import-csv -examples
example csv file called test.txt
header1,header2,header3
installed,2,3
not installed,2,3
Import the file into a powershell variable and filter it.
$p = Import-CSV .\test.txt
$p | Where { $_.header1 -eq 'installed'}

Came across this question when trying to work out how to do the same thing myself.
What I came up with as a solution was as follows:
$Filtered = $Table|Select-String -Pattern $ExcludeStrings -NotMatch -SimpleMatch
$FilteredArray = #()
$Filtered.Line | foreach {$split = $_ -split ';';
$tidied = $split -replace '#|{|}|^ ';
$obj = New-Object System.Object
$tidied|foreach {$obj|Add-Member -MemberType NoteProperty -Name ($_.split('=')[0]) -Value ($_.split('=')[1])
}
$FilteredArray += $obj
}
The first line is just showing me filtering the array that I got via Import-CSV to get only the results I wanted.
Unfortunately, this leaves me with a MatchInfo array, with each array being in the following format:
#{Column1=Value1; Column2=Value2; Column3=Value3;}
This does not export back out to CSV nicely at all, as is.
So, I opted to create a new output array ($FilteredArray).
I then piped the Line property of the matchinfo array from Select-String into a foreach loop, and for each line, I split it on the semi-colon, then I remove the #{} chars and any leading spaces.
I then create a new System.Object, then pipe the tidied output into another foreach loop, creating new NotePropertys in the obj, naming them from the current line in the tidied output up to the = character, and the value being the remaining text after the = character.
What I end up with is:
PS C:\Users\Graham> $obj
Column1 : Value1
Column2 : Value2
Column3 : Value3
And the filtered array looks like:
PS C:\Users\Graham> $FilteredArray[0..3]|Format-table
Column1 Column2 Column2
------- ------- -------
Value1 Value2 Value3
Value1 Value2 Value3
Value1 Value2 Value3

Related

Export the matched output to a CSV as Column 1 and Column 2 using Powershell

I have below code to match the pattern and save it in CSV file. I need to save regex1 and regex2 as col1 and col2 in csv instead of saving all in 1st col.
$inputfile = ( Get-Content D:\Users\naham1224\Desktop\jil.txt )
$FilePath = "$env:USERPROFILE\Desktop\jil2.csv"
$regex1 = "(insert_job: [A-Za-z]*_*\S*)"
$regex2 = "(machine: [A-Z]*\S*)"
$inputfile |
Select-String -Pattern $regex2,$regex1 -AllMatches |
ForEach-Object {$_.matches.groups[1].value} |
Add-Content $FilePath`
Input file contains : input.txt
/* ----------------- AUTOSYS_DBMAINT ----------------- */
insert_job: AUTOSYS_DBMAINT job_type: CMD
command: %AUTOSYS%\bin\DBMaint.bat
machine: PWISTASASYS01
owner: svc.autosys#cbs
permission:
date_conditions: 1
days_of_week: su,mo,tu,we,th,fr,sa
start_times: "03:30"
description: "Runs DBmaint process on AE Database - if fails - MTS - will run next scheduled time"
std_out_file: ">$$LOGS\dbmaint.txt"
std_err_file: ">$$LOGS\dbmaint.txt"
alarm_if_fail: 0
alarm_if_terminated: 0
send_notification: 0
notification_msg: "Check DBMaint output in autouser.PD1\\out directory"
notification_emailaddress: jnatal#cbs.com
/* ----------------- TEST_ENV ----------------- */
insert_job: TEST_ENV job_type: CMD
command: set
machine: PWISTASASYS01
owner: svc.autosys#cbs
permission:
date_conditions: 1
days_of_week: su,mo,tu,we,th,fr,sa
start_times: "03:30"
description: "output env"
std_out_file: ">C:\Users\svc.autosys\Documents\env.txt"
std_err_file: ">C:\Users\svc.autosys\Documents\env.txt"
alarm_if_fail: 1
alarm_if_terminated: 1
Current output :
Current output
Expected output :
Expected output
I am trying various ways to do so but no luck. any suggestions and help is greatly appreciated.
Here is how I would do this:
$inputPath = 'input.txt'
$outputPath = 'output.csv'
# RegEx patterns to extract data.
$patterns = #(
'(insert_job): ([A-Za-z]*_*\S*)'
'(machine): ([A-Z]*\S*)'
)
# Create an ordered Hashtable to collect columns for one row.
$row = [ordered] #{}
# Loop over all occurences of the patterns in input file
Select-String -Path $inputPath -Pattern $patterns -AllMatches | ForEach-Object {
# Extract key and value from current match
$key = $_.matches.Groups[ 1 ].Value
$value = $_.matches.Value
# Save one column of current row.
$row[ $key ] = $value
# If we have all columns of current row, output it as PSCustomObject.
if( $row.Count -eq $patterns.Count ) {
# Convert hashtable to PSCustomObject and output (implicitly)
[PSCustomObject] $row
# Clear Hashtable in preparation for next row.
$row.Clear()
}
} | Export-Csv $outputPath -NoTypeInformation
Output CSV:
"insert_job","machine"
"insert_job: AUTOSYS_DBMAINT","machine: PWISTASASYS01"
"insert_job: TEST_ENV","machine: PWISTASASYS01"
Remarks:
Using Select-String with parameter -Path we don't have to read the input file beforehand.
An ordered Hashtable (a dictionary) is used to collect all columns, until we have an entire row to output. This is the crucial step to produce multiple columns instead of outputting all data in a single column.
Converting the Hashtable to a PSCustomObject is necessary because Export-Csv expects objects, not dictionaries.
While the CSV looks like your "expected output" and you possibly have good reason to expect it like that, in a CSV file the values normally shouldn't repeat the column names. To remove the column names from the values, simply replace $value = $_.matches.Value by $_.matches.Groups[ 2 ].Value, which results in an output like this:
"insert_job","machine"
"AUTOSYS_DBMAINT","PWISTASASYS01"
"TEST_ENV","PWISTASASYS01"
As for what you have tried:
Add-Content writes only plain text files from string input. While you could use it to create CSV files, you would have to add separators and escape strings all by yourself, which is easy to get wrong and more hassle than necessary. Export-CSV otoh takes objects as inputs and cares about all of the CSV format details automatically.
As zett42 mentioned Add-Content is not the best fit for this. Since you are looking for multiple values separated by commas Export-Csv is something you can use. Export-Csv will take objects from the pipeline, convert them to lines of comma-separated properties, add a header line, and save to file
I took a little bit of a different approach here with my solution. I've combined the different regex patterns into one which will give us one match that contains both the job and machine names.
$outputPath = "$PSScriptRoot\output.csv"
# one regex to match both job and machine in separate matching groups
$regex = '(?s)insert_job: (\w+).+?machine: (\w+)'
# Filter for input files
$inputfiles = Get-ChildItem -Path $PSScriptRoot -Filter input*.txt
# Loop through each file
$inputfiles |
ForEach-Object {
$path = $_.FullName
Get-Content -Raw -Path $path | Select-String -Pattern $regex -AllMatches |
ForEach-Object {
# Loop through each match found in the file.
# Should be 2, one for AUTOSYS_DBMAINT and another for TEST_ENV
$_.Matches | ForEach-Object {
# Create objects with the values we want that we can output to csv file
[PSCustomObject]#{
# remove next line if not needed in output
InputFile = $path
Job = $_.Groups[1].Value # 1st matching group contains job name
Machine = $_.Groups[2].Value # 2nd matching group contains machine name
}
}
}
} | Export-Csv $outputPath # Pipe our objects to Export-Csv
Contents of output.csv
"InputFile","Job","Machine"
"C:\temp\powershell\input1.txt","AUTOSYS_DBMAINT","PWISTASASYS01"
"C:\temp\powershell\input1.txt","TEST_ENV","PWISTATEST2"
"C:\temp\powershell\input2.txt","AUTOSYS_DBMAINT","PWISTASAPROD1"
"C:\temp\powershell\input2.txt","TEST_ENV","PWISTATTEST1"

Powershell - Finding the output of get-contents and searching for all occurrences in another file using wild cards

I'm trying to get the output of two separate files although I'm stuck on the wild card or contains select-string search from file A (Names) in file B (name-rank).
The contents of file A is:
adam
george
william
assa
kate
mark
The contents of file B is:
12-march-2020,Mark-1
12-march-2020,Mark-2
12-march-2020,Mark-3
12-march-2020,william-4
12-march-2020,william-2
12-march-2020,william-7
12-march-2020,kate-54
12-march-2020,kate-12
12-march-2020,kate-44
And I need to match on every occurrence of the names after the '-' so my ordered output should look like this which is a combination of both files as the output:
mark
Mark-1
Mark-2
Mark-3
william
william-2
william-4
william-7
Kate
kate-12
kate-44
kate-54
So far I only have the following and I'd be grateful for any pointers or assistance please.
import-csv (c:\temp\names.csv) |
select-string -simplematch (import-csv c:\temp\names-rank.csv -header "Date", "RankedName" | select RankedName) |
set-content c:\temp\names-and-ranks.csv
I imagine the select-string isn't going to be enough and I need to write a loop instead.
The data you give in the example does not give you much to work with, and the desired output is not that intuitive, most of the time with Powershell you would like to combine the data in to a much richer output at the end.
But anyway, with what is given here and what you want, the code bellow will get what you need, I have left comments in the code for you
$pathDir='C:\Users\myUser\Downloads\trash'
$names="$pathDir\names.csv"
$namesRank="$pathDir\names-rank.csv"
$nameImport = Import-Csv -Path $names -Header names
$nameRankImport= Import-Csv -Path $namesRank -Header date,rankName
#create an empty array to collect the result
$list=#()
foreach($name in $nameImport){
#get all the match names
$match=$nameRankImport.RankName -like "$($name.names)*"
#add the name from the First list
$list+=($name.names)
#if there are any matches, add them too
if($match){
$list+=$match
}
}
#Because its a one column string, Export-CSV will now show us what we want
$list | Set-Content -Path "$pathDir\names-and-ranks.csv" -Force
For this I would use a combination of Group-Object and Where-Object to first group all "RankedName" items by the name before the dash, then filter on those names to be part of the names we got from the 'names.csv' file and output the properties you need.
# read the names from the file as string array
$names = Get-Content -Path 'c:\temp\names.csv' # just a list of names, so really not a CSV
# import the CSV file and loop through
Import-Csv -Path 'c:\temp\names-rank.csv' -Header "Date", "RankedName" |
Group-Object { ($_.RankedName -split '-')[0] } | # group on the name before the dash in the 'RankedName' property
Where-Object { $_.Name -in $names } | # use only the groups that have a name that can be found in the $names array
ForEach-Object {
$_.Name # output the group name (which is one of the $names)
$_.Group.RankedName -join [environment]::NewLine # output the group's 'RankedName' property joined with a newline
} |
Set-Content -Path 'c:\temp\names-and-ranks.csv'
Output:
Mark
Mark-1
Mark-2
Mark-3
william
william-4
william-2
william-7
kate
kate-54
kate-12
kate-44

Powershell Select Expanded Property and Custom Expression

I want to get the content of the first value of a specific column using Import-Csv output as a table alongside the name of the file.
I can do:
$File = '\\webserver\Data_20190626.csv'
Import-Csv -Path $File -Delimiter ',' | select 'Effective Date' -First 1
Which gives me my expected output:
Effective Date
--------------
25-May-2019
What I want to see is:
Effective Date FileName
-------------- ---------
25-May-2019 Data_20190626.csv
I have tried this:
$File = '\\webserver\Data_20190626.csv'
Import-Csv -Path $File -Delimiter ',' | select 'Effective Date', #{N='FileName';E={$_.Name}} -First 1
Which resulted in:
Effective Date FileName
-------------- ---------
25-May-2019
How should I proceed?
You can achieve this with Select-Object's property hash table and splitting the file path by \ with the .Split() method. The [-1] indicates the last item in the split result.
$File = '\\webserver\Data_20190626.csv'
Import-Csv -Path $File |
Select-Object 'Effective Date',#{n='FileName';e={$File.Split('\')[-1]}} -First 1
I'm surprised there's no PSPath or anything.
import-csv $file -delimiter ',' |
select 'Effective Date', #{n='Filename';e={split-path -leaf $file}} -first 1
this is a slightly different way to do things. [grin] the #region >>> create a file to work with section can be ignored - it is there to provide the data file you didn't provide.
it gets the EffectiveDate info by using the way that PoSh can dot-address one property of an entire collection and then grabbing the 1st item from the resulting array.
it also avoids the often confusing side effects of allowing spaces or other special chacters in property names by using EffectiveDate instead of Effective Date. if you need to keep using the poorly thot out embedded space, then change that string as needed.
$File = "$env:TEMP\Data_20190626.csv"
#region >>> create a file to work with
#'
EffectiveDate
2019-06-25
2006-06-06
2005-05-05
'# |
ConvertFrom-Csv |
Export-Csv -LiteralPath $File -NoTypeInformation
#endregion >>> create a file to work with
$Results = [PSCustomObject]#{
EffectiveDate = (Import-Csv -LiteralPath $File).EffectiveDate[0]
FileName = $File
}
$Results
output ...
EffectiveDate FileName
------------- --------
2019-06-25 C:\Temp\Data_20190626.csv

Powershell Searching within a text file and exporting horizontally to CSV

I have a big file which content somme blocks of text data like this.
[SERVER1]
LIBELLE=DATA, SOMME DATA
VARIABLES=A,B,C,D,E
PHYSICAL NAME=E:\SOMME\PATH\FILE.INI
ARTICLE SIZE=50
MACHINE=SOME SERVER
PARAMETER =
OPTION =
[SERVER2]
LIBELLE=DATA2, SOMME DATA2
VARIABLES=A,B,C,D,E
PHYSICAL NAME=Z:\SOMME\PATH\FILE2.INI
ARTICLE SIZE=150
MACHINE=SOME SERVER XY
PARAMETER =
OPTION 1 = VOID
OPTION 2 =
OPTION 3 =
OPTION 4 =
OPTION 5 =
What i would like to do is retrieving every block [SERVERX] and put it into a CSV file in this format (horizontally)
ColumnA__ | ColumnB (LIBELLE)___ | ColumnC (VARIABLES) | ColumnD ect...
[SERVER1] | DATA1, SOMME DATA1 | A,B,C,D,E___________ | Ect...
[SERVER2] | DATA2, SOMME DATA2 | A,B,C,D,E___________ | Ect...
I've tried this, the output work as i want but it need to be automated and exported to scv, which doesn't work for me.
$mydata = Get-Content my_file.txt
write-Host $mydata[0] $mydata[1] $mydata[2] $mydata[3] $mydata[4] $mydata[5] | Export-Csv -Path $rep\results.csv -Force -UseCulture -NoTypeInformation
Tried also somthing with select-string but i dont know if this is the right way to do my job..
select-String -path $my_file -Pattern '\[*\]', 'IDENTIFIANTS=','LIBELLE=','VARIABLES=' | Select-Object -Property LineNumber, Line | Export-Csv -Path $rep\results.csv -Force -UseCulture -NoTypeInformation
Thanks for your advices.
So, I would read the whole file in as a multi-line string using the -Raw switch for Get-Content. Then split the file up based on the [ character to denote records. The get the properties from the ConvertFrom-StringData cmdlet (have to prepend "SERVER=" to each record), and make an object from it. Then we find out what all properties any given record can have, make sure to add them all to the first record if it doesn't have it (this is done because when you export to CSV it bases the columns off of the first entries' property list). Then you can export a CSV.
$Data = (Get-Content my_file.txt -Raw) -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}
$Records = $Data -replace '\\','\\'|%{$Record="SERVER="+$_.trim()|ConvertFrom-StringData;New-Object PSObject -Prop $Record}
$Props = $Records|%{$_.psobject.properties.name}|select -Unique
$Props | Where{$_ -notin $Records[0].PSObject.Properties.Name}|%{Add-Member -InputObject $Records[0] -NotepropertyName $_ -NotepropertyValue $Null}
$Records|Export-CSV .\my_file.csv -notype
Edit: For those of you out there running PowerShell 2.0 (3 versions out of date at this point in time), you can't use the -Raw parameter. Here's the alternative:
$Data = (Get-Content my_file.txt) -Join "`r`n" -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}
Alternative: Thanks #Matt for the suggestion, it is always good to have a different point of view on these things. As Matt suggested, you can use Out-String to combine the array of strings that Get-Content generates, and end up with a single multi-line string. Here's the usage!
$Data = (GC my_file.txt | Out-String) -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}

PowerShell Import-Csv Issue - Why is my output being treated as a single column and not a CSV?

So I have a CSV file which I need to manipulate a bit, select the data I need and export to another CSV file.
The code I have is:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
Import-Csv -Header #("a","b","c","d") -Path $rawCSV |
select -Skip 7 |
Where-Object { $_.b.length -gt 1 } |
ft b,a,c,d |
Out-File $outputCSV
So this code uses the Import-Csv command to allow me to select just the columns I need, add some headers in the order I want and then I am simply putting the output in to a CSV file called $outputCSV. The contents of this output file look something like this:
b a c d
- - - -
john smith 29 England
mary poopins 79 Walton
I am not sure what the delimiter is in this output and rather than these columns being treated as individuals, they are treated as just one column. I have gone on further to replace all the spaces with a comma using the code:
$b = foreach ($line in $a)
{
$fields = $line -split '`n'
foreach ($field in $fields)
{
$field -replace " +",","
}
}
Which produces a file that looks like this:
b,a,c,d
john,smith,29,England
mary,poppins,79,Walton
But these are all still treated as one column instead of four separate columns as I need.
* UPDATE *
Using the answer given by #, I now get a file looking like this:
Don't use ft to reorder your columns - it's intended to format output for the screen, not really suitable for CSV.
"Manual" solution:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Write your headers to the output file
"b","a","c","d" -join ',' | Out-File $outputCSV -Force
$ReorderedData = foreach($Row in $Data){
# Reorder the columns in each row
'{0},{1},{2},{3}' -f $Row.b , $Row.a , $Row.c, $Row.d
}
# Write the reordered rows to the output file
$ReorderedData | Out-File $outputCSV -Append
Using Export-Csv:
As of PowerShell 3.0, you could also push the rows into a [pscustomobject] and pipe that to Export-Csv (pscustomobject preserves the order in which you supply the properties):
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Take the columns you're interested in, put them into new custom objects and export to CSV
$Data | ForEach-Object {
[pscustomobject]#{ "b" = $_.b; "a" = $_.a; "c" = $_.c; "d" = $_.d }
} | Export-Csv -NoTypeInformation $outputCSV
Export-Csv will take care of enclosing strings in quotes to escape ',' properly (one thing less for you to worry about)
First of all, what your raw CSV file looks like? If it's already like this
john,smith,29,England
mary,poppins,79,Walton
then import-csv will give you an array of objects which you can easily manipulate (and objects are the main reason to use PowerShell ;). For example, to check what you have after import:
$r = Import-Csv -Path $rawCSV -Header #("b","a","c","d")
$r.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array
$r[0] | get-member
TypeName: System.Management.Automation.PSCustomObject
Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
a NoteProperty System.String a=smith
b NoteProperty System.String b=john
c NoteProperty System.String c=29
d NoteProperty System.String d=England
For now you have array of objects with properties named "a","b","c","d". To manipulate objects you have select-object cmdlet:
$r | Select-Object a,b,c,d
a b c d
- - - -
smith john 29 England
poppins mary 79 Walton
And after all use export-csv to set the output file:
$r | where { $_.b.length -gt 1 } |
select a,b,c,d |
Export-Csv -NoTypeInformation -Encoding utf8 -path $outputCSV
I could think of two possible reasons why your data teated as one column:
consuming application expect different encoding and can't find
delimiters
delimiters are not commas but something else