I am importing data from various csv files, usually with 4 or 5 fields.
e.g. one might look like:
id, name, surname, age
1,tom,smith,32
2,fred,bloggs,50
I have managed to grab the header row titles into and array that looks like:
id, name, surname, age
the first data row looks like:
#{ID=1; name=tom; surname=smith; age=32}
say I assign it to $myRow
what I want to be able to do is access the ID, name etc field in $myRow by index, 0, 1, 2 etc, not by the property name.
Is this possible?
Thanks
You can do something like this, but it may be slow for large sets of rows and/or properties:
$users =
import-csv myusers.csv |
foreach {
$i=0
foreach ($property in $_.psobject.properties.name)
{
$_ | Add-Member -MemberType AliasProperty -Name $i -Value $property -passthru
$i++
}
}
That just adds an Alias property for each property name in the object
When I wanted to do something similar, I went about it differently.
I used Import-Csv to get the contents into a table. Then I stepped through the table, row by row, and used an inner loop to retrieve the field values, one by one into variables with the same name as the column name.
This created a context where I could apply the values to variables embedded in some kind of template. Here is an edited version of the code.
foreach ($item in $list) {
$item | Get-Member -membertype properties | foreach {
Set-variable -name $_.name -value $item.$($_.name)
}
Invoke-expression($template) >> Outputfile.txt
}
I'm writing the expanded templates to an output file, but you get the idea. This end up working more or less the way mail merge applies a mailing list to a form letter.
I wouldn't use this approach for more than a few hundred rows and a dozen columns. It gets slow.
Explanation:
The inner loop needs more explanation. $list is a table that contains
the imported image of a csv file. $item is one row from this table.
Get-Member gets each field (called a property) from that row. Each
field has a name and a value. $_.name delivers the name of the
current field. $item.($_.name) delivers the value. Set-Variable
creates a variable. It's very inefficient to create the same
variables over and over again for each row in the table, but I don't
care.
This snippet was clipped from a larger snippet that imports a list and a template, produces an expansion of the template for each item in the list, and outputs the series of expansions into a text file. I didn't include the whole snippet because it's too far afield from the question that was asked.
You can actually index your array with ($MyRow[1]).age in order to get the age of the first row.
Related
I have two array's, one contains multiple columns from a CSV file read in, and the other just contains server names, both type string. For this comparison, I plan on only using the name column from the CSV file. I don't want to use -compare because I want to still be able to use all CSV columns with the results. Here is an example of data from each array.
csvFile.Name:
linu40944
windo2094
windo4556
compareFile:
linu40944
windo2094
linu24455
As you can see, they contain similar server names, except $csvFile.Name contains 25,000+ records, and $compareFile contains only 3,500.
I've tried:
foreach ($server in $compareFile) {
if ($csvFile.Name -like $server) {
$count++
}
}
Every time I run this, it takes forever to run, and results in $count having a value in the millions when it should be roughly 3,000. I've tried different variations of -match, -eq, etc. where -like is. Also note that my end goal is to do something else where $count is, but for now I'm just trying to make sure it is outputting as much as it should, which it is not.
Am I doing something wrong here? Am I using the wrong formatting?
One possible thought given the size of your data.
Create a hashtable (dictionary) for every name in the first/larger file. Name is the Key. Value is 0 for each.
For each name in your second/smaller/compare file, add 1 to the value in your hashtable IF it exists. If it does not exist, what is your plan???
Afterwards, you can dump all keys and values and see which ones are 0, 1, or >1 which may or may not be of value to you.
If you need help with this code, I may be able to edit my answer. Since you are new, to StackOverflow, perhaps you want to try this first yourself.
Build custom objects from $compareFile (so that you can compare the same property), then use Compare-Object with the parameter -PassThru for the comparison. Discriminate the results using the SideIndicator.
$ref = $compareFile | ForEach-Object {
New-Object -Type PSObject -Property #{
'Name' = $_
}
}
Compare-Object $csvFile $ref -Property Name -PassThru | Where-Object {
$_.SideIndicator -eq '<='
} | Select-Object -Property * -Exclude SideIndicator
The trailing Select-Object removes the additional property SideIndicator that Compare-Object adds to the result.
I have a lot of csv files with values arranged like so:
X1,Y1
X2,Y2
...,...
Xn,Yn
I find it very tedious processing these with excel, so I want to setup a batch script to process these files such that they appear like this:
#where N is a specified value like 65536
X1,N-Y1,1
X2,N-Y2,2
...,...,...
Xn,N-Yn,n
I have only recently started using powershell for image processing (really simple scripts) and file name appending, so I am not certain how to go about this. A lot of the scripts I have encountered looking to answer this question use csv files with titles per column whereas my files are just arrays of values without object titles in the first row. I would like to avoid running multiple scripts to add titles.
My bonus question is something I have yet to find a good answer to at all, and is the most tedious part of processing. Using excels sort function, I usually change the order of the Yn values in Col2 such that they are sorted in the exported csv like so:
X1,N-Yn,n
...,...,...
Xn-1,N-Y2,2
Xn,N-Y1,1
Using the Col3 values as the sorting order (largest to smallest), then I delete this column so that the final saved csv only contains the first two columns (crucial step). Any help at all would be greatly appreciated, I apologize for the long-winded-ness of this question.
I have encountered looking to answer this question use csv files with titles per column whereas my files are just arrays of values without object titles in the first row.
The -Header parameter of Import-Csv is for adding column headers when the file does not contain them. It takes an array of strings, of however many columns there are.
I would like to avoid running multiple scripts to add titles.
If you couldn't use -Header, you could read the lines with Get-Content into memory, add a header in memory, and then use ConvertFrom-CSV all in one script.
That said, if I'm reading it rightly, you want:
No headers in the input file, and I imagine no headers in the output file
The whole point of adding the third column and sorting and removing it is just to reverse the lines?
The only column you keep is column 1?
I wouldn't use Import-Csv for this, it won't make it much nicer.
$n = 65536
# Read lines into a list, and reverse it
$lines = [Collections.Generic.List[String]](Get-Content -LiteralPath 'c:\test\test.csv')
$lines.Reverse()
# Split each line into two, create a new line with X and N-Y
# write new lines to an output file
$lines | ForEach-Object {
$x, $y = $_.split(',')
"$x,$($n - [int]$y)"
} | Set-Content -LiteralPath 'c:\test\output.csv' -Encoding Ascii
If you do want to use CSV handling, then:
$n = 65536
$counter = 1
Import-Csv -LiteralPath 'C:\test\test.csv' -Header 'ColX', 'ColY' |
Add-Member -MemberType ScriptProperty -Name 'ColN-Y' -Value {$n - $_.ColY} -PassThru |
Add-Member -MemberType ScriptProperty -Name 'N' -Value {$script:counter++} -PassThru |
Sort-Object -Property 'N' -Descending |
Select-Object -Property 'ColX', 'ColN-Y' |
Export-Csv -LiteralPath 'c:\test\output.csv' -NoTypeInformation
But the output will have CSV headers and double-quoted values.
I would try something like, by extending the original table with a calculatable script-property as a new column:
#Your N number
$N = 65536
# Import CSV file without header columns
$table = Import-Csv -Header #("colX","colY") `
-Delimiter ',' `
-Path './numbers.csv'
Write-Host "Original table"
$table | Format-Table
# Manipulate table
$newtable = $table |
Add-Member -MemberType ScriptProperty -Name colNX -Value { $N-$this.colX } - PassThru
Write-Host "New table"
$newtable | Format-Table
I require an additional column added to this output called "Location" with each row of data containing the word "Varonis".
$fs | ForEach-Object {
$machine = $_.ServerName
$_.Volumes | Select-Object #{n='machine';e={$machine}}, Share, FileWalkMethod
} | Export-Csv D:\data\splunk\otl_varonis\otl_varonis_monitoring.csv -NoType
Current output:
"hmanas01n","E$","VaronisWindows"
Desired output:
"hmanas01n","E$","VaronisWindows", "Varonis"
Essentially this is a question of how to add custom fields to an existing object (i.e. data)...
In the code example you give, the columns of output are defined by the following command:
Select-Object #{n='machine';e={$machine}}, Share, FileWalkMethod
This effectively filters the data to just the Share and FileWalkMethod properties, as well as adding a new column called machine. The machine column is defined with two key-value pairs: n (or name/label) is the column title and e (or expression) is a bit of code that sets the value for that column; in this case whatever the $machine variable is set to.
You need to replicate the same mechanism that adds the machine column by adding this to the end of the Select-Object statement, where the expression is just a static value:
#{n="Location";e={"Varonis"}}
How can I split a hashtable starting from a specific key/value pair?
I have hashtable like the following, just longer:
Name Value
---- -----
Name Alpha
Age 2
Position Trick
Start date 01-01-31
End date Unknown
Name Corax
Age 21
Position Sneak
Earnings 40'000
End Date Unknown
Name Horus
Age 22
Position Dead
Why Heresy
End date 03-30-30
I tried Group-Object but it failed.
I particularly wanted to separate it by Name and everything aside from Name, Age and Position are not consistent.
My actual issue is that I want to parse the hashtable for the Name and Age when Why = Heresy, and unfortunately, the original source of the data is a list of strings, which is the reason why I convert it to a hashtable.
Hashtables are not ordered, so you can't rely on a concept of "before" and "after". If you know the specific names of one complete set of keys, then you can loop through the hashtable and build two new ones, so if you wanted one hashtable to contain the Name, Age, and Position, and the other to contain everything else, you can do something like this:
$New1 = #{}
$New2 = #{}
$KeysGroup1 = #('Name','Age','Position') # This could just be one value
$MyHashTable.GetEnumerator().ForEach({
if ($_.Key -in $KeysGroup1) {
$New1[$_.Key] = $_.Value
} else {
$New2[$_.Key] = $_.Value
}
})
You can use an ordered dictionary if order is important for you. You can use a shortcut to create a literal ordered dictionary by preceding a literal hashtable with the [ordered] type accelerator:
$myOrdered = [ordered]#{ a = 1; z = 5; g = 2 }
From there you could do a similar approach to above that relies on order.
To create an ordered dictionary that isn't based on a literal:
$myOrdered = New-Object System.Collections.Specialized.OrderedDictionary
# or in PowerShell v5
$myOrdered = [System.Collections.Specialized.OrderedDictionary]::new()
$myOrdered.Add('key','value')
Edit based on comments
It sounds more like what you have is an array of hashtables and you now want to go about filtering these.
A [hashtable] can (and is often) used as a sort of proto-object, and it can be very useful for that because it often supports the same syntax, and it has built-in literal support.
But you're starting to run into their limits, and at this point I think you want to be dealing with an array of objects and not an array of hashtables.
Luckily, there are really easy ways to create objects in PowerShell right from a hashtable:
$obj = New-Object -TypeName PSObject -Property $myHash
$obj = [PSCustomObject]$myHash
$objArray = $myHashArray.ForEach({[PSCustomObject]$myHash})
Once you've got your array of objects, the real fun begins:
$heretics = $objArray.Where({$_.Why -eq 'Heresy'})
You'll notice I didn't even bother filtering out the other properties here. You shouldn't, until you really need to. Then you can use Select-Object or just access the properties you need. So for display purposes you might just do:
$heretics | Format-Table Name,Age
There's more stuff you can do with an object that you can't with hashtables, like add special types of properties:
$objArray | Add-Member -MemberType ScriptProperty -Name IsHeretic -Value { $this.Why -eq 'Heresy' } -Force
$heretics = $objArray.Where({$_.IsHeretic})
So, in the end; I figured something out, Thanks a lot to #briantist for guidance to the right direction.
My solution code is:
$startobj = (0..($hastable.count -1)) | where {$hashtable[$_].keys -like "*Name*"}
$endobj = (0..($hashtable.count -1)) | where {$hashtable[$_].keys -like "*End date*"}
$objindex = (0..($hashtable.count -1))
$numberofobjindex = 0..($hashtable.Where({$_.keys -like "*Name*"}).count - 1)
$hashtableparsed = foreach ($numba in $numberofobjindex) {
$Conv2data = $hashtable[($startobj[$numba]..$endobj[$numba])]
[PSCustomObject] #{
Name = $Conv2data.Name
Age = $Conv2data.Age
Why = $Conv2data.Why
}
}
$hashtableparsed
I realized that the hashtable data I was presented with had a repeating pattern with Name at the start of each cycle, and End date at the end.
So I basically did an indexing of the hashtable, and marked and indexed all instances where the cycle would start and end.
I then counted the cycles, and FOR EACH cycle captured the data of hashtable data in the lines particularly of that cycle and turned it into a PSCustomObject
In PowerShell I want to pass the name of a header in a CSV file into another function in the PowerShell script.
How can I retrieve the value-text of a header name into a variable in CSV?
e.g. if I have the following CSV data:
ID Name Country
-- ---- -------
1 John United States
2 Beatrice Germany
3 Jouni Finland
4 Marcel France
In the above example how can I retrieve the Country column value text as "Country" text into a variable in my script?
(Note: I am familiar with the notation $_.Country to retrieve the value of, for example, "Germany" from a row by importing the CSV in Powershell)
My specific issue is that currently I have the following function in my script:
function GetItemIdFromTitle([string]$LookupTitle, [ref]$LookupId)
{
$LookupField = $LookupList.Fields["DEPTCATEGORY"]
$LookupItem = $LookupList.Items | where {$_['DEPTCATEGORY'] -like "*$LookupTitle*"}
$LookupId.Value = $LookupItem.ID
}
This currently takes a string value -> $LookupTitle and uses that to find an item in a SharePoint list. As you can see in the script I am hard-coding in the column name as "DEPTCATEGORY". This is the column name that will be looked up to in the SharePoint list.
Instead of hard-coding the column name I want to pass in the name of the column for the corresponding $LookupTitle value and replace the hard-coded "DEPTCATEGORY".
I am calling the above function as follows:
#GET THE LOOKUP COLUMN ID
GetItemIdFromTitle $_.DEPTCAT ([ref]$LookupIdValue)
( $_.DEPTCAT is the value from the row in the CSV column. )
Can I do something like
$myCountryColumnName = $_.Country.Property.Title
or
$myCategoryColumnName = $_.DEPTCAT.Property.Name
to get the column name from the CSV?
If you have an object in $obj, you could list all the property headers like this:
$obj | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name'
This is an array, so you can reference them individually like this:
($obj | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name')[0]
This would just give you the name of the first property for instance.
Assuming that you have already read in the CSV using Import-CSV you can examine the property of the resulting object. Similar to this answer but maintains column order in the resulting array
Examining the first row / element
$data = import-csv $path
$data[0].psobject.properties.name
So the second line will return a string array of the properties.
To get the column name from the csv, first, put the names of the column headers into an array (this will also allow you to loop through each column, if needed) ...
$inFilePath = "C:\path\to\file.csv"
$csvColumnNames = (Get-Content $inFilePath | Select-Object -First 1).Split(",")
... , secondly, index into the array by column position (index starts at 0). Given your original example it would be;
$myCountryColumnName = $csvColumnNames[2]
This is more of a general comment than an answer.
I needed to pull the first column header name from CSVs and I started with selecting the NoteProperty fields from Get-Member. This doesn't work because the order of the NoteProperty column header names might not match the order of the column headers in the CSV file.
futureSPQR's method will work every time because the text won't get reordered on you. Below is my one-liner version of his method to get the first column header name.
((Get-Content filename.csv)[0] -split(','))[0]
$f = Import-Csv "...csv"
Get-Member -InputObject $f[0] |
Where-Object {$_.MemberType -eq "NoteProperty"} |
select-object Name
If you're looking to check if a header name exists in the array, use below code
$Array | Get-member -MemberType 'NoteProperty'| where {$_.Name -eq "ColumnName"}