I am trying to export Groupmembers list from azuread, my whole script works fairly well, but I need each new line in the final file to have the name of the file it is importing from (as well as the content it is importing)
the part of the script i am using to do this is as follows
(found this code here
Merging multiple CSV files into one using PowerShell)
get-childItem "C:\Users\user\Documents\Azure\Intune\management\*.csv" | foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath | Select -Skip 1
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "C:\Users\user\Documents\Azure\Intune\management\master_list.csv" $linesToWrite
}
I would probably do something like this:
$files = Get-ChildItem -Path "C:\Users\user\Documents\Azure\Intune\management\*.csv"
ForEach ($file in $files) {
Import-Csv -Path $File.FullName |
Select-Object -Property *, #{n='FileName';e={$file.Name}} |
Export-Csv -Path "C:\Users\user\Documents\Azure\Intune\management\master_list.csv" -NoTypeInformation -Append
}
Note that you need v3 or later of PowerShell to get the -Append flag for Export-Csv.
Another way to do it. This way would be potentially memory intensive if the files are large but I like the method and it fits well with the way my brain works.
$result = New-Object System.Collections.ArrayList
foreach($file in Get-ChildItem 'C:\Users\User\Documents\Azure\InTune\Management\*.csv'){
$result.AddRange((Import-CSV $file | Add-Member -InputObject $_ -Name 'File' -MemberType NoteProperty -Value $file.Name))
}
$result | Export-CSV 'C:\Users\user\Documents\Azure\Intune\management\master_list.csv' -NoTypeInformation
I think that would be version agnostic but I always lose track of which features happen in which version. Actually I think Add-Member would put it at v3+.
Related
getting memory exception while running this code. Is there a way to filter one file at a time and write output and append after processing each file. Seems the below code loads everything to memory.
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Get-ChildItem $inputFolder -File -Filter '*.csv' |
ForEach-Object { Import-Csv $_.FullName } |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
May be can you export and filter your files one by one and append result into your output file like this :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Remove-Item $outputFile -Force -ErrorAction SilentlyContinue
Get-ChildItem $inputFolder -Filter "*.csv" -file | %{import-csv $_.FullName | where machine_type -eq 'workstations' | export-csv $outputFile -Append -notype }
Note: The reason for not using Get-ChildItem ... | Import-Csv ... - i.e., for not directly piping Get-ChildItem to Import-Csv and instead having to call Import-Csv from the script block ({ ... } of an auxiliary ForEach-Object call, is a bug in Windows PowerShell that has since been fixed in PowerShell Core - see the bottom section for a more concise workaround.
However, even output from ForEach-Object script blocks should stream to the remaining pipeline commands, so you shouldn't run out of memory - after all, a salient feature of the PowerShell pipeline is object-by-object processing, which keeps memory use constant, irrespective of the size of the (streaming) input collection.
You've since confirmed that avoiding the aux. ForEach-Object call does not solve the problem, so we still don't know what causes your out-of-memory exception.
Update:
This GitHub issue contains clues as to the reason for excessive memory use, especially with many properties that contain small amounts of data.
This GitHub feature request proposes using strongly typed output objects to help the issue.
The following workaround, which uses the switch statement to process the files as text files, may help:
$header = ''
Get-ChildItem $inputFolder -Filter *.csv | ForEach-Object {
$i = 0
switch -Wildcard -File $_.FullName {
'*workstations*' {
# NOTE: If no other columns contain the word `workstations`, you can
# simplify and speed up the command by omitting the `ConvertFrom-Csv` call
# (you can make the wildcard matching more robust with something
# like '*,workstations,*')
if ((ConvertFrom-Csv "$header`n$_").machine_type -ne 'workstations') { continue }
$_ # row whose 'machine_type' column value equals 'workstations'
}
default {
if ($i++ -eq 0) {
if ($header) { continue } # header already written
else { $header = $_; $_ } # header row of 1st file
}
}
}
} | Set-Content $outputFile
Here's a workaround for the bug of not being able to pipe Get-ChildItem output directly to Import-Csv, by passing it as an argument instead:
Import-Csv -LiteralPath (Get-ChildItem $inputFolder -File -Filter *.csv) |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Note that in PowerShell Core you could more naturally write:
Get-ChildItem $inputFolder -File -Filter *.csv | Import-Csv |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Solution 2 :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8 # modify encoding if necessary
$Delimiter=','
#find header for your files => i take first row of first file with data
$Header = Get-ChildItem -Path $inputFolder -Filter *.csv | Where length -gt 0 | select -First 1 | Get-Content -TotalCount 1
#if not header founded then not file with sise >0 => we quit
if(! $Header) {return}
#create array for header
$HeaderArray=$Header -split $Delimiter -replace '"', ''
#open output file
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
#write header founded
$w.WriteLine($Header)
#loop on file csv
Get-ChildItem $inputFolder -File -Filter "*.csv" | %{
#open file for read
$r = New-Object System.IO.StreamReader($_.fullname, $encoding)
$skiprow = $true
while ($line = $r.ReadLine())
{
#exclude header
if ($skiprow)
{
$skiprow = $false
continue
}
#Get objet for current row with header founded
$Object=$line | ConvertFrom-Csv -Header $HeaderArray -Delimiter $Delimiter
#write in output file for your condition asked
if ($Object.machine_type -eq 'workstations') { $w.WriteLine($line) }
}
$r.Close()
$r.Dispose()
}
$w.close()
$w.Dispose()
You have to read and write to the .csv files one row at a time, using StreamReader and StreamWriter:
$filepath = "C:\Change\2019\October"
$outputfile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8
$files = Get-ChildItem -Path $filePath -Filter *.csv |
Where-Object { $_.machine_type -eq 'workstations' }
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
$skiprow = $false
foreach ($file in $files)
{
$r = New-Object System.IO.StreamReader($file.fullname, $encoding)
while (($line = $r.ReadLine()) -ne $null)
{
if (!$skiprow)
{
$w.WriteLine($line)
}
$skiprow = $false
}
$r.Close()
$r.Dispose()
$skiprow = $true
}
$w.close()
$w.Dispose()
get-content *.csv | add-content combined.csv
Make sure combined.csv doesn't exist when you run this, or it's going to go full Ouroboros.
So I'm trying to process CSV files, then giving the output new name. I can do it with one file by explicitly specifying the file name. But is there a way / wildcard I can use to make the script to process multiple files at the same time? Let's just say I want to process anything with .csv as an extension. Here's my script that's used to process a specific file
$objs =#();
$output = Import-csv -Path D:\TEP\FilesProcessing\Test\file1.csv | ForEach {
$Object = New-Object PSObject -Property #{
Time = $_.READ_DTTM
Value = $_.{VALUE(KWH)}
Tag = [String]::Concat($_.SUBSTATION,'_',$_.CIRCUITNAME,'_',$_.PHASE,'_',$_.METERID,'_KWH')
}
$objs += $Object;
}
$objs
$objs | Export-CSv -NoTypeInformation D:\TEP\FilesProcessing\Test\file1_out.csv
You can combine Get-ChildItem and Import-Csv.
Here's an example that specifies different input and output directories to avoid name collisions:
$inputPath = "D:\TEP\FilesProcessing\Test"
$outputPath = "D:\TEP\FilesProcessing\Output"
Get-ChildItem (Join-Path $inputPath "*.csv") | ForEach-Object {
$outputFilename = Join-Path $outputPath $_.Name
Import-Csv $_.FullName | ForEach-Object {
New-Object PSObject -Property #{
"Time" = $_.READ_DTTM
"Value" = $_.{VALUE(KWH)}
"Tag" = "{0}_{1}_{2}_{3}_KWH" -f $_.SUBSTATION,$_.CIRCUITNAME,$_.PHASE,$_.METERID
}
} | Export-Csv $outputFilename -NoTypeInformation
}
Note that there's no need for creating an array and repeatedly appending it. Just output the custom objects you want and export afterwards.
Use the Get-Childitem and cut out all the unnecessary intermediate variables so that you code it in a more Powershell type way. Something like this:
Get-CHhilditems 'D:\TEP\FilesProcessing\Test\*.csv' | % {
Import-csv $_.FullName | % {
New-Object PSObject -Property #{
Time = $_.READ_DTTM
Value = $_.{VALUE(KWH)}
Tag = '{0}_{1}_{2}_{3}_KWH' -f $_.SUBSTATION, $_.CIRCUITNAME, $_.PHASE, $_.METERID
}
} | Export-CSv ($_.FullName -replace '\.csv', '_out.csv') -NoTypeInformation
}
The Get-ChildItem is very useful for situations like this.
You can add wildcards directly into the path:
Get-ChildItem -Path D:\TEP\FilesProcessing\Test\*.csv
You can recurse a path and use the provider to filter files:
Get-ChildItem -Path D:\TEP\FilesProcessing\Test\ -recurse -include *.csv
This should get you what you need.
$Props = #{
Time = [datetime]::Parse($_.READ_DTTM)
Value = $_.{VALUE(KWH)}
Tag = $_.SUBSTATION,$_.CIRCUITNAME,$_.PHASE,$_.METERID,'KWH' -join "_"
}
$data = Get-ChildItem -Path D:\TEP\FilesProcessing\Test\*.csv | Foreach-Object {Import-CSV -Path $_.FullName}
$data | Select-Object -Property $Props | Export-CSv -NoTypeInformation D:\TEP\FilesProcessing\Test\file1_out.csv
Also when using Powershell avoid doing these things:
$objs =#();
$objs += $Object;
In a nutshell, I have an excel file that I need :
- Only 2 Columns (ComputerName, Results)
- Only need rows that contain specific items (IE. Start with DriveLetter:\, HKLM, %windir%, etc.)
I'm just not sure on the proper keyword syntax here. The original file is an xlsx.
Please forgive the crudeness of my script. I gathered bits and pieces trying to get it to work.
#File Import to Variable
Function Remove-File($fileName) {
if(Test-Path -path $fileName) { Remove-Item -path $fileName }
}
$excelFile = ".\Computers.xlsx"
if(Test-Path -path $excelFile) {
$csvFile = ($env:temp + "\" + ((Get-Item -path $excelFile).name).Replace(((Get-Item -path $excelFile).extension),".csv"))
Remove-File $csvFile
$excelObject = New-Object -ComObject Excel.Application
$excelObject.Visible = $false
$workbookObject = $excelObject.Workbooks.Open($excelFile)
$workbookObject.SaveAs($csvFile,6) # http://msdn.microsoft.com/en-us/library/bb241279.aspx
$workbookObject.Saved = $true
$workbookObject.Close()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($workbookObject) | Out-Null
$excelObject.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excelObject) | Out-Null
[System.GC]::Collect()
[System.GC]::WaitForPendingFinalizers()
$spreadsheetDataObject = Import-Csv -path $csvFile # Use the $spreadsheetDataObject for your analysis
Remove-File $csvFile
}
#Filter Out All Columns except ComputerName, Results and subseqently create a CSV file
$PathCSV = ".\Computers.csv"
$spreadsheetDataObject | Select-Object ComputerName,Results | Export-Csv -Path $PathCSV -NoTypeInformation
$Keywords = "*HKLM*","*C:\*","*%windir%*"
$Filter = "($(($Keywords|%{[RegEx]::Escape($_)}) -join "|"))"
Import-CSV $PathCSV | Where-Object{$_Results -match $Keywords} | Export-Csv -Path ".\Computers2.csv" -NoTypeInformation
I found the issue. I needed the _.Results..... and I needed to -match $Filter
Import-CSV $PathCSV | Where-Object{$_.Results -match $Filter} | Export-Csv -Path ".\Computers2.csv" -NoTypeInformation
I need to take a slew of csv files from a directory and get them into an array in Powershell (to eventually manipulate and write back to a CSV).
The problem is there are 5 file types. I need around 8 columns from each. The columns are essentially the same, but have different headings.
Is there an easy way to do this? I started creating a custom object with my 8 fields, looping through the files importing each one, looking at the filename (which tells me the column names I need) and then a bunch of ifs to add it to my custom object array.
I was wondering if there is a simpler way...like with a template saying which columns from each file.
wound up doing this. It may have not been the most efficient, but works. I wound up writing out each file separately and combining at the end as PS really got bogged down (over a million rows combined).
$Newcsv = #()
$path = "c:\scrap\BWFILES\"
$files = gci -path $path -recurse -filter *.csv | Where-Object { ! ($_.psiscontainer) }
$counter=1
foreach($file in $files)
{
$csv = Import-Csv $file.FullName
if ($file.Name -like '*SAV*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"SV"}},DMBRCH,DMACCT,DMSHRT
}
if ($file.Name -like '*TIME*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"TM"}},TMBRCH,TMACCT,TMSHRT
}
if ($file.Name -like '*TRAN*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"TR"}},DMBRCH,DMACCT,DMSHRT
}
if ($file.Name -like '*LN*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"LN"}},LNBRCH,LNNOTE,LNSHRT
}
$Newcsv | Export-Csv "C:\scrap\$file.name$counter.csv" -force -notypeinformation
$counter++
}
get-childItem "c:\scrap\*.csv" | foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "c:\scrap\combined.csv" $linesToWrite
}
With a hashtable for reference, a little RegEx matching, and using the automatic variable $Matches in a ForEach-Object loop (alias % used) that could all be shortened to:
$path = "c:\scrap\BWFILES\"
$Reference = #{
'SAV' = 'SV'
'TIME' = 'TM'
'TRAN' = 'TR'
'LN'='LN'
}
Set-Content -Value "PRODUCT,BRCH,ACCT,SHRT" -Path 'c:\scrap\combined.csv'
gci -path $path -recurse -filter *.csv | Where-Object { !($_.psiscontainer) -and $_.Name -match ".*(SAV|TIME|TRAN|LN).*"}|%{
$Product = $Reference[($Matches[1])]
Import-CSV $_.FullName | Select-Object #{Name="PRODUCT";Expression={$Product}},*BRCH,#{l='Acct';e={$_.LNNOTE, $_.DMACCT, $_.TMACCT|?{$_}}},*SHRT | ConvertTo-Csv -NoTypeInformation | Select -Skip 1 | Add-Content 'c:\scrap\combined.csv'
}
That should produce the exact same file. Only kind of tricky part was the LNNOTE/TMACCT/DMACCT field since obviously you can't just do the same as like *SHRT.
I am trying to figure out how to correct this script I've wrote. I know it is something wrong with the way it is importing the list of hostnames. I don't know how to fix it.
Part 1: This is supposed to import a .csv with the hostnames and dig the registry for the application's uninstall information, put it into an array, and export into .csv's for later use. Also it creates .txt files in order to later compare the applications on the system to a baseline.
$path = "\\path"
$computers = Import-Csv -Path "\\Path\hostnames.csv"
$array = #()
foreach($pc in $computers)
{
$computername = $pc.computername
#$computername = "KNOWN_HOSTNAME" #test line for one system
$UninstallKey = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall"
$reg = [microsoft.win32.registrykey]::OpenRemoteBaseKey('LocalMachine',$computername)
$regkey = $reg.OpenSubKey($UninstallKey)
$subkeys = $regkey.GetSubKeyNames()
foreach($key in $subkeys)
{
$thisKey=$UninstallKey+"\\"+$key
$thisSubKey=$reg.OpenSubKey($thisKey)
$obj = New-Object PSObject
$obj | Add-Member -MemberType NoteProperty -Name "ComputerName" -Value $computername
$obj | Add-Member -MemberType NoteProperty -Name "DisplayName" -Value $($thisSubKey.GetValue("DisplayName"))
$obj | Add-Member -MemberType NoteProperty -Name "DisplayVersion" -Value $($thisSubKey.GetValue("DisplayVersion"))
$obj | Add-Member -MemberType NoteProperty -Name "Publisher" -Value $($thisSubKey.GetValue("Publisher"))
$obj | Add-Member -MemberType NoteProperty -Name "InstallDate" -Value $($thisSubKey.GetValue("InstallDate"))
$array += $obj
$ExportArray = $array | Where-Object { $_.DisplayName } |
select ComputerName , DisplayName, DisplayVersion, Publisher, InstallDate
$ExportArray |
Export-csv $path\$computername.csv -NoTypeInformation
$ExportArray2 = $array |
Where-Object { $_.DisplayName } |
select DisplayName, DisplayVersion, Publisher
$ExportArray2 |
Export-csv $path\$computername.txt -NoTypeInformation
}
}
Part 2: This portion compiles the .csv's into one excel document for reporting
$csvs = Get-ChildItem $path\* -Include *.csv
$outputfilename = "Network_" + (Get-Date -Format yyyyMMdd)
$excelapp = new-object -comobject Excel.Application
$excelapp.sheetsInNewWorkbook = $csvs.Count
$xlsx = $excelapp.Workbooks.Add()
$sheet=1
foreach ($csv in $csvs)
{
$row=1
$column=1
$worksheet = $xlsx.Worksheets.Item($sheet)
$worksheet.Name = $csv.Name
$file = (Get-Content $csv.PSPath | ForEach-Object {$_ -replace '"', ""})
foreach($line in $file)
{
$linecontents = $line -split ‘,(?!\s*\w+”)’
foreach($cell in $linecontents)
{
$worksheet.Cells.Item($row,$column) = $cell
$column++
}
$column = 1
$row++
}
$sheet++
}
$output = $path + “\” + $outputfilename + ".xlsx"
$xlsx.SaveAs($output)
$excelapp.quit()
Part 3: This portion loads up a baseline, and the .txt's created preciously, and checks for differences in the files. (also deletes blank ouput files)
$bline = Get-ChildItem $path\* -Include Baseline.txt
$txts = Get-ChildItem $path\* -Include *.txt -Exclude Baseline.txt
foreach ($txt in $txts)
{
Compare-Object -referenceobject $(Get-Content $bline) -differenceobject $(Get-Content $txt) |
ft inputobject, #{n = "file"; e = {if ($_.SideIndicator -eq '=>') {"System"} else {"Baseline"}}} |
Out-File $txt'_has_diff'.csv -Width 256
Get-ChildItem $path |
where {$_.Length -eq 0} |
Remove-Item
}
Thank you
Edit:
The Hostnames.csv files I've tried are:
HOSTNAME1
HOSTNAME2
and
"HOSTNAME1","HOSTNAME2"
It's a little unclear what the problem is, because you say there is "something wrong with the way it is importing the list of hostnames", but you haven't specified what kind of results you're getting and how they differ from the intended results.
However, based on your sample data I think I can infer what the problem is: You're trying to use Import-Csv on non-CSV data. Neither of your examples looks like a CSV file. They both look like lists. A list in which the items are separated by commas, such as
"HOSTNAME1","HOSTNAME2","HOSTNAME3","HOSTNAME4"
is not called a "CSV file". CSV files are a form of "flat file", in which the data represents the rows and columns of a single database table. An example of a CSV file would be something like this, where the first line is a list of field (column) names, and the other lines are records (rows) with the comma-separated values corresponding to the columns in the header row:
"Hostname","OS","OS Version","Primary Function","Location"
"BOSEXCH01","Windows","Server 2012","Microsoft Exchange","Boston"
"BOSDC01","Windows","Server 2008 R2","Active Directory domain controller","Boston"
"MYWEB","Linux","Ubuntu 13.04","Apache web server","Phoenix"
The cmdlet Import-Csv imports a CSV file into an array of objects in which the properties are the field names in the header row, and the values are the comma-separated items in each row corresponding to the property names derived from the header row. Export-Csv does the reverse—it creates a CSV file from an array of objects.
It looks like what you're trying to do is read a simple list of hostnames into an array of strings. If your data looks like the first example,
HOSTNAME1
HOSTNAME2
[etc...]
you can read it into an array by simply using Get-Content, as follows (note that I changed the extension to .txt to reflect the actual format of the data):
$computers = Get-Content "\\Path\hostnames.txt"
If your data looks like the second example,
"HOSTNAME1","HOSTNAME2",[etc...]
you can read it into array like this:
$computers = (Get-Content "\\Path\hostnames.txt") -split ','
On the other hand, it appears that you are using Export-Csv correctly: You're exporting a bunch of objects with the same properties into a flat file, which is the correct usage of the term "CSV".