How to pipe foreach output to export-csv - powershell

The code below outputs the following.
TYPE System.String
Length
100
How can I get it to actually output the content of the strings?
$fileIn | % { $array = $_.split(" ")
if ($array[0] -eq "User") {
$name = $array[1]+"."+$array[2]
$remaining = ""
for ($i = 3; $i -ne $array.length; $i++) {$remaining+=$array[$i]+" "}
Get-ADUser $name -properties description, company | % { $name + " - " +
$remaining + " - " + $_.description + " - " +
$_.company | Export-CSV $output}
}
}

Export-CSV is for exporting objects with properties to csv. You're trying to export a single string, which only includes a value and a Length property.
Also, - is not a valid delimiter in csv(at least not in .NET). The type information can be removed with a -NoTypeInformation-switch. Try this:
$fileIn | % { $array = $_.split(" ")
if ($array[0] -eq "User") {
$name = $array[1]+"."+$array[2]
$remaining = ""
for ($i = 3; $i -ne $array.length; $i++) {$remaining+=$array[$i]+" "}
Get-ADUser $name -properties description, company | % {
New-Object psobject -Property #{
"Name" = $name
"Remaining" = $remaining
"Description" = $_.Description
"Company" = $_.Company
}
}
}
} | Select-Object Name, Remaining, Description, Company |
Export-CSV $output -Delimiter ';' -NoTypeInformation
I tried to understand what you where trying to do here. To give you a summary of the changes:
I'm creating an object containing the information you want to export, for every row in your $filein
I'm setting the order of the properties with select-object AFTER an object for every line in $filein has been created
I'm exporting the array of objects to a csv-file with delimiter ; (just to show how you specify it), and without the type-information at the start. If you use export-csv inside the foreach loop, it would overwrite the file every time and you'd just have one row + header-row in the end. In PS3.0 you could have done it inside the loop, using -Append switch.
EDIT If you really need the string format, you need to use something else then Export-CSV, ex. Out-File with -Append switch. Ex:
$fileIn | % { $array = $_.split(" ")
if ($array[0] -eq "User") {
$name = $array[1]+"."+$array[2]
$remaining = ""
for ($i = 3; $i -ne $array.length; $i++) {$remaining+=$array[$i]+" "}
Get-ADUser $name -properties description, company | % {
"$name - $remaining - $($_.description) - $($_.company)" | Out-File -Append $output
}
}
}

Related

Check all lines in a huge CSV file in PowerShell

I want to work with a CSV file of more than 300,000 lines. I need to verify information line by line and then display it in a .txt file in the form of a table to see which file was missing for all servers. For example
Name,Server
File1,Server1
File2,Server1
File3,Server1
File1,Server2
File2,Server2
...
File345,Server76
File346,Server32
I want to display in table form this result which corresponds to the example above:
Name Server1 Server2 ... Server 32 ....Server 76
File1 X X
File2 X X
File3 X
...
File345 X
File346 X
To do this actually, I have a function that creates objects where the members are the Server Name (The number of members object can change) and I use stream reader to split data (I have more than 2 columns in my csv so 0 is for the Server name and 5 for the file name)
$stream = [System.IO.StreamReader]::new($File)
$stream.ReadLine() | Out-Null
while ((-not $stream.EndOfStream)) {
$line = $stream.ReadLine()
$strTempo = $null
$strTempo = $line -split ","
$index = $listOfFile.Name.IndexOf($strTempo[5])
if ($index -ne -1) {
$property = $strTempo[0].Replace("-", "_")
$listOfFile[$index].$property = "X"
}
else {
$obj = CreateEmptyObject ($listOfConfiguration)
$obj.Name = $strTempo[5]
$listOfFile.Add($obj) | Out-Null
}
}
When I export this I have a pretty good result. But the script take so much time (between 20min to 1hour)
I didn't know how optimize actually the script. I'm beginner to PowerShell.
Thanks for the futures tips
You might use HashSets for this:
$Servers = [System.Collections.Generic.HashSet[String]]::New()
$Files = #{}
Import-Csv -Path $Path |ForEach-Object {
$Null = $Servers.Add($_.Server)
if ($Files.Contains($_.Name)) { $Null = $Files[$_.Name].Add($_.Server) }
else { $Files[$_.Name] = [System.Collections.Generic.HashSet[String]]$_.Server }
}
$Table = foreach($Name in $Files.get_Keys()) {
$Properties = [Ordered]#{ Name = $Name }
ForEach ($Server in $Servers) {
$Properties[$Server] = if ($Files[$Name].Contains($Server)) { 'X' }
}
[PSCustomObject]$Properties
}
$Table |Format-Table -Property #{ expression='*' }
Note that in contrast to PowerShell's usual behavior, the .Net HashSet class is case-sensitive by default. To create an case-insensitive HashSet use the following constructor:
[System.Collections.Generic.HashSet[String]]::New([StringComparer]::OrdinalIgnoreCase)
See if this works faster. Change filename as required
$Path = "C:\temp\test1.txt"
$table = Import-Csv -Path $Path
$columnNames = $table | Select-Object -Property Server -Unique| foreach{$_.Server} | Sort-Object
Write-Host "names = " $columnNames
$groups = $table | Group-Object {$_.Name}
$outputTable = [System.Collections.ArrayList]#()
foreach($group in $groups)
{
Write-Host "Group = " $group.Name
$newRow = New-Object -TypeName psobject
$newRow | Add-Member -NotePropertyName Name -NotePropertyValue $group.Name
$servers = $group.Group | Select-Object -Property Server | foreach{$_.Server}
Write-Host "servers = " $servers
foreach($item in $columnNames)
{
if($servers.Contains($item))
{
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue 'X'
}
else
{
#if you comment out next line code doesn't work
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue ''
}
}
$outputTable.Add($newRow) | Out-Null
}
$outputTable | Format-Table

checking a large csv file for matching ID's taking long time

The below powershell is functioning properly- however it is taking a long time like 10 mins on a 50k line csv. I am sure I am making it work harder then it needs- I only need to match the ID column and then return the columns for each match. Any ideas of how to make this faster of more efficient?
$ID = #()
$fname = #()
$lname = #()
$mname = #()
$streetadd = #()
$apartment = #()
$city = #()
$state = #()
$zip = #()
$Ids = #(0317,11432,1104,9999,1955)
Write-Host "PLEASE WAIT >>> "
Import-Csv C:\mycsv.csv |`
ForEach-Object {
$ID += $_."ID"
$fname += $_."First Name"
$lname += $_."Last Name"
$mname += $_."Middle Name"
$streetadd += $_."Street Address"
$apartment += $_."Apartment"
$city += $_."City"
$state += $_."State"
$zip += $_."Zip"
}
foreach ($Id in $Ids) {
foreach ($elem in $ID) {
# Write-Host $Id
if ($Id -contains $elem)
{
#Write-Host "Customer Exists!"
$Where = [array]::IndexOf($ID, $elem)
Write-Host $ID[$Where] $fName[$Where] $lname[$Where] $mname[$where] $streetadd[$where] $apartment[$where] $city[$where] $state[$where] $zip[$where]
}
}
}
Trying the below code from answer below #Moerwald and getting no results-
$Ids = #(1317,1132,110,9999,1955)
$rows = #(Import-Csv C:\mycsv-csv.csv |? { $Ids -contains $_.id})
foreach ($r in $rows) {
write-host $r.id; $r.fname
}
$Ids = #(0317,11432,1104,9999,1955)
$rows = #(Import-Csv C:\mycsv.csv |? { $Ids -contains $_.ID})
$rows will be array of filtered rows. You can iterate over the array via:
$rows | % { Write-Host "$($_.Id)"}
$_ references a filtered row, andhas properties that corresponding to the column names.
? is a shortcut for the where-object cmdlet.
% is a shortcut for the foreach-object cmdlet.
Update:
This code works:
$s =#'
Student ID,OtherID,First Name,Last Name,Middle Name,Birth Date,,,,,,,,Street Address Line 1,Street Address Line 2,Apartment,City,State,Zip
1317,,a,b,c,6/11/2019,,,,,,,,1 5th dr,,,main,nv,55555
1132,,d,e,f,6/10/2019,,,,,,,,7 24th dr,,,duke,az,55555
'#
$csv = convertfrom-csv $s
$Ids = #(1317,1132, 11432,1104,9999,1955)
$rows = $csv |? { $Ids -contains $_.'Student ID'}
$rows | % { $_.'Student ID'}
This returns:
1317
1132
Here is the link to the running version.

Converting CSV Column to Rows

I'm trying to create a PowerShell script that transpose a CSV file from column to rows.
I found examples of doing the opposite (converting row based CSV to column) but I found nothing on column to rows. My problem being that I don't know exactly how many column I'll have. I tried adapting the row to column to column to rows but unsuccessfully.
$a = Import-Csv "input.csv"
$a | FT -AutoSize
$b = #()
foreach ($Property in $a.Property | Select -Unique) {
$Props = [ordered]#{ Property = $Property }
foreach ($Server in $a.Server | Select -Unique){
$Value = ($a.where({ $_.Server -eq $Server -and
$_.Property -eq $Property })).Value
$Props += #{ $Server = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
$b | FT -AutoSize
$b | Out-GridView
$b | Export-Csv "output.csv" -NoTypeInformation
For example my CSV can look like this:
"ID","DATA1"
"12345","11111"
"54321","11111"
"23456","44444"
or this (number of column can vary):
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555",
and I would like the script to convert it like this:
"ID","DATA"
"12345","11111"
"12345","22222"
"12345","33333"
"54321","11111"
"23456","44444"
"23456","55555"
The trick is to query the members of the table to get the column names. Once you do that then the rest is straightforward:
function Flip-Table ($Table) {
Process {
$Row = $_
# Get all the columns names, excluding the ID field.
$Columns = ($Row | Get-Member -Type NoteProperty | Where-Object Name -ne ID).Name
foreach ($Column in $Columns) {
if ($Row.$Column) {
$Properties = [Ordered] #{
"ID" = $Row.ID
"DATA" = $Row.$Column
}
New-Object PSObject -Property $Properties
}
}
# Garbage collection won't kick in until the end of the script, so
# invoke it every 100 input rows.
$Count++;
if (($Count % 100) -eq 0) {
[System.GC]::GetTotalMemory('forceFullCollection') | out-null
}
}
}
Import-Csv input.csv | Flip-Table | Export-Csv -NoTypeInformation output.csv
Well, here is mine. I'm not as fancy as the rest:
$in = Get-Content input.csv | Select -Skip 1
$out = New-Object System.Collections.ArrayList
foreach($row in $in){
$parts = $row.Split(',')
$id = $parts[0]
foreach($data in $parts[1..$parts.Count]){
if($data -ne '' -AND $data -ne $null){
$temp = New-Object PSCustomObject -Property #{'ID' = $id;
'Data' = $data}
$out.Add($temp) | Out-Null
}
}
}
$out | Export-CSV output.csv -NoTypeInformation
You can do something like this
# Convert csv to object
$csv = ConvertFrom-Csv #"
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555"
"#
# Ignore common members and the ID property
$excludedMembers = #(
'GetHashCode',
'GetType',
'ToString',
'Equals',
'ID'
)
$results = #()
# Iterate around each csv row
foreach ($row in $csv) {
$members = $row | Get-Member
# Iterate around each member from the 'row object' apart from our
# exclusions and empty values
foreach ($member in $members |
Where { $excludedMembers -notcontains $_.Name -and $row.($_.Name)}) {
# add to array of objects
$results += #{ ID=$row.ID; DATA=$row.($member.Name)}
}
}
# Write the csv string
$outstring = "ID,DATA"
$results | foreach { $outstring += "`n$($_.ID),$($_.DATA)" }
# New csv object
$csv = $outstring | ConvertFrom-Csv
Probably not the most elegant solution, but should do what you need
I left some comments explaining what it does
If you only want to accept a limited number DATA columns (e.g. 5), you could do:
ForEach ($i in 1..5) {$CSV | ? {$_."Data$i"} | Select ID, #{N='Data'; E={$_."Data$i"}}}
And if you have a potential unlimited number of DATA columns:
ForEach ($Data in ($CSV | Select "Data*" -First 1).PSObject.Properties.Name) {
$CSV | ? {$_.$Data} | Select ID, #{N='Data'; E={$_.$Data}}
}

Change the script to export Groups and nested objects differently

I created a similiar script like that:
$Groups = Get-QADGroup
$Result = #()
$Groups | ForEach-Object {
$Group = $_
$Members = Get-QADGroupMember $Group -Indirect | ? objectClass -eq "user"
$Obj = '' | Select-Object -Property Name, Members
$Obj.Name = $Group.Name
$Obj.Members = ($Members | % {$_.SamAccountName + "_" + 'Test'})
$Result += $Obj
}
$Result | Export-Csv -Path C:\Temp\groups.csv -NoTypeInformation -Encoding Unicode -Delimiter ";"
The output looks something like this (example data):
"Name";"Members"
"RootGroup01";"Subuser01_Test";"Subuser02_Test";"Subuser03_Test"
"RootGroup02";"Subuser02_Test"
"RootGroup03";"Subuser01_Test";"Subuser02_Test";"Subuser04_Test";
Is it possible to change the script, that I get something like this?:
"RootGroup01";"RootGroup02";"RootGroup03"
"Subuser01_Test";"Subuser02_Test";"Subuser01_Test"
"Subuser02_Test";;"Subuser02_Test"
"Subuser03_Test";;"Subuser04_Test"
The group names should be the header and the belonging users are in the right column. If there is no user, the column cell just stays empty.
You can write such function to transpose your data:
function Transpose-Data{
param(
[String[]]$Names,
[Object[][]]$Data
)
for($i = 0;; ++$i){
$Props = [ordered]#{}
for($j = 0; $j -lt $Data.Length; ++$j){
if($i -lt $Data[$j].Length){
$Props.Add($Names[$j], $Data[$j][$i])
}
}
if(!$Props.get_Count()){
break
}
[PSCustomObject]$Props
}
}
Then you invoke it in the following way:
$Groups = #(Get-QADGroup)
$Members = #(
$Groups | ForEach-Object {
,#(
Get-QADGroupMember $_ -Indirect |
? objectClass -eq user |
% {$_.SamAccountName + "_" + 'Test'}
)
}
)
Transpose-Data $Groups.Name $Members |
Export-Csv -Path C:\Temp\groups.csv -NoTypeInformation -Encoding Unicode -Delimiter ";"

Powershell Compare text and replace if it does not meet specific criteria

Here is a sample of the csv file I import.
CN,DistinguishedName,extensionattribute7,extensionattribute1
CNPTL73J79ZN1,"CN=CNPTL73J79ZN1,OU=Laptops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",tianyang.li,
USPTD079YZLN1,"CN=USPTD079YZLN1,OU=Desktops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",gary.ortiz,
USPTD07WM53M1,"CN=USPTD07WM53M1,OU=Desktops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",allen.watson,
USPTL7CC1P0P1,"CN=USPTL7CC1P0P1,OU=Laptops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",u0147066,
USPTL77BTZ4R1,"CN=USPTL77BTZ4R1,OU=Laptops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",U0172604,
U0165724-TPL-A,"CN=U0165724-TPL-A,OU=Laptops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",U0165724,167
U0130173-TPL-A,"CN=U0130173-TPL-A,OU=Laptops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",U0130173,167
U0068498-TPL-A,"CN=U0068498-TPL-A,OU=Laptops,OU=Workstations,OU=MSP01,DC=ten,DC=domain,DC=com",u0068498,167
A couple of things I need to do :
Check if the format of CN starts with UXXXXXXX
If it does not, check extensionattribute7 for proper formatted user id of Uxxxxxxx
If that exists, replace the CN with the name of Uxxxxxxx-TPL-ZZZ. the -TPL-ZZZ will be consistent though out all names.
I am totally confused how to search for the Uxxxxxxx but I need something like this, although I know this is completely incorrect.
Import-Csv c:\Temp\Windows7_Only.csv
if ($_CN -NotMatch'[U][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]')
{
Replace the name if extensionattribute7 contains a value of U####### and add the suffix of -TPL-ZZZ
}
Here is my script so far:
#Create an LDAP searcher object and pass in the DN of the domain we wish to query
$Searcher = New-Object System.DirectoryServices.DirectorySearcher([ADSI]"LDAP://DC=ten,DC=domain,DC=com")
#Pass in the ceriteria we are searching for.
$Searcher.Filter = "(&(objectCategory=computer)(objectClass=computer)(!UserAccountControl:1.2.840.113556.1.4.803:=2)(operatingSystem=Windows 7*))"
$Searcher.PageSize = 100000
# Populate General Sheet(1) with information
$results = $Searcher.Findall()
$results | ForEach-Object { $_.GetDirectoryEntry() } |
select #{ n = 'CN'; e = { $_.CN -replace "'", "''" } },
#{ n = 'DistinguishedName'; e = { $_.DistinguishedName -replace "'", "''" } },
#{ n = 'extensionattribute7'; e = { $_.extensionattribute7 -replace "'", "''" } },
#{ n = 'extensionattribute1'; e = { $_.extensionattribute1 -replace "'", "''" } } |
Export-Csv 'C:\temp\Windows7_Only.csv' -NoType -Force
$csv = Import-Csv -Path "c:\Temp\Windows7_Only.csv"
foreach ($row in $csv)
{
if (($row.CN -notmatch '^U\d{7}') -and ($row.DistinguishedName -like "*Laptops*") -and ($row.extensionattribute7 -match '^U\d{7}$'))
{
$row.CN = $row.extensionattribute7 + "-TPL-ZZZ"
}
elseif (($row.CN -notmatch '^U\d{7}') -and ($row.DistinguishedName -like "*Desktops*") -and ($row.extensionattribute7 -match '^U\d{7}$'))
{
$row.CN = $row.extensionattribute7 + "-TPD-ZZZ"
}
$csv | export-csv c:\fixed.csv -Force
}
Good start, though let me say that if you have access to the Active Directory snap-in you should totally use that rather than creating LDAP searchers and what not.
Now, about your comparison... As Matt said your Match should be against $.CN. What that means is $, which represents the current record as it loops through records, and the .CN portion indicates that it should look at the CN property of the record.
Then you can use -Match and (again) like Matt said (he's new here but proving to be knowledgeable), that can be shortened to "U\d{8}".
Now, you actually want to find those that aren't like U\d{8}, so let's precede that with a ! which is an alias for -Not. Then let's check and see if ExtendedAttribute7 is the right thing. So then this looks like:
!$_.CN -like "U\d{8}" -and $_.ExtendedAttribute7 -match "U\d{8}"
Excellent! We have our filter for the rows that need updating. That is pretty much what Alexander did. As for me, I'd go more (using your script as a base):
#Create an LDAP searcher object and pass in the DN of the domain we wish to query
$Searcher = New-Object System.DirectoryServices.DirectorySearcher([ADSI]"LDAP://DC=ten,DC=domain,DC=com")
#Pass in the ceriteria we are searching for.
#In this case we're looking for users with a particular SAM name.
$Searcher.Filter = "(&(objectCategory=computer)(objectClass=computer)(!UserAccountControl:1.2.840.113556.1.4.803:=2)(operatingSystem=Windows 7*))"
$Searcher.PageSize = 100000
# Populate General Sheet(1) with information
$results = $Searcher.Findall()
$Computers = #()
ForEach($Item in $results){
$Comp = $Item.GetDirectoryEntry()
If($Comp.distinguishedName -like "*desktops*"){$Suffix = "TPD-ZZZ"}else{$Suffix = "TPL-ZZZ"}
$CN = If(!$Comp.CN -match "U\d{8}" -and $Comp.extensionattribute7 -match "U\d{8}"){$Comp.extensionattribute7+$Suffix}else{$Comp.CN}
$Computers += [PSCustomObject][Ordered]#{
'CN' = $CN -replace "'", "''"
'DistinguishedName' = $Comp.DistinguishedName[0] -replace "'", "''"
'extensionattribute7' = $Comp.extensionattribute7[0] -replace "'", "''"
'extensionattribute1' = $Comp.extensionattribute1[0] -replace "'", "''"
}
}
$Computers | Export-Csv 'C:\temp\Windows7_Only.csv' -NoType -Force
$Computers
Assuming that I understand your requirements correctly:
$csv = Import-Csv -Path "c:\Temp\Windows7_Only.csv"
foreach ($row in $csv) {
if ($row.DistinguishedName -like "*Desktops*") {
$suffix = "-TPD-ZZZ"
}
elseif ($row.DistinguishedName -like "*Laptops*") {
$suffix = "-TPL-ZZZ"
}
if ( ($row.CN -notmatch '^U\d{7}') `
-and ($row.extensionattribute7 -match '^U\d{7}$') ) {
$row.CN = $row.extensionattribute7 + $suffix
}
}
$csv | export-csv c:\fixed.csv -Force -NoTypeInformation