How can I prevent from forming multiple tables in a Powershell foreach? - powershell

I'm developing a script that connects to Azure AD and extracts failed login for any user so I'm probably going to get more than a row for user.
I have this code in a foreach (there is anything after part of code):
$ConvertedOutput | Select-Object #{Label="UserId"; Expression={$_.UserId}},
#{Label="CreationTime"; Expression={$_.CreationTime}},
#{Label="UserAgent"; Expression={$FinalOutput[0]."Value"}},
#{Label="Operation"; Expression={$_.Operation}},
#{Label="LogonError"; Expression={$_.LogonError}},
#{Label="ClientIP"; Expression={$_.ClientIP}} | Format-Table
How can I prevent from forming multiple tables? I only wanted the table for the first record, then additional records under the same table.
Thanks
here is the output

# Create an empty array to store your results in
[array]$results = #()
# This is your existing loop
foreach (...) {
...
$ConvertedOutput = <your existing code>
...
# Append your object to the results array
$results += $ConvertedOutput | Select-Object ... <your existing code>
}
# Now your results object contains all of the values from inside your loop
# So let's display that!
Write-Output $results

Welcome to stack overflow. I general, it is recommended to supply sample (fake) data in text format rather then pictures (of just headers), the makes life easier for us to answer your question.
Reading your code part, it doesn't add much value unless you planned to further extend it. All expressions generate the same keys and values as the original object, meaning that you can simplify this to just: Select-Object UserId, CreationTime, UserAgent, Operation, LogonError, LogonError, ClientIP or even: Select-Object * (or just omit the complete Select-Object), if you do not select a column subset.
With regards to your question,
By default PowerShell normally concatenates the output by it self, meaning that there is probably something else (that you are not sharing, e.g. a Write-Host command) that causes the data to be released preliminary from the pipeline.
Let me show this with fictive object lists created on the fly from three separate CSV lists:
$Result = &{
ConvertFrom-Csv #'
Id,FirstName,LastName
1,Nancy,Davolio
2,Andrew,Fuller
3,Janet,Leveling
'#
ConvertFrom-Csv #'
Id,FirstName,LastName
4,Margaret,Peacock
5,Steven,Buchanan
6,Michael,Suyama
'#
ConvertFrom-Csv #'
Id,FirstName,LastName
7,Robert,King
8,Laura,Callahan
9,Anne,Dodsworth
'#
}
With the above command, $Result contains the following data:
PS C:\> $Result
Id FirstName LastName
-- --------- --------
1 Nancy Davolio
2 Andrew Fuller
3 Janet Leveling
4 Margaret Peacock
5 Steven Buchanan
6 Michael Suyama
7 Robert King
8 Laura Callahan
9 Anne Dodsworth
One important thing to mention here, is that the columns of the three list should be have the same columns (or at least the first row should contain all expected columns), see: Not all properties displayed
If this doesn't help you further, I recommend you to add you more details to your question.

Related

How can I get these variables to output together? [duplicate]

I'm fairly new to PowerShell, and am wondering if someone knows of any better way to accomplish the following example problem.
I have an array of mappings from IP address to host-name. This represents a list of active DHCP leases:
PS H:\> $leases
IP Name
-- ----
192.168.1.1 Apple
192.168.1.2 Pear
192.168.1.3 Banana
192.168.1.99 FishyPC
I have another array of mappings from MAC address to IP address. This represents a list of IP reservations:
PS H:\> $reservations
IP MAC
-- ---
192.168.1.1 001D606839C2
192.168.1.2 00E018782BE1
192.168.1.3 0022192AF09C
192.168.1.4 0013D4352A0D
For convenience, I was able to produce a third array of mappings from MAC address to IP address and host name using the following code. The idea is that $reservations should get a third field, "Name", which is populated whenever there's a matching "IP" field:
$reservations = $reservations | foreach {
$res = $_
$match = $leases | where {$_.IP -eq $res.IP} | select -unique
if ($match -ne $NULL) {
"" | select #{n="IP";e={$res.IP}}, #{n="MAC";e={$res.MAC}}, #{n="Name";e={$match.Name}}
}
}
The desired output is something like this:
PS H:\> $ideal
IP MAC Name
-- --- ----
192.168.1.1 001D606839C2 Apple
192.168.1.2 00E018782BE1 Pear
192.168.1.3 0022192AF09C Banana
192.168.1.4 0013D4352A0D
Is there any better way of doing this?
After 1.5 years, the cmdlet I had pasted in the original answer has undergone so many updates that it has become completely outdated. Therefore I have replaced the code and the ReadMe with a link to the latest version.
Join-Object
Combines two object lists based on a related property between them.
Description
Combines properties from one or more objects. It creates a set that can be saved as a new object or used as it is. An object join is a means for combining properties from one (self-join) or more object lists by using values common to each.
Main features
Intuitive (SQL like) syntax
Smart property merging
Predefined join commands for updating, merging and specific join types
Well defined pipeline for the (left) input objects and output objects (preserves memory when correctly used)
Performs about 40% faster than Compare-Object on large object lists
Supports (custom) objects, data tables and dictionaries (e.g. hash tables) for input
Smart properties and calculated property expressions
Custom relation expressions
Easy installation (dot-sourcing)
Supports PowerShell for Windows (5.1) and PowerShell Core
The Join-Object cmdlet reveals the following proxy commands with their own (-JoinType and -Property) defaults:
InnerJoin-Object (Alias InnerJoin or Join), combines the related objects
LeftJoin-Object (Alias LeftJoin), combines the related objects and adds the rest of the left objects
RightJoin-Object (Alias RightJoin), combines the related objects and adds the rest of the right objects
FullJoin-Object (Alias FullJoin), combines the related objects and adds the rest of the left and right objects
CrossJoin-Object (Alias CrossJoin), combines each left object with each right object
Update-Object (Alias Update), updates the left object with the related right object
Merge-Object (Alias Merge), updates the left object with the related right object and adds the rest of the new (unrelated) right objects
ReadMe
The full ReadMe (and source code) is available from GitHub: https://github.com/iRon7/Join-Object
Installation
There are two versions of this Join-Object cmdlet (both versions supply the same functionality):
Join Module
Install-Module -Name JoinModule
Join Script
Install-Script -Name Join
(or rename the Join.psm1 module to a Join.ps1 script file)
and invoked the script by dot sourcing:
. .\Join.ps1
Answer
To answer the actual example in the question:
$reservations |LeftJoin $leases -On IP
IP MAC Name
-- --- ----
192.168.1.1 001D606839C2 Apple
192.168.1.2 00E018782BE1 Pear
192.168.1.3 0022192AF09C Banana
192.168.1.4 0013D4352A0D
Performance
A little word on performance measuring:
The PowerShell pipeline is designed to stream objects (which safes memory), meaning that both¹ lists of input objects usually aren't (shouldn't be) resident in memory. Normally they are retrieved from somewhere else (i.e. a remote server or a disk). Also, the output usually matters where linq solutions are fast but might easily put you on the wrong foot in drawing conclusions because linq literally defers the execution (lazy evaluation), see also: fastest way to get a uniquely index item from the property of an array.
In other words, if it comes to (measuring) performance in PowerShell, it is important to look to the complete end-to-end solution, which is more likely to look something like:
import-csv .\reservations.csv |LeftJoin (import-csv .\leases.csv) -On IP |Export-Csv .\results.csv
(1) Note: unfortunately, there is no easy way to build two parallel input streams (see: #15206 Deferred input pipelines)
(more) Examples
More examples can be found in the related Stack Overflow questions at:
Combining Multiple CSV Files
Combine two CSVs - Add CSV as another Column
CMD or Powershell command to combine (merge) corresponding lines from two files
Can I use SQL commands (such as join) on objects in powershell, without any SQL server/database involved?
Powershell match properties and then selectively combine objects to create a third
Compare Two CSVs, match the columns on 2 or more Columns, export specific columns from both csvs with powershell
Merge two CSV files while adding new and overwriting existing entries
Merging two CSVs and then re-ordering columns on output
Efficiently merge large object datasets having multiple matching keys
Is there a PowerShell equivalent of paste (i.e., horizontal file concatenation)?
How to compare two CSV files and output the rows that are in either of the file but not in both
How to join two CSV files in Powershell with SQL LIKE syntax
How can merge 3 cycle and export in one table
Merging two arrays object into one array object in powershell
And in the Join-Object test script.
Please give a 👍 if you support the proposal to Add a Join-Object cmdlet to the standard PowerShell equipment (#14994)
This can also be done using my module Join-Object
Install-Module 'Join-Object'
Join-Object -Left $leases -Right $reservations -LeftJoinProperty 'IP' -RightJoinProperty 'IP'
Regarding performance, I tested against a sample data of 100k lines:
Hashtable example posted by #js2010 run in 8 seconds.
Join-Object by me run in 14 seconds.
LeftJoin by #iRon run in 1 minute and 50 seconds
Here's a simple example using a hashtable. With big arrays, this turns out to be faster.
$leases =
'IP,Name
192.168.1.1,Apple
192.168.1.2,Pear
192.168.1.3,Banana
192.168.1.99,FishyPC' | convertfrom-csv
$reservations =
'IP,MAC
192.168.1.1,001D606839C2
192.168.1.2,00E018782BE1
192.168.1.3,0022192AF09C
192.168.1.4,0013D4352A0D' | convertfrom-csv
$hashRes=#{}
foreach ($resRecord in $reservations) {
$hashRes[$resRecord.IP] = $resRecord
}
$leases | foreach {
$other = $hashRes[$_.IP]
[pscustomobject]#{IP=$_.IP
MAC=$other.MAC
Name=$_.name}
}
IP MAC Name
-- --- ----
192.168.1.1 001D606839C2 Apple
192.168.1.2 00E018782BE1 Pear
192.168.1.3 0022192AF09C Banana
192.168.1.99 FishyPC
Easiest way I've found to Merge two Powershell Objects is using ConvertTo-Json and ConvertFrom-Json
One liner based on the OPs Senario:
$leases | foreach {(ConvertTo-Json $_) -replace ("}$", (ConvertTo-Json ($reservations | where IP -eq $_.IP | select * -ExcludeProperty IP)) -Replace "^{", ",")}
| ConvertFrom-Json
Results in:
IP Name Mac
-- ---- ---
192.168.1.1 Apple 001D606839C2
192.168.1.2 Pear 00E018782BE1
For another example lets make a couple objects:
$object1 = [PSCustomObject]#{"A" = "1"; "B" = "2"}
$object2 = [PSCustomObject]#{"C" = "3"; "D" = "4"}
Merge them together using Json by replacing the opening and closing brackets:
(ConvertTo-Json $object1) -replace ("}$", $((ConvertTo-Json $object2) -Replace "^{", ",")) | ConvertFrom-Json
Output:
A B C D
- - - -
1 2 3 4
Another example using a group of objects:
$mergedObjects = [PSCustomObject]#{"Object1" = $Object1; "Object2" = $Object2}
Object1 Object2
------- -------
#{A=1; B=2} #{C=3; D=4}
Can just do the same again within a foreach:
$mergedObjects | foreach {(ConvertTo-Json $_.Object1) -replace ("}$", $((ConvertTo-Json $_.Object2) -Replace "^{", ",")) | ConvertFrom-Json}
Output:
A B C D
- - - -
1 2 3 4
You can use script block like this
$leases | select IP, NAME, #{N='MAC';E={$tmp=$_.IP;($reservations| ? IP -eq $tmp).MAC}}

Generate word's list from two CSV files and export to CSV

I'm trying to use PowerShell to import two CSV files, one CSV is called accts and one is called names. The accts file has 8 numbers and the names files has 4 names. I need to output a combination where the end goal is to output a list of 32 words with the prefix of STX. I'll take this list of 32 and create 32 AD objects. Ideally I need to also run a check to see if the word/AD object already exists.
accts: 12345 23456 34567 45678 56789 67890 78901 89012
names: john dave joe mike
Output should be:
stx-12345-john
stx-23456-john
stx-34567-john
stx-45678-john
stx-56789-john
stx-67890-john
stx-78901-john
stx-89012-john
stx-12345-dave
stx-23456-dave
stx-34567-dave
stx-45678-dave
stx-56789-dave
stx-67890-dave
stx-78901-dave
stx-89012-dave
etc.
I've thought about trying to use arrays, but perhaps that would be to complex. Any advice?
Hmm i feel like you shouldn't be working on an AD with Powershell if this is to far stretched for you at this time. Yes you would use arrays and just loop through them.
So i've exploded my code as much as i can so you can learn the logic behind it.
$accts = #("12345","23456","34567","45678","56789","67890","78901","89012")
$names = #("john","dave","joe","mike")
$adobject = #()
foreach($n in $names){
# for each name, go through the accts and add the prefix
foreach($a in $accts){
#store all items in an array
$adobject += "stx-"+$a+"-"+$n
}
}
cls
# go through your array you just created
foreach($o in $adobject){
if(get-aduser $o){
Write-host "user $o already exists"
}else{
new-aduser $o #create user
}
}
Little tasks like this are the best way to learn coding and logic. You should be able to translate this to using CSV's.

Learning PowerShell. Create usernames no longer than 8 characters and check for collision

I'm learning powershell right now.
I need to import a CSV like this:
lastname,firstname
lastname,firstname
lastname,firstname
etc
Then create a list of usernames no longer then 8 characters and check for collisions.
I have found bits and pieces of scripting around but not sure how to tie it all together.
I use Import-Csv to import my file.csv:
$variablename = import-csv C:\path\to\file.csv
but then I am not sure if I just import it into an array or not. I am not familiar with how for loops work in powershell exactly.
Any direction? Thanks.
There are a couple of concepts that are central to understanding PowerShell. Firstly, remember that you are always working with objects. So after importing your CSV file, your $variablename will refer to a collection of sub-objects.
Secondly, you can use the PowerShell pipeline to send the output of one cmdlet to the input of another. Some cmdlets will understand if you send them a collection, and automatically process each row.
If think what you're looking for though is the foreach-object cmdlet, which will allow you to run code against each item in the collection. Code inside the foreach-object block can refer to the $_ automatic variable which will contain the current object.
Assuming your CSV file is well formatted and has a header row with the column names, you can refer to each column by name e.g. $_.lastname & $_.firstname.
To put it all together:
import-csv C:\path\to\file.csv |
foreach-object {
write-host "Processing: $($_.lastname), $($_.firstname)"
# logic here to calculate username and create AD account
}
PowerShell can have a bit of a learning curve if you are coming from a different scripting environment. Here are a couple of resources that I've found helpful:
PowerShell 'gotchas' http://www.rlmueller.net/PSGotchas.htm
Keith Hill's Effective PowerShell: https://rkeithhill.wordpress.com/2009/03/08/effective-windows-powershell-the-free-ebook/
Also, check out the Technet Script Center, where there are many hundreds of Active Directory scripts. https://technet.microsoft.com/en-us/scriptcenter/bb410849.aspx
The script below should help you grasp a few concepts on how to work with csvs and manipulate data using PowerShell.
# the code below uses a 'here string' to mimic the import of a csv.
$users = #'
smith,b
smith,bob
smith,bobby
smith,sonny
smithson,john
smithson,jane
smithers,rob
'# -split "`r*`n"
$users |
ConvertFrom-Csv -Header 'surname','firstname' |
Select-Object #{Name='username'; Expression={"$($_.surname)$($_.firstname) "}}, surname, firstname |
Group-Object { $_.username.Substring(0,8).Trim() } |
Select-Object #{Name='username'; Expression={$_.Name}}, Count |
Format-Table -AutoSize
The $users | line takes the list of $users and pipes into the next command.
The ConvertFrom-Csv -Header... line converts the string into a csv.
The Select-Object #{Name... line creates an expression alias, which concatenates surname+forename. You'll notice the extra 8 spaces we append to the end of the string so we know we will have at least 8 characters in the string.
The Group-Object {... line groups the username, using the first 8 characters, if available. The .Trim() gets rid of any trailing spaces.
The Select-Object #{Name='username'... line takes the Name field from the group-object and renames to username and also shows the count from the grouping operation.
The Format-Table -AutoSize line is purely for output formatting to the console and gives you an output like the one below.
username Count
-------- -----
smithb 1
smithbob 2
smithson 3
smithers 1
An amended version of the above code, which you can use on your real csv. Change the surname, firstname column names to suit your csv.
# you would use the code below, to import your list of names
# uncomment the `# -Header surname,firstname` bit if your csv has no headers
$users = Import-Csv -Path 'c:\path\to\names.csv' # -Header surname,firstname
$users |
Select-Object #{Name='username'; Expression={"$($_.surname)$($_.firstname) "}}, surname, firstname |
Group-Object { $_.username.Substring(0,8).Trim() } |
Select-Object #{Name='username'; Expression={$_.Name}}, Count

Reformat column names in a csv with PowerShell

Question
How do I reformat an unknown CSV column name according to a formula or subroutine (e.g. rename column " Arbitrary Column Name " to "Arbitrary Column Name" by running a trim or regex or something) while maintaining data?
Goal
I'm trying to more or less sanitize columns (the names) in a hand-produced (or at least hand-edited) csv file that needs to be processed by an existing PowerShell script. In this specific case, the columns have spaces that would be removed by a call to [String]::Trim(), or which could be ignored with an appropriate regex, but I can't figure a way to call or use those techniques when importing or processing a CSV.
Short Background
Most files and columns have historically been entered into the CSV properly, but recently a few columns were being dropped during processing; I determined it was because the files contained a space (e.g., Select-Object was being told to get "RFC", but Import-CSV retrieved "RFC ", so no matchy-matchy). Telling the customer to enter it correctly by hand (though preferred and much simpler) is not an option in this case.
Options considered
I could manually process the text of the file, but that is a messy and error prone way to re-invent the wheel. I wonder if there's a syntax with Select-Object that would allow a softer match for column names, but I can't find that info.
The closest I have come conceptually is using a calculated property in the call to Select-Object to rename the column, but I can only find ways to rename a known column to another known column. So, this would require enumerating the columns and matching them exactly (preferred) or a softer match (like comparing after trimming or matching via regex as a fallback) with expected column names, then creating a collection of name mappings to use in constructing calculated properties from that information to select into a new object.
That seems like it would work, but more it's work than I'd prefer, and I can't help but hope that there's a simpler way I haven't been able to find via Google. Maybe I should try Bing?
Sample File
Let's say you have a file.csv like this:
" RFC "
"1"
"2"
"3"
Code
Now try to run the following:
$CSV = Get-Content file.csv -First 2 | ConvertFrom-Csv
$FixedHeaders = $CSV.PSObject.Properties.Name.Trim(' ')
Import-Csv file.csv -Header $FixedHeaders |
Select-Object -Skip 1 -Property RFC
Output
You will get this output:
RFC
---
1
2
3
Explanation
First we use Get-Content with parameter -First 2 to get the first two lines. Piping to ConvertFrom-Csv will allow us to access the headers with PSObject.Properties.Name. Use Import-Csv with the -Header parameter to use the trimmed headers. Pipe to Select-Object and use -Skip 1 to skip the original headers.
I'm not sure about comparisons in terms of efficiency, but I think this is a little more hardened, and imports the CSV only once. You might be able to use #lahell's approach and Get-Content -raw, but this was done and it works, so I'm gonna leave it to the community to determine which is better...
#import the CSV
$rawCSV = Import-Csv $Path
#get actual header names and map to their reformatted versions
$CSVColumns = #{}
$rawCSV |
Get-Member |
Where-Object {$_.MemberType -eq "NoteProperty"} |
Select-Object -ExpandProperty Name |
Foreach-Object {
#add a mapping to the original from a trimmed and whitespace-reduced version of the original
$CSVColumns.Add(($_.Trim() -replace '(\s)\s+', '$1'), "$_")
}
#Create the array of names and calculated properties to pass to Select-Object
$SelectColumns = #()
$CSVColumns.GetEnumerator() |
Foreach-Object {
$SelectColumns += {
if ($CSVColumns.values -contains $_.key) {$_.key}
else { #{Name = $_.key; Expression = $CSVColumns[$_.key]} }
}
}
$FormattedCSV = $rawCSV |
Select-Object $SelectColumns
This was hand-copied to a computer where I don't have the rights to run it, so there might be an error - I tried to copy it correctly
You can use gocsv https://github.com/DataFoxCo/gocsv to see the headers of the csv, you can then rename the headers, behead the file, swap columns, join, merge, any number of transformations you want

looping through a csv

just wondering if i could do this in powershell, or even a c#/vb.net command line program.
I have data that looks like this:
(source: kalleload.net)
I have a Teams Table. It looks like this:
| id | teamname | teamcity |
so for example, C2 has the value "Atlanta Braves". I need to split this up into "Atlanta" and "Braves". Data is consistent. for example "New York Mets" is actually "NewYork Mets".
So i need to go through column C and D and insert all the teams (no duplicates into the db).
One line of PowerShell will read in the CSV file and create a custom object for each home and away team listing (with a property for the city name and for the team name). The last command in the pipeline will eliminate the duplicates.
$TeamsAndCities = import-csv -path c:\mycsvfile.csv | foreach-object { $_.away, $_.home | select-object #{Name='City';Expression={$_.split(' ')[0]}}, #{Name='Team';Expression={$_.split(' ')[1]}} } | select-object -unique
You can do database access from PowerShell as well, but that might be suited to a new question with some more details about the database you are connecting to.
I rarely code in VBA/VB but...
Something like
Dim rngAwayTeam As Range, rngHomeTeam As Range
set rngAwayTeam = Worksheets("YourWorksheet").Range("C2")
set rngHomeTeam = Worksheets("YourWorksheet").Range("D2")
Dim rowOffset As Integer
rowOffset = 1
Do While (rngAwayTeam.Offset(rowOffset,1).Text <> "")
'Do something with rngAwayTeam.Offset(rowOffset,1).Text
'and rngHomeTeam.Offset(rowOffset,1).Text
rowOffset = rowOffset + 1
Loop
There are other ways I'm sure, but, here is what I would do.
Yes that is an excel macro. Again, I rarely use VBA or .Net, just trying to help you out the best I can. You could just use a C# COM object for the database side of things. (Still new, can't comment.)
You can do it in C# console application quite easily.
All you have to do is loop through each line in the file, adding it to an array using split on the comma (,).
Then you can use your array to display the values or retrieve a specific value on a row.