I'm reading in a CSV file which contains 25,000 records, and am reading each column into a psobject. Here is what I have so far:
$file = Import-CSV .\server.csv
$tempobj = New-Object psobject -Property #{
'Name' = $file.Name
'Group' = $file.Group
}
When this is ran, I get the correct results I want, being that $file.Name contains all the server names, and $file.Group contains the groups for servers. However, my issue is that I need to edit the names of each server without interfering with the .Group. Here is an example of what a server name look like as is.
WindowsAuthServer #{wdk9870WIN}
I need to remove WindowsAuthServer #{ and WIN} from each server name, leaving only the server name left, or for this example, wdk9870.
I tried using the -replace function ($tempobj.Name -replace "WindowsAuthServer #{",""), but it requires that I save the results to a new array, which then messes up or removes .Group entirely
Is there a different way to go about doing this? I'm lost.
Suppose your server.csv looks like this:
"Name","Group"
"WindowsAuthServer #{wdk9870WIN}","Group1"
"WindowsAuthServer #{wdk9880WIN}","Group2"
"WindowsAuthServer #{wdk9890WIN}","Group1"
"WindowsAuthServer #{wdk9900WIN}","Group1"
And you want to change the values in the Name column only, then something like this would probably do it:
Import-Csv .\server.csv | ForEach-Object {
New-Object psobject -Property #{
'Name' = ([regex]'#\{(\w+)WIN\}').Match($_.Name).Groups[1].Value
'Group' = $_.Group
}
}
This will output:
Name Group
---- -----
wdk9870 Group1
wdk9880 Group2
wdk9890 Group1
wdk9900 Group1
If you want, you can simply pipe this info to the Export-Csv cmdlet to save as a new CSV file. For that, just append | Export-Csv -Path .\server_updated.csv -NoTypeInformation to the code.
Hope that helps
Related
I have 2 PScustomObjects that i would like to combine into a single csv as follows:
$Output1 = [PSCustomObject]#{
Timestamp1 = (Get-Date)
InstanceName1 = $ProcessName1
Count1 = $Processes1.Count
Memory1 = $MEMLoad1
CPU1 = $CPULoad1
}
$Output2 = [PSCustomObject]#{
Timestamp2 = (Get-Date)
InstanceName2 = $ProcessName2
Count2 = $Processes2.Count
Memory2 = $MEMLoad2
CPU2 = $CPULoad2
}
The CSV should have the titles TimeStamp1, InstanceName1......TimeStamp2, InstanceName2....
This code runs in a loop and exports data each pass.
Is there a way to do this? Also is there a way to do this dynamically to avoid replicating large amounts of code if i am to export data on say 100 PsCustomObjects, maybe lookping through the input data for the object and the putting in in one object and passing that to the csv while dynamically changing titles?
I use the following line to export. I've tried -InputObject $Output1, $Output2 but that gives gibberish.
Export-Csv -InputObject $Output1 -path $Path1 -NoTypeInformation -Append -Force
The only solution i have so far is to export multiple CSV's but that gets bulky fast.
Any help is greatly appreciated!
The best solution is to ensure all the objects your script/function outputs use the same schema (that is, they should all share the same common property names).
If that's not possible, you can use Select-Object to ensure all possible properties are selected for all objects:
$Output1,$Output2 |Select-Object Count1,Count2,CPU1,CPU2,InstanceName1,InstanceName2,Memory1,Memory2,Timestamp1,Timestamp2 |Export-Csv ...
This is obviously not very practical, so if you have many different object schemas, you'll want to automate the discovery of all possible property names.
To do that, we can inspect the psobject hidden memberset:
$allTheObjectsToExport = $Output1,$Output2
$propertyNames = $allTheObjectsToExport |ForEach-Object {
# Discover all the object's properties
$_.psobject.Properties |ForEach-Object {
# Get each property name
$_.Name
}
} |Sort-Object -Unique
# Now the CSV will have a column for every single unique property name
$allTheObjectsToExport |Select-Object $propertyNames |Export-Csv -Path $Path1 -NoTypeInformation
I'm needing a little help creating a PowerShell script that will compare two csv files, and identify the differences specific to each file. This will be used for Active Directory user management, and will either create or disable accounts based on which csv the users are in.
I need to use the Student ID column header for the comparison since that is the only common and unique attribute shared on the accounts.
StudentRoster.csv (reference file) - I need to see a list/export of Students from this file that are not in the ActiveDirectory.csv. I will use this list to create those users in AD.
ActiveDirectory.csv - I need to see a list/export of Students in this file that are not in the StudentRoster.csv. Users on this list will be disabled in AD.
Headers/Columns:
Student ID, Last Name, First Name, Grade
I'm using the below code, but I'm not sure how to tell it to flag the user as either create/disable using the SideIndicator, and how to export to the create or disable csv.
Compare-Object -ReferenceObject $(gc C:\users\Me\desktop\Test\StudentRoster.csv) -DifferenceObject $(gc C:\users\Me\desktop\Test\ActiveDirectory.csv) | Export-Csv C:\users\Me\desktop\test\export.csv
$referenceObject = Import-Csv C:\users\Me\desktop\Test\StudentRoster.csv
$differenceObject = Import-Csv C:\users\Me\desktop\Test\ActiveDirectory.csv
$compareParams = #{
ReferenceObject = $referenceObject
DifferenceObject = $differenceObject
Property = 'Student ID'
PassThru = $true
}
$comparison = Compare-Object #compareParams | Where-Object {$psitem.SideIndicator -eq '<='}
$comparison | Export-Csv C:\users\ardavis\desktop\export.csv -NoTypeInformation
You'll want to Import-Csv to an object with properties as headers, then Compare-Object on the property "Student ID" and PassThru to get all fields.
We filter with Where-Object to choose the items existing only in the left side of the comparison, which in this case is "StudentRoster.csv".
I have a large 10 million row file (currently CSV). I need to read through the file, and remove duplicate items based on multiple columns.
Example line of data would look something like:
ComputerName, IPAddress, MacAddress, CurrentDate, FirstSeenDate
I would want to check MacAddress and ComputerName for duplicates and if a duplicate is discovered keep the unique entry with the oldest FirstSeenDate.
I have read a CSV into a variable using import-csv and then processed the variable using sort-object...etc but it's horribly slow.
$data | Group-Object -Property ComputerName,MaAddress | ForEach-Object{$_.Group | Sort-Object -Property FirstSeenDate | Select-Object -First 1}
I am thinking I could use stream.reader and read the CSV line by line building a unique array based on array contains logic.
Thoughts?
I would probably use Python if performance were a major concern. Or LogParser.
However, if I had to use PowerShell, I would probably try something like this:
$CultureInfo = [CultureInfo]::InvariantCulture
$DateFormat = 'M/d/yyyy' # Use whatever date format is appropriate
# We need to convert the strings that represent dates. You can skip the ParseExact() calls if the dates are already in a string sortable format (e.g., yyyy-MM-dd).
$Data = Import-Csv $InputFile | Select-Object -Property ComputerName, IPAddress, MacAddress, #{n = 'CurrentDate'; e = {[DateTime]::ParseExact($_.CurrentDate, $DateFormat, $CultureInfo)}}, #{n = 'FirstSeenDate'; e = {[DateTime]::ParseExact($_.FirstSeenDate, $DateFormat, $CultureInfo)}}
$Results = #{}
foreach ($Record in $Data) {
$Key = $Record.ComputerName + ';' + $Record.MacAddress
if (!$Results.ContainsKey($Key)) {
$Results[$Key] = $Record
}
elseif ($Record.FirstSeenDate -lt $Results[$Key].FirstSeenDate) {
$Results[$Key] = $Record
}
}
$Results.Values | Sort-Object -Property ComputerName, MacAddress | Export-Csv $OutputFile -NoTypeInformation
That may very well be faster because Group-Object is often a bottleneck even though it is quite powerful.
If you really want to try using a stream reader, try using the Microsoft.VisualBasic.FileIO.TextFieldParser class, which is a part of the .Net framework in spite of it's slightly misleading name. You can access it by running Add-Type -AssemblyName Microsoft.VisualBasic.
You could do an import in a database (i.e. SQLite example )
and then query:
SELECT
MIN(FirstSeenDate) AS FirstSeenDate,
ComputerName,
IPAddress,
MacAddress
FROM importedData
GROUP BY ComputerName, IPAddress, MacAddress
I have a .csv file that contains well over 1000 entries of former employees of the company I work for, and I am attempting to split it up by employee into a format that I can insert into an email template. To exacerbate things, many (but not all) of the employees have more than one entry, and the .csv contains a considerable amount of information that is completely unnecessary for the people I will be sending the emails to. Ideally I would like to exclude these columns, but I am having quite a bit of difficulty with it. It seems to me that format-table will be the cmdlet I will be needing, but I am not 100% certain. Thus far I have tried:
import-csv C:\filepath\xx.csv | format-table -groupby (key value)
which does return the information organized in the way I would like, however I do not know how to split the information from there so I can only send the information for the specific employee. This also includes quite a bit of extraneous information, as previously mentioned. I have also tried:
import-csv C:\filepath\xx.csv | format-table -groupby (key value) -property (properties I want,separated by commas)
However, this is still returning the same extraneous information. I also tried using a foreach loop to iterate through the .csv, store the information I want in variables, then store those variables in an array, then pipe that array into format-table, which results in a mess. I have also tried putting the .csv into a hashtable grouped by the key value I want, however that results in the rest of the information being put into a string. I believe I should be able to split this string using regex, however I am not at all familiar with regex and have no idea where to start with that. I'm pretty well stumped at this point. Any help would be greatly appreciated.
EDIT
After following #4c74356b41's suggestion (for which I am very grateful), I am able to output a list of tables with only the pertinent information. However, I am still unable to split this list of tables into the individual tables I need. I currently have a .txt file which I am using as a template, which I would like to add the table for the individual users to. I have so far had success with using get-content to retrieve the contents of the .txt, -replace to replace several other fields on the template from information in the original .csv (eg name, manager, etc), and then out-file to store the edited template into a temp file, which is then saved as a draft to outlook. I attempted to add another foreach loop inside the previous foreach loop to add the tables into the template, but that is returning templates with the line 'Microsoft.PowerShell.Commands.Internal.Format.FormatStartData' in place of the table. It is also creating a draft for each entry rather than just one for each unique user id. Here is all the code I have so far:
$olFolderDrafts = 16
$ol = New-Object -comObject Outlook.Application
$ns = $ol.GetNameSpace("MAPI")
$file= import-csv H:\filepath\term_rep.csv
$data= import-csv H:\filepath\term_rep.csv | select-object "Inactive Emp Name","User ID","Loc","Equipment Descr","Tag Number","Serial Number"
$table= $data | Format-Table -GroupBy "User ID"
foreach($i in $file)
{$user = $i | select-object -expandproperty "Inactive Emp Name"
$uid= $i | select-object -expandproperty "User ID"
$manager = $i | Select-Object -expandproperty "Manager"
$loc= $i | select -ExpandProperty "Loc"
$tagnum= $i | select -ExpandProperty "Tag Number"
$sernum= $i | select -ExpandProperty "Serial Number"
$path="C:\filepath\term_email.txt"
$newpath = [system.io.path]::GetTempFileName()
(Get-content $path) -replace "USER_NAME",$user `
-replace "USER_ID",$uid | out-file $newpath
foreach($id in $table){
(Get-Content $newpath) -replace "EQUIPMENT_LIST", $id | out-file $newpath
}
$subject="Equipment for user $uid"
$body= get-content $newpath
$mail = $ol.CreateItem(0)
$mail.To = $manager
$mail.CC = $null
$mail.Subject = $subject
$mail.Body = $body -join "`n"
$mail.save()}
To reduce noise and duplicates I would find a column that is specific to a user, such as UserName, or Email. Then group by that, and select the first item in each group. Then you could pipe to Select to reduce noise from extraneous columns. Something like:
import-csv C:\filepath\xx.csv | Group UserName | ForEach{$_.Group[0]} | Select FirstName,LastName,Email,UserName,SeparationDate
Then you can do with it what you want... Pipe to another ForEach loop to work with each record one at a time, or pipe it to Export-CSV to generate a new CSV with reduced clutter that you could work with. You could even have PowerShell create the emails for you, depending on how fancy you want to get.
If you need further help please update your question with an example of what your desired output to be, or what exactly you are trying to accomplish with each entry.
Edit: Ok, so when it comes down to it you want a insert a table of things for any records matching a specific user into an email, so that you can get their stuff back from their manager after they're separated from the company. Cool, we can do that. To start off with, you're using the Outlook ComObject, and generating your email that way. Awesome, I've done it myself, and you can do some great stuff that way! I highly recommend using HTML here. It makes the email look more professional, and lets us insert a nice looking table into the email instead of some formatted text that will look funny due to spacing once its in the email.
So, let's back up from the issue just a hair, and redo your email template. Pop open MS Word and open your template. Now, go to File and Save As. Select 'Web Page, Filtered' as your document type, and name it the same thing you already had.
So, rather than loading the list twice, then exporting, and blah, blah, blah, we're going to shorten things a little. We load your CSV once. For that matter, we don't need to read the body of the email once for each user, so let's just load it up one time here before the loop as well. Then we pipe the User ID field to Select-Object using the -Unique switch, so that we only get one email generated per user. So, the script up to there looks like this:
$olFolderDrafts = 16
$ol = New-Object -comObject Outlook.Application
$ns = $ol.GetNameSpace("MAPI")
$file= import-csv H:\filepath\term_rep.csv
$path="C:\filepath\term_email.txt"
$body = (Get-content $path) -join "`n"
foreach($i in ($file.'User ID'|Select -Unique))
{
Now, there's no need to assign all of those variables like you were, so I'm skipping that part. So we'll find the first instance of the current user, and save that to a variable.
$User = $File|Where{$_.'User ID' -eq $i}|Select -First 1
Once we've got that we will find all of their records in the list, select only the properties that you are interested in, and convert the output to an HTML table using ConvertTo-HTML.
$Equip = $File|Where{$_.'User ID' -eq $i}|select-object "Inactive Emp Name","User ID","Loc","Equipment Descr","Tag Number","Serial Number"|ConvertTo-Html -Property '*' -As Table -Fragment
Now that is a pretty long line, but most of it is just selecting the right properties. The magic happens in the last bit where we tell it to convert the data to a HTML. Specifically we want everything (specified by -Property '*'), that we want it to be converted as a table, and that this is just a fragment, so it doesn't try to put all the preceding and following tags in there (so no <HTML> and </HTML> type tags since it's being injected into the middle of existing HTML.)
Then we do the replaces, make the email, assign the properties, yada, yada, yada, you pretty much already had all this...
$HTMLbody = $body -replace "USER_NAME",$user.'Inactive Emp Name' -replace "USER_ID",$user.'User ID' -replace "EQUIPMENT_LIST", $Equip
$subject="Equipment for user " + $User.'User ID'
$mail = $ol.CreateItem(0)
$mail.To = $User.Manager
$mail.Subject = $subject
$mail.HTMLBody = $HTMLbody
$mail.save()
}
Save it, and done! Loop to the next user, and repeat. So put that together and you get:
$olFolderDrafts = 16
$ol = New-Object -comObject Outlook.Application
$ns = $ol.GetNameSpace("MAPI")
$file= import-csv H:\filepath\term_rep.csv
$path="C:\temp\email.htm"
$body = (Get-content $path) -join "`n"
foreach($i in ($file.'User ID'|Select -Unique))
{
$User = $File|Where{$_.'User ID' -eq $i}|Select -First 1
$Equip = $File|Where{$_.'User ID' -eq $i}|select-object "Inactive Emp Name","User ID","Loc","Equipment Descr","Tag Number","Serial Number"|ConvertTo-Html -Property '*' -As Table -Fragment
$HTMLbody = $body -replace "USER_NAME",$user.'Inactive Emp Name' -replace "USER_ID",$user.'User ID' -replace "EQUIPMENT_LIST", $Equip
$subject="Equipment for user " + $User.'User ID'
$mail = $ol.CreateItem(0)
$mail.To = $User.Manager
$mail.Subject = $subject
$mail.HTMLBody = $HTMLbody
$mail.save()
}
Say your csv has some columns: name, surname, bla-bla, bla-bla, bla-bla.
$data = import-csv C:\filepath\xx.csv | select name, surname
$data | format-table -groupby (key value)
I have 2 csv files. Each have different headers and different number of columns, and have different number of entries.
Here are some examples of the first couple lines
CSV 1
ID,Last_Name,First_Name,Middle_Name,Email_Addr,Title,Gender
###1,smith,bill,p,smith#soso.com,boss,m
###2,smith2,billy,p,smith2#soso.com,someguy,m
CSV 2
ID,Name Id,Last Name,First Name,Middle Name,Gender
###2,ID1010,smith2,billy,p,M
I am trying to import them and compare the ID column. When a match is found I am wanting a new csv file with All info from CSV 1 and the matched Name ID from csv 2.
New CSV Example:
ID,Last_Name,First_Name,Middle_Name,Email_Addr,Title,Gender,Name Id
###1,smith,bill,p,smith#soso.com,boss,m,
###2,smith2,billy,p,smith2#soso.com,someguy,m,ID1010
Ive been looking and came across this Stackoverflow from about a year ago that seemed to be on the right track but I cant seem to get code modified for my needs. Here is what I have tried.
$csv1 = Import-Csv -Path C:\STAFF\test1sky.csv
$csv2 = Import-Csv -Path C:\STAFF\test1power.csv
ForEach($Record in $csv2){
$MatchedValue = (Compare-Object $csv1 $Record -Property "ID" -IncludeEqual -ExcludeDifferent -PassThru).value
$Record = Add-Member -InputObject $Record -Type NoteProperty -Name "Name Id" -Value $MatchedValue
}
$csv2|Export-Csv 'C:\STAFF\combined.csv' -NoTypeInformation
I get the correct header in the new file but I never get the Name ID values to come though.
Any idea where I went wrong? I maybe on the wrong path completely and there be a easier way, but I need to be able to do this nightly without user interaction. Any help is appreciated!!
Let's try to simplify this. Add the 'Name ID' field to all records in CSV1. Then loop through it, and get the matches, and update the field. Something like:
$CSV1 = C:\Path\To\File1.csv
$CSV2 = C:\Path\To\File2.csv
$CSV1|ForEach{$_|Add-Member 'Name ID' $Null}
ForEach($Record in $CSV1){
$Record.'Name ID' = $CSV2|Where{$_.ID -eq $Record.ID}|Select -Expand 'Name ID'
}
$CSV1 = import-csv C:\Path\To\File1.csv
$CSV2 = import-csv C:\Path\To\File2.csv
#adds a row named "Name ID" to the PS Object( the CSV Import)
$CSV1|ForEach{$_|Add-Member 'Name ID' $Null}
ForEach($Record in $CSV1){
#gets the value from CSV1 for comparing to CSV2
$NameValue=Record."Last_Name"
#gets the Power Shell Object from the CSV2 Import that matches the Name ID from $csv1
$Nameobject= $CSV2|Where-object "Last Name" -contains $Namevalue
#Sets the Field "Name ID" in the PS Object $CSV1 Record to the Name ID from $csv2
$record."Name ID" = $Nameobject."Name ID"
}
You can easily grab addtional fields by adding other references to the CSV1 File by manipulating the CSV2 PS Object.
$record."Middle Name" = $nameobject."Middle_Name"
Since you have the entire object in the for loop form $csv2 you can call any of its fields or manipulate them by using variables and " |select -Property "Value" Like this
$objlength = $nameobject |select "First_Name"
$objlength.length
but i prefer to call it directly from the object as the output looks cleaner like this
$nameobject."First_Name".length
The operation you are looking for is called a relational join. Sometimes it's called an inner join, and sometimes just a join. My knowledge of join comes from SQL, not from Powershell.
Here's a description of "Join-Object". It seems to be what you are looking for.
http://blogs.msdn.com/b/powershell/archive/2012/07/13/join-object.aspx