How can I add string and create new column in my csv file using PowerShell - powershell

In my existing CSV file I have a column called "SharePoint ID" and it look like this
1.ylkbq
2.KlMNO
3.
4.MSTeam
6.
7.MSTEAM
8.LMNO83
and I'm just wondering how can I create a new Column in my CSV call "SharePoint Email" and then add "#gmail.com" to only the actual Id like "ylkbq", "KLMNO" and "LMNO83" instead of applying to all even in the blank space. And Maybe not add/transfer "MSTEAM" to the new Column since it's not an Id.
$file = "C:\AuditLogSearch\New folder\OriginalFile.csv"
$file2 = "C:\AuditLogSearch\New folder\newFile23.csv"
$add = "#GMAIL.COM"
$properties = #{
Name = 'Sharepoint Email'
Expression = {
switch -Regex ($_.'SharePoint ID') {
#Not sure what to do here
}
}
}, '*'
Import-Csv -Path $file |
Select-Object $properties |
Export-Csv $file2 -NoTypeInformation

Using calculated properties with Select-Object this is how it could look:
$add = "#GMAIL.COM"
$expression = {
switch($_.'SharePoint ID')
{
{[string]::IsNullOrWhiteSpace($_) -or $_ -match 'MSTeam'}
{
# Null value or mathces MSTeam, leave this Null
break
}
Default # We can assume these are IDs, append $add
{
$_.Trim() + $add
}
}
}
Import-Csv $file | Select-Object *, #{
Name = 'SharePoint Email'
Expression = $expression
} | Export-Csv $file2 -NoTypeInformation
Sample Output
Index SharePoint ID SharePoint Email
----- ------------- ----------------
1 ylkbq ylkbq#GMAIL.COM
2 KlMNO KlMNO#GMAIL.COM
3
4 MSTeam
5
6 MSTEAM
7 LMNO83 LMNO83#GMAIL.COM
A more concise expression, since I misread the point, it can be reduced to just one if statement:
$expression = {
if(-not [string]::IsNullOrWhiteSpace($_.'SharePoint ID') -and $_ -notmatch 'MSTeam')
{
$_.'SharePoint ID'.Trim() + $add
}
}

Related

Trying to extract specific text and merge output with existing output

I want to extract text from a .txt file. The way the file is layed out is in this format (below first block). Optimally, I would like for the powershell script to take the content of username and votecount and output them side by side. With an integer of 25>= add the letter D beside it. With the output adding itself to a pre-existing output file. Say this week is week 1. And testuser voted 25 times. They should have the output "testuser" 25D. But say in week 2 they voted 24 times. Then it should be "testuser" 49D. However say they had 25 again. Output should then be "testuser" 50DD or 50D2?.. I have what I think should work as an initial baseline for the script which in itself doesn't work.. But combining an output with a pre existing output is beyond my capability. This needs to parse an entire txt file of some 100+ people. So imagine there's like an extra 100 users..
{
"username": "testuser",
"votecount": "42",
"votesclaimed": "0",
"lastvotetime": "2022-11-04 09:08:29",
"steamid": "00000000000000000000"
}
Below is what I am working with.
Get-Content -Raw C:\Users\--------\Desktop\votes.txt |
ConvertFrom-txt |
ForEach-Object {
[pscustomobject] #{
UserName = $_.username
VoteCount = '{0}{1}' -f $_.votecount, ('', 'D')[[int] $_.votecount -gt 25]
}
} |
Export-Csv -NoTypeInformation -Encoding utf8 C:\Users\---------\Desktop\outvotes.csv
Try following :
$match = Select-String -Path "c:\temp\test.txt" -Pattern '^\s*"(?<key>[^"]+)"\s*:\s*"(?<value>[^"]+)'
$table = [System.Collections.ArrayList]::new()
foreach( $row in $match.Matches )
{
$key = $row.Groups["key"].Value
$value = $row.Groups["value"].Value
if($key -eq "username") {
$newRow = New-Object -TypeName psobject
$table.Add($newRow) | Out-Null
}
$newRow | Add-Member -NotePropertyName $key -NotePropertyValue $value
}
$table | Format-Table
$groups = $table | Group-Object {$_.username}

Add a calculated field to an imported csv based on datetime

Could someone please assist with this one.
My current code imports a csv file which has three columns, so far it will update the column names to be more readable. I need to add a fourth column which is a calculated field based on a datetime field.
So need to check the datetime field then display a number of days before it is 90 days old.
e.g. "Today's date" - "03/03/2020 8:00:00 AM" = 31 days
90 days - 31 days = 59 days (the 59 days is to go into the calculated field column
Bit of a newb with powershell and have all other functions working, but this is what I'm left with and need to add it into the below call, when the csv is imported, header columns updated then exported to a new file.
$input = "C:\Data\test\unchanged101.csv"
$output = "C:\Data\test\unchanged101conv.csv"
$checkDate = (Get-Date).AddDays(-90)
$data = Import-Csv $input |
Where-Object {
($_."pwdlastset" -as [DateTime]) -lt $CheckDate
}
$headerConversion = #(
#{ Name = 'User account'; Expression = { $_.'cn' } }
#{ Name = 'Last modified date'; Expression = { $_.'pwdlastset' } }
#{ Name = 'Email address'; Expression = { $_.'mail' } }
)
(Import-Csv -Path $input) |
Select-Object -Property $headerConversion | Select-Object *,"Days Left" |
Export-Csv -Path $output -NoTypeInformation
The new column is the "Days Left" where I need to display the number of days left until it is 90 days old. How to I get the result from the code here, into that column for each row?
$checkDate = (Get-Date).AddDays(-90)
$data = Import-Csv $input |
Where-Object {
($_."pwdlastset" -as [DateTime]) -lt $CheckDate
}
Been working on this one for the past few days and just cant figure the last part out.
First of all, you should not use variable name $input as this is an Automatic variable
If you subtract one datetime object from another, the result is a TimeSpan object which has a property called Days you could use
$inputFile = "C:\Data\test\unchanged101.csv"
$outputFile = "C:\Data\test\unchanged101conv.csv"
$checkDate = (Get-Date).AddDays(-90).Date # .Date sets this to midnight
$result = Import-Csv $inputFile |
Where-Object { [DateTime]$_.pwdlastset -lt $CheckDate } |
Select-Object #{ Name = 'User account'; Expression = { $_.cn } },
#{ Name = 'Last modified date'; Expression = { $_.pwdlastset } },
#{ Name = 'Email address'; Expression = { $_.mail } },
#{ Name = 'Days Left'; Expression = { ($checkDate - [DateTime]$_.pwdlastset).Days } }
# output on screen
$result | Format-Table -AutoSize
# output to csv
$result | Export-Csv -Path $outputFile -NoTypeInformation
I would first add the new column and then go trough all the lines (with a foreach loop) check the remaining days and write them to the new "Days Left" column.
Note: I omitted your header conversion here. You need to run it first and then the "Days Left" code...
$data = Import-Csv $input #import everything
foreach($line in $data){
$daysLeft = 0
$daysSinceLastSet = ((Get-Date) - [DateTime]$line.pwdlastset).Days
if ($daysSinceLastSet -lt 90){
$daysLeft = 90-$daysSinceLastSet
}
$line."Days Left" = $daysLeft
}
$data | Export-Csv -Path $output -NoTypeInformation

Find out Text data in CSV File Numeric Columns in Powershell

I am very new in powershell.
I am trying to validate my CSV file by finding out if there is any text value in my numeric fields. I can define with columns are numeric.
This is my source data like this
ColA ColB ColC ColD
23 23 ff 100
2.30E+01 34 2.40E+01 23
df 33 ss df
34 35 36 37
I need output something like this (only text values if found in any column)
ColA ColC ColD
2.30E+01 ff df
df 2.40E+01
ss
I have tried some code but not getting any results, get only some output like as under
System.Object[]
---------------
xxx fff' ddd 3.54E+03
...
This is what I was trying
#
cls
function Is-Numeric ($Value) {
return $Value -match "^[\d\.]+$"
}
$arrResult = #()
$arraycol = #()
$FileCol = #("ColA","ColB","ColC","ColD")
$dif_file_path = "C:\Users\$env:username\desktop\f2.csv"
#Importing CSVs
$dif_file = Import-Csv -Path $dif_file_path -Delimiter ","
############## Test Datatype (Is-Numeric)##########
foreach($col in $FileCol)
{
foreach ($line in $dif_file) {
$val = $line.$col
$isnum = Is-Numeric($val)
if ($isnum -eq $false) {
$arrResult += $line.$col
$arraycol += $col
}
}
}
[pscustomobject]#{$arraycol = "$arrResult"}| out-file "C:\Users\$env:username\Desktop\Errors1.csv"
####################
can someone guide me right direction?
Thanks
You can try something like this,
function Is-Numeric ($Value) {
return $Value -match "^[\d\.]+$"
}
$dif_file_path = "C:\Users\$env:username\desktop\f2.csv"
#Importing CSVs
$dif_file = Import-Csv -Path $dif_file_path -Delimiter ","
#$columns = $dif_file | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name'
# Use this to specify certain columns
$columns = "ColB", "ColC", "ColD"
foreach($row in $dif_file) {
foreach ($col in $columns) {
if ($col -in $columns) {
if (!(Is-Numeric $row.$col)) {
$row.$col = ""
}
}
}
}
$dif_file | Export-Csv C:\temp\formatted.txt
Look up name of columns as you go
Look up values of each col in each row and if it is not numeric, change to ""
Exported updated file.
I think not displaying columns that have no data creates the challenge here. You can do the following:
$csv = Import-Csv "C:\Users\$env:username\desktop\f2.csv"
$finalprops = [collections.generic.list[string]]#()
$out = foreach ($line in $csv) {
$props = $line.psobject.properties | Where {$_.Value -notmatch '^[\d\.]+$'} |
Select-Object -Expand Name
$props | Where {$_ -notin $finalprops} | Foreach-Object { $finalprops.add($_) }
if ($props) {
$line | Select $props
}
$out | Select-Object ($finalprops | Sort)
Given the nature of Format-Table or tabular output, you only see the properties of the first object in the collection. So if object1 has ColA only, but object2 has ColA and ColB, you only see ColA.
The output order you want is quite different than the input CSV; you're tracking bad text data not by first occurrence, but by column order, which requires some extra steps.
test.csv file contents:
ColA,ColB,ColC,ColD
23,23,ff,100
2.30E+01,34,2.40E+01,23
df,33,ss,df
34,35,36,37
Sample code tested to meet your description:
$csvIn = Import-Csv "$PSScriptRoot\test.csv";
# create working data set with headers in same order as input file
$data = [ordered]#{};
$csvIn[0].PSObject.Properties | foreach {
$data.Add($_.Name, (New-Object System.Collections.ArrayList));
};
# add fields with text data
$csvIn | foreach {
$_.PSObject.Properties | foreach {
if ($_.Value -notmatch '^-?[\d\.]+$') {
$null = $data[$_.Name].Add($_.Value);
}
}
}
$removes = #(); # remove `good` columns with numeric data
$rowCount = 0; # column with most bad values
$data.GetEnumerator() | foreach {
$badCount = $_.Value.Count;
if ($badCount -eq 0) { $removes += $_.Key; }
if ($badCount -gt $rowCount) { $rowCount = $badCount; }
}
$removes | foreach { $data.Remove($_); }
0..($rowCount - 1) | foreach {
$h = [ordered]#{};
foreach ($key in $data.Keys) {
$h.Add($key, $data[$key][$_]);
}
[PSCustomObject]$h;
} |
Export-Csv -NoTypeInformation -Path "$PSScriptRoot\text-data.csv";
output file contents:
"ColA","ColC","ColD"
"2.30E+01","ff","df"
"df","2.40E+01",
,"ss",
#Jawad, Finally I have tried
function Is-Numeric ($Value) {
return $Value -match "^[\d\.]+$"
}
$arrResult = #()
$columns = "ColA","ColB","ColC","ColD"
$dif_file_path = "C:\Users\$env:username\desktop\f1.csv"
$dif_file = Import-Csv -Path $dif_file_path -Delimiter "," |select $columns
$columns = $dif_file | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name'
foreach($row in $dif_file) {
foreach ($col in $columns) {
$val = $row.$col
$isnum = Is-Numeric($val)
if ($isnum -eq $false) {
$arrResult += $col+ " " +$row.$col
}}}
$arrResult | out-file "C:\Users\$env:username\desktop\Errordata.csv"
I get correct result in my out file, order is very ambiguous like
ColA ss
ColB 5.74E+03
ColA ss
ColC rrr
ColB 3.54E+03
ColD ss
ColB 8.31E+03
ColD cc
any idea to get proper format? thanks
Note: with your suggested code, I get complete source file with all data , not the specific error data.

Combining like objects in an array

I am attempting to analyze a group of text files (MSFTP logs) and do counts of IP addresses that have submitted bad credentials. I think I have it worked out except I don't think that the array is passing to/from the function correctly. As a result, I get duplicate entries if the same IP appears in multiple log files. What am I doing wrong?
Function LogBadAttempt($FTPLog,$BadPassesArray)
{
$BadPassEx="PASS - 530"
Foreach($Line in $FTPLog)
{
if ($Line -match $BadPassEx)
{
$IP=($Line.Split(' '))[1]
if($BadPassesArray.IP -contains $IP)
{
$CurrentIP=$BadPassesArray | Where-Object {$_.IP -like $IP}
[int]$CurrentCount=$CurrentIP.Count
$CurrentCount++
$CurrentIP.Count=$CurrentCount
}else{
$info=#{"IP"=$IP;"Count"='1'}
$BadPass=New-Object -TypeName PSObject -Property $info
$BadPassesArray += $BadPass
}
}
}
return $BadPassesArray
}
$BadPassesArray=#()
$FTPLogs = Get-Childitem \\ftpserver\MSFTPSVC1\test
$Result = ForEach ($LogFile in $FTPLogs)
{
$FTPLog=Get-Content ($LogFile.fullname)
LogBadAttempt $FTPLog
}
$Result | Export-csv C:\Temp\test.csv -NoTypeInformation
The result looks like...
Count IP
7 209.59.17.20
20 209.240.83.135
18441 209.59.17.20
13059 200.29.3.98
and would like it to combine the entries for 209.59.17.20
You're making this way too complicated. Process the files in a pipeline and use a hashtable to count the occurrences of each IP address:
$BadPasswords = #{}
Get-ChildItem '\\ftpserver\MSFTPSVC1\test' | Get-Content | ? {
$_ -like '*PASS - 530*'
} | % {
$ip = ($_ -split ' ')[1]
$BadPasswords[$ip]++
}
$BadPasswords.GetEnumerator() |
select #{n='IP';e={$_.Name}}, #{n='Count';e={$_.Value}} |
Export-Csv 'C:\Temp\test.csv' -NoType

compare two csv using powershell and return matching and non-matching values

I have two csv files, i want to check the users in username.csv matches with userdata.csv copy
to output.csv. If it does not match return the name alone in the output.csv
For Ex: User Data contains 3 columns
UserName,column1,column2
Hari,abc,123
Raj,bca,789
Max,ghi,123
Arul,987,thr
Prasad,bxa,324
username.csv contains usernames
Hari
Rajesh
Output.csv should contain
Hari,abc,123
Rajesh,NA,NA
How to achieve this. Thanks
Sorry for that.
$Path = "C:\PowerShell"
$UserList = Import-Csv -Path "$($path)\UserName.csv"
$UserData = Import-Csv -Path "$($path)\UserData.csv"
foreach ($User in $UserList)
{
ForEach ($Data in $UserData)
{
If($User.Username -eq $Data.UserName)
{
# Process the data
$Data
}
}
}
This returns only matching values. I also need to add the non-matching values in output
file. Thanks.
something like this will work:
$Path = "C:\PowerShell"
$UserList = Import-Csv -Path "$($path)\UserName.csv"
$UserData = Import-Csv -Path "$($path)\UserData.csv"
$UserOutput = #()
ForEach ($name in $UserList)
{
$userMatch = $UserData | where {$_.UserName -eq $name.usernames}
If($userMatch)
{
# Process the data
$UserOutput += New-Object PsObject -Property #{UserName =$name.usernames;column1 =$userMatch.column1;column2 =$userMatch.column2}
}
else
{
$UserOutput += New-Object PsObject -Property #{UserName =$name.usernames;column1 ="NA";column2 ="NA"}
}
}
$UserOutput | ft
It loops through each name in the user list. Line 9 does a search of the userdata CSV for a matching user name if it finds it it adds the user data for that user to the output if no match is found it adds the user name to the output with NA in both columns.
had to change your userList csv:
usernames
Hari
Rajesh
expected output:
UserName column1 column2
-------- ------- -------
Hari abc 123
Rajesh NA NA
I had a similar situation, where I needed a "changed record collection" holding the entire record when the current record was either new or had any changes when compared to the previous record. This was my code:
# get current and previous CSV
$current = Import-Csv -Path $current_file
$previous = Import-Csv -Path $previous_file
# collection with new or changed records
$deltaCollection = New-Object Collections.Generic.List[System.Object]
:forEachCurrent foreach ($row in $current) {
$previousRecord = $previous.Where( { $_.Id -eq $row.Id } )
$hasPreviousRecord = ($null -ne $previousRecord -and $previousRecord.Count -eq 1)
if ($hasPreviousRecord -eq $false) {
$deltaCollection.Add($current)
continue forEachCurrent
}
# check if value of any property is changed when compared to the previous
:forEachCurrentProperty foreach ($property in $current.PSObject.Properties) {
$columnName = $property.Name
$currentValue = if ($null -eq $property.Value) { "" } else { $property.Value }
$previousValue = if ($hasPreviousRecord) { $previousRecord[0]."$columnName" } else { "" }
if ($currentValue -ne $previousValue -or $hasPreviousRecord -eq $false) {
$deltaCollection.Add($currentCenter)
continue forEachCurrentProperty
}
}
}