I have a large csv file with 41000 rows of data. Within this I have to go through and inspect each row for a particular string PRCXX and if the row contains that in column 6 then I take the value of column 10 and replace the value in column 9 with it.
I have code that works however it takes a very long time to parse through line by line. I am looking for some help to try and optimize it. I have tried switching to a ForEach loop however I am not sure how exactly to get it to work with what I am trying to accomplish and haven't been able to find any examples to work from.
Here is the code that I have that currently is working just takes along time to complete.
Import-Csv $TransFile -Header 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19 | ForEach-Object {
if ($_.6 -match "PRCXX") {
$_.9 = $_.10
}
$_ | Export-Csv test2.csv -NoTypeInformation -Append -Delimiter ","
}
Thanks for any help that you can provide.
Thanks to JosefZ this is fixed. Moved the Export to occur outside the loop and that cleared it up.
Import-Csv $TransFile -Header 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19 | ForEach-Object {
if ($_.6 -match "PRCXX") {
$_.9 = $_.10
}
$_
} | Export-Csv test2.csv -NoTypeInformation -Append -Delimiter ","
I have a directory on a server called 'servername'. In that directory, I have subdirectories whose name is a date. In those date directories, I have about 150 .csv file audit logs.
I have a partially working script that starts from inside the date directory, enumerates and loops through the .csv's and searches for a string in a column. Im trying to get it to export the row for each match then go on to the next file.
$files = Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\20180525'
ForEach ($file in $files) {
$Result = If (import-csv $file.FullName | Where {$_.'path/from' -like "*01May18.xlsx*"})
{
$result | Export-CSV -Path c:\temp\output.csv -Append}
}
What I am doing is searching the 'path\from' column for a string - like a file name. The column contains data that is always some form of \folder\folder\folder\filename.xls. I am searching for a specific filename and for all instances of that file name in that column in that file.
My issue is getting that row exported - export.csv is always empty. Id also like to start a directory 'up' and go through each date directory, parse, export, then go on to the next directory and files.
If I break it down to just one file and get it out of the IF it seems to give me a result so I think im getting something wrong in the IF or For-each but apparently thats above my paygrade - cant figure it out....
Thanks in advance for any assistance,
RichardX
The issue is your If block, when you say $Result = If () {$Result | ...} you are saying that the new $Result is equal what's returned from the if statement. Since $Result hasn't been defined yet, this is $Result = If () {$null | ...} which is why you are getting a blank line.
The If block isn't even needed. you filter your csv with Where-Object already, just keep passing those objects down the pipeline to the export.
Since it sounds like you are just running this against all the child folders of the parent, sounds like you could just use the -Recurse parameter of Get-ChildItem
Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\' -Recurse |
ForEach-Object {
Import-csv $_.FullName |
Where-Object {$_.'path/from' -like "*01May18.xlsx*"}
} | Export-CSV -Path c:\temp\output.csv
(I used a ForEach-Object loop rather than foreach just demonstrate objects being passed down the pipeline in another way)
Edit: Removed append per Bill_Stewart's suggestion. Will write out all entries for the the recursed folders in the run. Will overwrite on next run.
I don't see a need for appending the CSV file? How about:
Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\20180525' | ForEach-Object {
Import-Csv $_.FullName | Where-Object { $_.'path/from' -like '*01May18.xlsx*' }
} | Export-Csv 'C:\Temp\Output.csv' -NoTypeInformation
Assuming your CSVs are in the same format and that your search text is not likely to be present in any other columns you could use a Select-String instead of Import-Csv. So instead of converting string to object and back to string again, you can just process as strings. You would need to add an additional line to fake the header row, something like this:
$files = Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\20180525'
$result = #()
$result += Get-Content $files[0] -TotalCount 1
$result += ($files | Select-String -Pattern '01May18\.xlsx').Line
$result | Out-File 'c:\temp\output.csv'
I will do my best to break this down as simply as I can.
what I have so far that is working:
Currently I have two csv files...
test1.csv
test1ColumnN,test1ColumnI,test1ColumnD,selectDomainOne,selectDomainTwo,selectDomainThree
asdf,asdf,asdf,,,
nValue1,iValue1,dValue1,sValue1,,
qwer,asdf,zxcv,,,
nValue2,iValue2,dValue2,,,
qwer,zxcv,asdf,lkjh,,
nValue3,iValue3,dValue3,sValue3,,
zxcv,qwer,asdf,,poiu,
nValue1,iValue1,dValue1,,sValue1,
nValue4,iValue4,dValue4,,sValue4,
asdf,qwer,zxcv,fghj,mnbv,
nValue5,iValue5,dValue5,,,
asdf,cvbn,erty,,,uytr
nValue7,iValue7,dValue7,,,sValue7
nValue8,iValue8,dValue8,,,sValue8
nValue9,iValue9,dValue9,,,sValue9
qwer,asdf,zxcv,poiu,lkjh,mnbv
test2.csv
DomainCatagories,test2ColumnS,test2ColumnA,test2ColumnN,test2ColumnI,test2ColumnD
DomainOne,sValue1,aValue1,nValue1,,dValueN
DomainOne,sValue2,aValue2,,iValue2,dValue2
DomainOne,sValue3,aValue2,nValue3,iValue3,dValue3
DomainTwo,sValue1,aValue2,,iValue1,dValueN
DomainTwo,sValue4,aValue1,nValue4,,dValueN
DomainTwo,sValue5,aValue1,nValue5,iValue5,dValue5
DomainThree,sValue7,aValue2,nValue7,iValue7,dValue7
DomainThree,sValue8,aValue1,nValue8,iValue8,dValue8
DomainThree,sValue9,aValue2,nValue9,iValue9,dValue9
Now I want to add a column (inside test2.csv) to match the sValue# from both test1.csv and test2.csv with the condition of ($_.DomainCatagories='DomainOne' from test2.csv) and ($_.selectDomainOne from test1.csv)
To do this, I am using the following code...
#Create Column
$domainNameOne = #{}
$domainNameOne = Import-Csv 'C:\Scripts\Tests\test1.csv' | Where-Object {$_.selectDomainOne} | Select-Object -Expand 'selectDomainOne'
(Import-Csv 'C:\Scripts\Tests\test2.csv') |
Select-Object -Property *, #{n='Test1sValues';e={
if($_.DomainCatagories -eq 'DomainOne'){
if(($domainNameOne -contains $_.test2ColumnS) -and ($_.test2ColumnS)){
$_.test2ColumnS
} Else {
'Not found in test1'
}}}} | Export-Csv "C:\Scripts\Tests\test2-Temp" -NoType
Move-Item "C:\Scripts\Tests\test2-Temp" 'C:\Scripts\Tests\test2.csv' -Force
After the code is run, I get the following test2.csv (isCorrect)...
"DomainCatagories","test2ColumnS","test2ColumnA","test2ColumnN","test2ColumnI","test2ColumnD","Test1sValues"
"DomainOne","sValue1","aValue1","nValue1","","dValueN","sValue1"
"DomainOne","sValue2","aValue2","","iValue2","dValue2","Not found in test1"
"DomainOne","sValue3","aValue2","nValue3","iValue3","dValue3","sValue3"
"DomainTwo","sValue1","aValue2","","iValue1","dValueN",""
"DomainTwo","sValue4","aValue1","nValue4","","dValueN",""
"DomainTwo","sValue5","aValue1","nValue5","iValue5","dValue5",""
"DomainThree","sValue7","aValue2","nValue7","iValue7","dValue7",""
"DomainThree","sValue8","aValue1","nValue8","iValue8","dValue8",""
"DomainThree","sValue9","aValue2","nValue9","iValue9","dValue9",""
What I have that is not working:
Next I run the following code...
#Append Column
$domainNameThree = #{}
$domainNameThree = Import-Csv 'C:\Scripts\Tests\test1.csv' | Where-Object {$_.selectDomainThree} | Select-Object -Expand 'selectDomainThree'
(Import-Csv 'C:\Scripts\Tests\test2.csv') | % {
if($_.DomainCatagories -eq 'DomainThree'){
if(($domainNameThree -contains $_.test2ColumnS) -and ($_.test2ColumnS)){
$_.Test1sValues = $_.test2ColumnS
} Else {
$_.Test1sValues = 'Not found in test1'
}}} | Export-Csv "C:\Scripts\Tests\test2-Temp" -NoType
Move-Item "C:\Scripts\Tests\test2-Temp" 'C:\Scripts\Tests\test2.csv' -Force
Instead of adding the values in the correct rows, it completely blanks out the whole file and saves it as an empty file.
End Goal
What I want the code to produce, is this (notice values filled in on last 3 rows in the last column)...
"DomainCatagories","test2ColumnS","test2ColumnA","test2ColumnN","test2ColumnI","test2ColumnD","Test1sValues"
"DomainOne","sValue1","aValue1","nValue1","","dValueN","sValue1"
"DomainOne","sValue2","aValue2","","iValue2","dValue2","Not found in test1"
"DomainOne","sValue3","aValue2","nValue3","iValue3","dValue3","sValue3"
"DomainTwo","sValue1","aValue2","","iValue1","dValueN",""
"DomainTwo","sValue4","aValue1","nValue4","","dValueN",""
"DomainTwo","sValue5","aValue1","nValue5","iValue5","dValue5",""
"DomainThree","sValue7","aValue2","nValue7","iValue7","dValue7","sValue7"
"DomainThree","sValue8","aValue1","nValue8","iValue8","dValue8","sValue8"
"DomainThree","sValue9","aValue2","nValue9","iValue9","dValue9","sValue9"
What am I doing wrong in that 2nd code snippet?
The example you show from What I have that is not working: is missing a key portion. Export-Csv will take everything piped into it to populate the CSV but you are not providing any.
Problem is that you are not passing anything through the pipe. Merely just updating one property. The simplest thing to do is add $_ after the if statement. Or you could just use a calculated property which you have done before in another one of your questions. The example below from Compare dates with different formats in csv file even uses an if statement.
Import-Csv $csvFile | Select-Object *, #{n='MatchDates';e={ if((([datetime]$_.Date1).Date -eq $_.Date3) -and (([datetime]$_.Date2).Date -eq $_.Date3) -and (([datetime]$_.Date1).Date -eq $_.Date2)){ 'Match Found' }Else{ 'No Match Found' }}} |
Export-Csv "$csvFile-results.csv" -NoTypeInformation -Force
I need help to get the following PowerShell script to output too just one .CSV file instead of the current two. Can someone please help? I'm wanting to get the SNMP settings that are set on remote servers. The information I'm after is held in the registry. $DellAdmKey is one key that holds SubKeys and refers to the "communityNames". Within each of the SubKeys I get the "Value Data" which refers to the "TrapDestinations". $DellAdmKey2 is a Key called "ValidCommunities" but does not have Subkeys, it just has DWORD values that refer to "AcceptedCommunityNames" and the "Rights". So the foreach on the 1st line can't be used as "ValidCommunities" does not contain SubKeys
$servers = "ServerName"
$BaseKey = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey(‘LocalMachine’, $server)
$SubKey= $BaseKey.OpenSubKey("SYSTEM\\CurrentControlSet\\Services\\SNMP\Parameters",$true)
$DellAdmKey = $BaseKey.OpenSubKey("SYSTEM\\CurrentControlSet\\Services\\SNMP\\Parameters\\TrapConfiguration\\",$true)
$DellAdmKey2 = $BaseKey.OpenSubKey("SYSTEM\\CurrentControlSet\\Services\\SNMP\\Parameters\\ValidCommunities\\",$true)
$DellAdmKey.GetSubKeyNames() | foreach {
$action = $_
$subkey = $DellAdmKey.openSubKey($_)
$subkey.GetValueNames() | foreach {$_ | Select
#{name="ServerName";Expression={$server}},
#{Name="CommunityNames";Expression={$action}},
#{name="TrapDestinations";Expression={$subkey.getvalue($_)}}
} | Export-Csv c:\temp\1.csv -NoTypeInformation
}
$action = $_
$subkey = $DellAdmKey2.openSubKey($_)
$subkey.GetValueNames() | foreach {
$_ | Select #{name="ServerName";Expression={$server}},
#{name="AcceptedCommunityNames";Expression={$_}},
#{name="Rights";Expression={$subkey.getvalue($_)}}
} | Export-Csv c:\temp\2.csv -NoTypeInformation
If I understand your question (and your code) correctly, the third group of instructions is supposed to go inside the $DellAdmKey.GetSubKeyNames() | foreach { ... }. In that case you simply need to move the | Export-Csv outside that loop to capture all output in one CSV. You need to make sure all objects have the same set of properties, though.
$DellAdmKey.GetSubKeyNames() | foreach {
$action = $_
$subkey = $DellAdmKey.openSubKey($action)
$subkey.GetValueNames() | Select #{name="ServerName";Expression={$server}},
#{Name="CommunityNames";Expression={$action}},
#{name="TrapDestinations";Expression={$subkey.getvalue($_)}}
#{name="AcceptedCommunityNames";Expression={}},
#{name="Rights";Expression={}}
$subkey = $DellAdmKey2.openSubKey($action)
$subkey.GetValueNames() | Select #{name="ServerName";Expression={$server}},
#{Name="CommunityNames";Expression={}},
#{name="TrapDestinations";Expression={}},
#{name="AcceptedCommunityNames";Expression={$_}},
#{name="Rights";Expression={$subkey.getvalue($_)}}
} | Export-Csv 'c:\temp\out.csv' -NoTypeInformation
Edit: If you really just want to run the code as you posted (which seems somewhat dubious to me, since $_ in line 17 should be empty), you could change the second Export-Csv instruction to
... | Export-Csv 'c:\temp\1.csv' -Append -NoTypeInformation
provided you're running PowerShell v3. Prior to that something like this should do:
... | ConvertTo-Csv -NoTypeInformation | select -Skip 1 |
Out-File 'c:\temp\1.csv' -Append
Either way you should make sure all the objects you're exporting have the same set of properties, though. Otherwise you might end up with Rights values in the TrapDestination column or some such.
I have resorted to using this http://psrr.codeplex.com/ it make life easier. Thanks for the replies Ansgar.