I am new to Powershell and have a quick question.
The output I am playing with is from the sysinternals.com autorunsc tool in CSV format.
Purpose of the script is to search the CSV for certain binary names.
The import goes as expected but I cannot get it write my filtered info
$IOCFiles = "es.dll", "null.sys"
$data = import-csv "D:\autoruns.csv" -header("Entry_Location","Entry","Enabled","Category","Description","Publisher","Image_Path","Launch_String","MD5","SHA-1","SHA-256")
foreach ($K in $data)
{
foreach($element in $IOCFiles)
{
if ($k.Image_Path -match $element)
{
$hits = $k.Entry + " " + $K.Publisher + " " + $K.Image_Path
$hits | export-csv -Path c:\HITS.csv -NoTypeInformation
}
}
}
The above doesn't generate the actual data in the CSV but info on length of the values.
There are a number of good examples of import/exports but I didn't see one with the columns on import. I have to use the column import method due to something similar I am playing with.
Thank you!
Powershell makes this sort of thing pretty easy:
Import-csv "D:\autoruns.csv" -header("Entry_Location","Entry","Enabled","Category" |
Where-Object{($IOCFiles -match $_.Image_Path) -ne $null} |
Select-Object Entry, Publisher, Image_Path |
Export-CSV -Path c:\HITS.csv -NoTypeInformation
I haven't got the time to test that right now so there may be some issues with it, but that is the gist of it.
Related
I'm creating a CSV file in powershell.
Right now my code is:
Add-content -Path $filePath -Value "$($variable.Property)"
This works fine for the most part EXCEPT if the property contained a comma ie. "test, organization".
When I open up the CSV, the comma is taken with it (which is what i want) causing a extra separation. How do save "test, organization" to one column?
Referring to the documentation for Export-CSV, you will need to use a different delimiter, like a semi-colon.
When you read the CSV you should specify the delimiter as well: Import-CSV.
Try to quote your properties:
Add-content -Path $filePath -Value "'$($variable.Property)'"
Or use one of the built-in CSV commands, which automatically quote all values:
$foo.Bar | Export-Csv -Path $filePath
$foo.Bar| ConvertTo-Csv | Out-File -Path $filePath
If you just want to avoid issues with commas, you can change the delimiter between fields:
$foo | Export-Csv -Path $filePath -Delimiter '|'
Here is an article on how to use out-file or add-member with some cells of the row having commas in the variable values and some not.
https://imjustanengineer.blogspot.com/2022/01/so-youre-trying-to-use-powershell-out.html
Here is a code snippet, a more detailed explanation with a full working function is in the link. $outputArr is an array of all the lines of csv data you want to write to the csv. The loop checks each line to see if it contains commas inside of the individual cell entries and puts quotes around that entry if it does. If it does not, no adjustment is necessary and then a new array is appended to afterwards.
$index = 0;
foreach ($outputTemp in $outputArr)
{
if ($outputTemp.ToString().Contains(","))
{
$output += "`"$outputTemp`",";
}
else
{
$output += $outputTemp + ",";
}
$index++;
if ($index -eq $outputArr.Count)
{
if ($output.EndsWith(","))
{
$output = $output.Remove($output.Length - 1);
}
}
}
I found a diabolically simply answer after I opened a csv file in Excel and added text and commas to one column. When I saved, closed and reopened, the column still had all the words and commas properly formatted. So, then I opened the file in notepad++ and this is what I found:
column1text, column2text,"column3,text,with,commas"
In case it's not clear, and it took me a fair bit to recognize the little detail that makes all the difference, the opening double quote cannot have a space after the preceding comma.
column1text, column2text, "column3,text,with,commas"
splits all the words into separate columns because there is a space between
column2text, "column3,etc"
Take that space away
column2text,"column3,etc"
and everything within the double quotes stays in one column.
Example using active directory distinguishedName such as CN=somename,OU=Domain Controllers,DC=foo,DC=bar
$computers = get-adcomputer -filter *
foreach ($computer in $computers) {
$deviceName = $computer.Name
$dn = '"' + $computer.DistinguishedName + '"'
$guid = $computer.objectGUID
$lastLogon = $computer.LastLogonDate
$serialNumber = $computer.serialNumber
$whenCreated = $computer.whenCreated
"$guid, $lastLogon, $deviceName, $serialNumber, $whenCreated,$dn" | add-content "c:\temp\filename.csv"
}
It does not work if a space is added between $whenCreated, and $dn like so:
"$guid, $lastLogon, $deviceName, $serialNumber, $whenCreated, $dn" | add-content "c:\temp\filename.csv"
This took up an afternoon, so I hope this saves somebody some time and frustration.
I am new to PowerShell, and I have an issue figuring out how to create from one or multiple .csv files, one file for each value finding in a column, with all the rows that contains in that value.
Let me explain with a example:
I have this source CSV:
I need to create a file for each App in the file. the output of the script should be something like this.
App1.csv
App2.CSV
That for each app that appear in the Original CSV, a new file with all row that have such value, in this case App.
I want to do it in PowerShell so in can be automated, I try with Group-object, then foreach with the values, without luck. I wasn't even remotely close to found the solution.
Thanks in advance and happy coding!
I do not have the most elegant solution, but it works, I checked it on your files, but I ask you to upload the contents of the file as text next time to make it easier to copy. And don't forget choose delimiter in code (I usually use ';')
$some=Import-Csv "D:\testdir\file1.csv" -Delimiter ';'|
Sort-Object header2|
Group-Object -Property header2
foreach ($i in $some){
$i.Group|
Export-Csv -Delimiter ';' -Encoding UTF8 -NoTypeInformation -Force -Path ("D:\testdir\"+$i.Name+".csv")
}
Import the CSV.
Select all the unique values from the column that contains the App name.
Loop through the app names and get all rows where the app name matches.
Export to CSV.
$Path = "./MyFile.csv"
$DestinationDir = "./"
$CSV = Import-CSV -Path $Path
$AppNames = $CSV.H2 | Select -Unique
Foreach ($App in $AppNames)
{
$DestinationPath = Join-Path -ChildPath $App -Path $DestinationDir
$CSV | Where {$_.H2 -eq $APP} | Export-Csv -NoTypeInformation -Path $DestinationPath
}
Example CSV:
H1,H2,H3,H4
Data,App1,Data,Data
Data,App1,Data,Data
Data,App1,Data,Data
Data,App2,Data,Data
Data,App2,Data,Data
For performance reasons, you probably want to use the steppable pipeline here:
$Pipeline = #{}
Import-Csv .\Source.csv |
ForEach-Object -Process {
if (!$Pipeline.Contains($_.Header2)) {
$Pipeline[$_.Header2] = { Export-CSV -notype -Path .\$($_.Header2).csv }.GetSteppablePipeline()
$Pipeline[$_.Header2].Begin($True)
}
$Pipeline[$_.Header2].Process($_)
} -End {
foreach ($Key in $Pipeline.Keys) { $Pipeline[$Key].End() }
}
For a detailed explanation, see: Mastering the (steppable) pipeline
I am trying to automate below API calls from a csv file.
http_uri
/ModuleName/api/12345/moverequest/MoveRequestQueue?batchSize=200
/ModuleName/api/Portal/GetGarageLocations?email=Dummy#mail.com
/ModuleName/api/DeliveryDate/CommitEta?ref=H7J3M1EA4LF
/ModuleName/api/35345/moverequest/MoveRequestQueue?batchSize=500
The output should be like below in a csv file.
ScenarioName Parameter Value
MoveRequestQueue batchSize 200
GetGarageLocations email Dummy#mail.com
CommitEta ref H7J3M1EA4LF
MoveRequestQueue batchSize 500
I am using below code
$csv = Import-Csv C:\Powershell\Documents\Source.csv
$scenario = #()
ForEach ($row in $csv){
$httpuri = $($row.http_uri)
#Iterating through CSV rows and segregate values
if ($httpuri -match "="){
$equalarr = $httpuri -split '='
if ($equalarr[0] -match "\?"){
$questionarr = $equalarr[0] -split '\?'
$scenarionamearr = $questionarr[0] -split '/'
$totalelements = $scenarionamearr.Count
$scenarioname = $scenarionamearr[$totalelements-1]
$Scenario += $scenarioname
$Scenario += $questionarr[1]
$Scenario += $equalarr[1]
}
}
}
#Adding columns to csv
$columnName = '"Scenario","Parameter","Value"'
Add-Content -Path C:\Powershell\Documents\Output.csv -Value $columnName
#Writing values to CSV
$Scenario | foreach { Add-Content -Path C:\Powershell\Documents\Output.csv -Value $_ }
But Outout is generated like below
Scenario Parameter Value
DequeueMoveRequestQueue
batchSize
200
GetCarrierLocations
email
x-qldanxqldanx
Since i am a newbie, searched a lot to solve this issue but couldn't succeed. Please throw some light on this.
Thanks in advance....
If you store your scenarios in structured objects you can use Powershell's built in Export-Csv command to generate your csv.
So, instead of
$Scenario += $scenarioname
$Scenario += $questionarr[1]
$Scenario += $equalarr[1]
store an array of powershell objects:
$Scenario += [PSCustomObject]#{
"Scenario" = $scenarioname;
"Parameter" = $questionarr[1];
"Value" = $equalarr[1];}
Then, when creating the csv file, just use Export-Csv:
$Scenario | Export-Csv -NoTypeInformation -Path C:\Powershell\Documents\Output.csv
So the issue is that you make an empty array, then add strings to it one at a time, which just makes it an array of strings. Then when you output it to the file it just adds each string to the file on its own line. What you want to do is create an array of objects, then use the Export-Csv cmdlet to output it to a CSV file.
Creating an array, and then adding things to it one at a time is not a good way to do it. PowerShell has to recreate the array each time you add something the way you're doing it. Better would be to have a pipeline that outputs what you want (objects, rather than strings), and capture them all at once creating the array one time. Or even better, just output them to the CSV file and not recollect them in general.
$CSV = Import-Csv C:\Powershell\Documents\Source.csv
$CSV.http_uri -replace '^.*/(.*)$','$1'|ForEach-Object{
$Record = $_ -split '[=\?]'
[PSCustomObject]#{
ScenarioName = $Record[0]
Parameter = $Record[1]
Value = $Record[2]
}
} | Export-Csv -Path C:\Powershell\Documents\Output.csv -Append
I'm receiving an automated report from a system that cannot be modified as a CSV. I am using PowerShell to split the CSV into multiple files and parse out the specific data needed. The CSV contains columns that may contain no data, 1 value, or multiple values that are comma separated within the CSV file itself.
Example(UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542, 340668, 292196"
"Approval","AA-334454, 344366, 323570, 322827, 360225, 358850, 345935"
"ITS","345935, 358850"
"Services",""
I want the data to have one entry per line like this (UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542"
"Risk","340668"
"Risk","292196"
#etc.
I've tried splitting the data and I just get an unknown number of columns at the end.
I tried a foreach loop, but can't seem to get it right (pseudocode below):
Import-CSV $Groups
ForEach ($line in $Groups){
If($_.'Members'.count -gt 1, add-content "$_.Group,$_.Members[2]",)}
I appreciate any help you can provide. I've searched all the stackexchange posts and used Google but haven't been able to find something that addresses this exact issue.
Import-Csv .\input.csv | ForEach-Object {
ForEach ($Member in ($_.Members -Split ',')) {
[PSCustomObject]#{Group = $_.Group; Member = $Member.Trim()}
}
} | Export-Csv .\output.csv -NoTypeInformation
# Get the raw text contents
$CsvContents = Get-Content "\path\to\file.csv"
# Convert it to a table object
$CsvData = ConvertFrom-CSV -InputObject $CsvContents
# Iterate through the records in the table
ForEach ($Record in $CsvData) {
# Create array from the members values at commas & trim whitespace
$Record.Members -Split "," | % {
$MemberCount = $_.Trim()
# Check if the count is greater than 1
if($MemberCount -gt 1) {
# Create our output string
$OutputString = "$($Record.Group), $MemberCount"
# Write our output string to a file
Add-Content -Path "\path\to\output.txt" -Value $OutputString
}
}
}
This should work, you had the right idea but I think you may have been encountering some syntax issues. Let me know if you have questions :)
Revised the code as per your updated question,
$List = Import-Csv "\path\to\input.csv"
foreach ($row in $List) {
$Group = $row.Group
$Members = $row.Members -split ","
# Process for each value in Members
foreach ($MemberValue in $Members) {
# PS v3 and above
$Group + "," + $MemberValue | Export-Csv "\path\to\output.csv" -NoTypeInformation -Append
# PS v2
# $Group + "," + $MemberValue | Out-File "\path\to\output.csv" -Append
}
}
Background
I've been looking through several posts here on Stack and can only find answers to "how to add one single row of data to a CSV file" (notably this one). While they are good, they only refer to the specific case of adding a single entry from memory. Suppose I have 100,000 rows I want to add to a CSV file, then the speed of the query will be orders of magnitude slower if I for each row write it to file. I imagine that it will be much faster to keep everything in memory, and once I've built a variable that contains all the data that I want to add, only then write it to file.
Current situation
I have log files that I receive from customers containing about half a million rows. Some of these rows begin with a datetime and how much memory the server is using. In order to get a better view of how the memory usage looks like, I want to plot the memory usage over time using this information. (Note: yes, the best solution would be to ask the developers to add this information as it is fairly common we need this, but since we don't have that yet, I need to work with what I got)
I am able to read the log files, extract the contents, create two variables called $timeStamp and $memoryUsage that finds all the relevant entries. The problem occurs when I occurs when I try to add this to a custom PSObject. It would seem that using a $csvObject += $newRow only adds a pointer to the $newRow variable rather than the actual row itself. Here's the code that I've got so far:
$header1 = "Time Stamp"
$header2 = "Memory Usage"
$csvHeaders = #"
$header1;$header2
"#
# The following two lines are a workaround to make sure that the $csvObject becomes a PSObject that matches the output I'm trying to achieve.
$csvHeaders | Out-File -FilePath $csvFullPath
$csvObject = Import-Csv -Path $csvFullPath -Delimiter ";"
foreach ($TraceFile in $traceFilesToLookAt) {
$curTraceFile = Get-Content $TraceFile.FullName
Write-Host "Starting on file: $($TraceFile.Name)`n"
foreach ($line in $curTraceFile) {
try {
if (($line.Substring(4,1) -eq '-') -and ($line.Substring(7,1) -eq '-')) {
$TimeStamp = $line.Split("|",4)[0]
$memoryUsage = $($line.Split("|",4)[2]).Replace(",","")
$newRow = New-Object PSObject -Property #{
$header1 = $TimeStamp;
$header2 = $memoryUsage
}
$reorderedRow = $newRow | Select-Object -Property $header1,$header2
$reorderedRow | Export-Csv -Path $csvFullPath -Append -Delimiter ";"
}
} catch {
Out-Null
}
This works fine as it appends the row each time it finds one to the CSV file. The problem is that it's not very efficient.
End goal
I would ideally like to solve it with something like:
$newRow = New-Object PSObject -Property #{
$header1 = $TimeStamp;
$header2 = $memoryUsage
}
$rowsToAddToCSV += $newRow
And then in the final step do a:
$rowsToAddToCSV | Export-Csv -Path $csvFullPath -Append -Delimiter ";"
I have not been able to create any form of workaround for this. Among other things, PowerShell tells me that op_Addition is not part of the object, that the object I'm trying to export (the collection of rows) doesn't match the CSV file etc.
Anything that appends thousands of items to an array in a loop is bound to perform poorly, because each time an item is appended, the array will be re-created with its size increased by one, all existing items are copied, and then the new item is put in the new free slot.
Any particular reason why you can't simply do something like this?
$traceFilesToLookAt | ForEach-Object {
Get-Content $_.FullName | ForEach-Object {
if ($_.Substring(4, 1) -eq '-' -and $_.Substring(7, 1) -eq '-') {
$line = $_.Split('|', 4)
New-Object PSObject -Property #{
'Time Stamp' = $line[0]
'Memory Usage' = $line[2].Replace(',', '')
}
}
}
} | Export-Csv -Path $csvFullPath -Append -Delimiter ";"
A regular expression match might be an even more elegant approach to extracting timestamp and memory usage from the input files, but I'm going to leave that as an exercise for you.