I am trying to automate below API calls from a csv file.
http_uri
/ModuleName/api/12345/moverequest/MoveRequestQueue?batchSize=200
/ModuleName/api/Portal/GetGarageLocations?email=Dummy#mail.com
/ModuleName/api/DeliveryDate/CommitEta?ref=H7J3M1EA4LF
/ModuleName/api/35345/moverequest/MoveRequestQueue?batchSize=500
The output should be like below in a csv file.
ScenarioName Parameter Value
MoveRequestQueue batchSize 200
GetGarageLocations email Dummy#mail.com
CommitEta ref H7J3M1EA4LF
MoveRequestQueue batchSize 500
I am using below code
$csv = Import-Csv C:\Powershell\Documents\Source.csv
$scenario = #()
ForEach ($row in $csv){
$httpuri = $($row.http_uri)
#Iterating through CSV rows and segregate values
if ($httpuri -match "="){
$equalarr = $httpuri -split '='
if ($equalarr[0] -match "\?"){
$questionarr = $equalarr[0] -split '\?'
$scenarionamearr = $questionarr[0] -split '/'
$totalelements = $scenarionamearr.Count
$scenarioname = $scenarionamearr[$totalelements-1]
$Scenario += $scenarioname
$Scenario += $questionarr[1]
$Scenario += $equalarr[1]
}
}
}
#Adding columns to csv
$columnName = '"Scenario","Parameter","Value"'
Add-Content -Path C:\Powershell\Documents\Output.csv -Value $columnName
#Writing values to CSV
$Scenario | foreach { Add-Content -Path C:\Powershell\Documents\Output.csv -Value $_ }
But Outout is generated like below
Scenario Parameter Value
DequeueMoveRequestQueue
batchSize
200
GetCarrierLocations
email
x-qldanxqldanx
Since i am a newbie, searched a lot to solve this issue but couldn't succeed. Please throw some light on this.
Thanks in advance....
If you store your scenarios in structured objects you can use Powershell's built in Export-Csv command to generate your csv.
So, instead of
$Scenario += $scenarioname
$Scenario += $questionarr[1]
$Scenario += $equalarr[1]
store an array of powershell objects:
$Scenario += [PSCustomObject]#{
"Scenario" = $scenarioname;
"Parameter" = $questionarr[1];
"Value" = $equalarr[1];}
Then, when creating the csv file, just use Export-Csv:
$Scenario | Export-Csv -NoTypeInformation -Path C:\Powershell\Documents\Output.csv
So the issue is that you make an empty array, then add strings to it one at a time, which just makes it an array of strings. Then when you output it to the file it just adds each string to the file on its own line. What you want to do is create an array of objects, then use the Export-Csv cmdlet to output it to a CSV file.
Creating an array, and then adding things to it one at a time is not a good way to do it. PowerShell has to recreate the array each time you add something the way you're doing it. Better would be to have a pipeline that outputs what you want (objects, rather than strings), and capture them all at once creating the array one time. Or even better, just output them to the CSV file and not recollect them in general.
$CSV = Import-Csv C:\Powershell\Documents\Source.csv
$CSV.http_uri -replace '^.*/(.*)$','$1'|ForEach-Object{
$Record = $_ -split '[=\?]'
[PSCustomObject]#{
ScenarioName = $Record[0]
Parameter = $Record[1]
Value = $Record[2]
}
} | Export-Csv -Path C:\Powershell\Documents\Output.csv -Append
Related
I am new to PowerShell or any scripting stuff.
I have here a DataResources(CSV File) which contains of bunch of data's that needs to be inserted into a Results.CSV
Here is my DataResources
For my Result.csv looks like this.
My goal is to export a NEW csv file based from a results.csv as a template
and if a HostName/Host match the data contains a value equivalent to UA and PWD will be inserted/updated from ExportResources CSV file to Exported new CSV File
after executing scripts that I have created it only modifies 1 row not all.
This is what I've done so far.
$DataSource = Import-Csv -Path D:\coding\Powershell\csv\Resources\ExportResources.csv
$DataResults = Import-Csv -Path D:\coding\Powershell\csv\Result\results.csv
foreach ($ItemDataResults in $DataResults)
{
$HostName = $ItemDataResults.HostName
$UA = $ItemDataResults.UA
$PASSWD = $ItemDataResults.PWD
}
$ItemDataSource = $DataSource | ? {$_.Host -eq $HostName}
if ($UA -eq "" -and $PASSWD -eq "")
{
$ItemDataResults.UA = $ItemDataSource.UA
$ItemDataResults.PWD = $ItemDataSource.Passwd
}
$DataResults | Export-Csv D:\coding\Powershell\csv\new.csv -NoTypeInformation
The result almost meet my expectation but the problem here that it only fills one hostname the rest are empty.
The issue with your script is that you loop through $DataResults once, but you have to iterate through it once for each item in $DataSource if you use the method that you're employing. I believe that a better method is to create a hashtable from one CSV, using the host name as the key, and the whole object as the value, then loop through the second array updating the value for each host. That would look something like this:
$DataSource = Import-Csv -Path D:\coding\Powershell\csv\Resources\ExportResources.csv
$DataResults = Import-Csv -Path D:\coding\Powershell\csv\Result\results.csv
$DataHT = #{}
$DataResults | ForEach-Object { $DataHT.Add($_.HostName,$_) }
ForEach( $Record in $DataSource ){
$DataHT[$Record.Host].UA = $Record.UA
$ItemDataResults.PWD = $Record.Passwd
}
$DataHT.Values | Export-Csv D:\coding\Powershell\csv\new.csv -NoTypeInformation
I created a script that lists all the folders, subfolders and files and export them to csv:
$path = "C:\tools"
Get-ChildItem $path -Recurse |select fullname | export-csv -Path "C:\temp\output.csv" -NoTypeInformation
But I would like that each folder, subfolder and file in pfad is written into separate column in csv.
Something like this:
c:\tools\test\1.jpg
Column1
Column2
Column3
tools
test
1.jpg
I will be grateful for any help.
Thank you.
You can split the Fullname property using the Split() method. The tricky part is that you need to know the maximum path depth in advance, as the CSV format requires that all rows have the same number of columns (even if some columns are empty).
# Process directory $path recursively
$allItems = Get-ChildItem $path -Recurse | ForEach-Object {
# Split on directory separator (typically '\' for Windows and '/' for Unix-like OS)
$FullNameSplit = $_.FullName.Split( [IO.Path]::DirectorySeparatorChar )
# Create an object that contains the splitted path and the path depth.
# This is implicit output that PowerShell captures and adds to $allItems.
[PSCustomObject] #{
FullNameSplit = $FullNameSplit
PathDepth = $FullNameSplit.Count
}
}
# Determine highest column index from maximum depth of all paths.
# Minus one, because we'll skip root path component.
$maxColumnIndex = ( $allItems | Measure-Object -Maximum PathDepth ).Maximum - 1
$allRows = foreach( $item in $allItems ) {
# Create an ordered hashtable
$row = [ordered]#{}
# Add all path components to hashtable. Make sure all rows have same number of columns.
foreach( $i in 1..$maxColumnIndex ) {
$row[ "Column$i" ] = if( $i -lt $item.FullNameSplit.Count ) { $item.FullNameSplit[ $i ] } else { $null }
}
# Convert hashtable to object suitable for output to CSV.
# This is implicit output that PowerShell captures and adds to $allRows.
[PSCustomObject] $row
}
# Finally output to CSV file
$allRows | Export-Csv -Path "C:\temp\output.csv" -NoTypeInformation
Notes:
The syntax Select-Object #{ Name= ..., Expression = ... } creates a calculated property.
$allRows = foreach captures and assigns all output of the foreach loop to variable $allRows, which will be an array if the loop outputs more than one object. This works with most other control statements as well, e. g. if and switch.
Within the loop I could have created a [PSCustomObject] directly (and used Add-Member to add properties to it) instead of first creating a hashtable and then converting to [PSCustomObject]. The choosen way should be faster as no additional overhead for calling cmdlets is required.
While a file with rows containing a variable number of items is not actually a CSV file, you can roll your own and Microsoft Excel can read it.
=== Get-DirCsv.ps1
Get-Childitem -File |
ForEach-Object {
$NameParts = $_.FullName -split '\\'
$QuotedParts = [System.Collections.ArrayList]::new()
foreach ($NamePart in $NameParts) {
$QuotedParts.Add('"' + $NamePart + '"') | Out-Null
}
Write-Output $($QuotedParts -join ',')
}
Use this to capture the output to a file with:
.\Get-DirCsv.ps1 | Out-File -FilePath '.\dir.csv' -Encoding ascii
I'm receiving an automated report from a system that cannot be modified as a CSV. I am using PowerShell to split the CSV into multiple files and parse out the specific data needed. The CSV contains columns that may contain no data, 1 value, or multiple values that are comma separated within the CSV file itself.
Example(UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542, 340668, 292196"
"Approval","AA-334454, 344366, 323570, 322827, 360225, 358850, 345935"
"ITS","345935, 358850"
"Services",""
I want the data to have one entry per line like this (UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542"
"Risk","340668"
"Risk","292196"
#etc.
I've tried splitting the data and I just get an unknown number of columns at the end.
I tried a foreach loop, but can't seem to get it right (pseudocode below):
Import-CSV $Groups
ForEach ($line in $Groups){
If($_.'Members'.count -gt 1, add-content "$_.Group,$_.Members[2]",)}
I appreciate any help you can provide. I've searched all the stackexchange posts and used Google but haven't been able to find something that addresses this exact issue.
Import-Csv .\input.csv | ForEach-Object {
ForEach ($Member in ($_.Members -Split ',')) {
[PSCustomObject]#{Group = $_.Group; Member = $Member.Trim()}
}
} | Export-Csv .\output.csv -NoTypeInformation
# Get the raw text contents
$CsvContents = Get-Content "\path\to\file.csv"
# Convert it to a table object
$CsvData = ConvertFrom-CSV -InputObject $CsvContents
# Iterate through the records in the table
ForEach ($Record in $CsvData) {
# Create array from the members values at commas & trim whitespace
$Record.Members -Split "," | % {
$MemberCount = $_.Trim()
# Check if the count is greater than 1
if($MemberCount -gt 1) {
# Create our output string
$OutputString = "$($Record.Group), $MemberCount"
# Write our output string to a file
Add-Content -Path "\path\to\output.txt" -Value $OutputString
}
}
}
This should work, you had the right idea but I think you may have been encountering some syntax issues. Let me know if you have questions :)
Revised the code as per your updated question,
$List = Import-Csv "\path\to\input.csv"
foreach ($row in $List) {
$Group = $row.Group
$Members = $row.Members -split ","
# Process for each value in Members
foreach ($MemberValue in $Members) {
# PS v3 and above
$Group + "," + $MemberValue | Export-Csv "\path\to\output.csv" -NoTypeInformation -Append
# PS v2
# $Group + "," + $MemberValue | Out-File "\path\to\output.csv" -Append
}
}
Background
I've been looking through several posts here on Stack and can only find answers to "how to add one single row of data to a CSV file" (notably this one). While they are good, they only refer to the specific case of adding a single entry from memory. Suppose I have 100,000 rows I want to add to a CSV file, then the speed of the query will be orders of magnitude slower if I for each row write it to file. I imagine that it will be much faster to keep everything in memory, and once I've built a variable that contains all the data that I want to add, only then write it to file.
Current situation
I have log files that I receive from customers containing about half a million rows. Some of these rows begin with a datetime and how much memory the server is using. In order to get a better view of how the memory usage looks like, I want to plot the memory usage over time using this information. (Note: yes, the best solution would be to ask the developers to add this information as it is fairly common we need this, but since we don't have that yet, I need to work with what I got)
I am able to read the log files, extract the contents, create two variables called $timeStamp and $memoryUsage that finds all the relevant entries. The problem occurs when I occurs when I try to add this to a custom PSObject. It would seem that using a $csvObject += $newRow only adds a pointer to the $newRow variable rather than the actual row itself. Here's the code that I've got so far:
$header1 = "Time Stamp"
$header2 = "Memory Usage"
$csvHeaders = #"
$header1;$header2
"#
# The following two lines are a workaround to make sure that the $csvObject becomes a PSObject that matches the output I'm trying to achieve.
$csvHeaders | Out-File -FilePath $csvFullPath
$csvObject = Import-Csv -Path $csvFullPath -Delimiter ";"
foreach ($TraceFile in $traceFilesToLookAt) {
$curTraceFile = Get-Content $TraceFile.FullName
Write-Host "Starting on file: $($TraceFile.Name)`n"
foreach ($line in $curTraceFile) {
try {
if (($line.Substring(4,1) -eq '-') -and ($line.Substring(7,1) -eq '-')) {
$TimeStamp = $line.Split("|",4)[0]
$memoryUsage = $($line.Split("|",4)[2]).Replace(",","")
$newRow = New-Object PSObject -Property #{
$header1 = $TimeStamp;
$header2 = $memoryUsage
}
$reorderedRow = $newRow | Select-Object -Property $header1,$header2
$reorderedRow | Export-Csv -Path $csvFullPath -Append -Delimiter ";"
}
} catch {
Out-Null
}
This works fine as it appends the row each time it finds one to the CSV file. The problem is that it's not very efficient.
End goal
I would ideally like to solve it with something like:
$newRow = New-Object PSObject -Property #{
$header1 = $TimeStamp;
$header2 = $memoryUsage
}
$rowsToAddToCSV += $newRow
And then in the final step do a:
$rowsToAddToCSV | Export-Csv -Path $csvFullPath -Append -Delimiter ";"
I have not been able to create any form of workaround for this. Among other things, PowerShell tells me that op_Addition is not part of the object, that the object I'm trying to export (the collection of rows) doesn't match the CSV file etc.
Anything that appends thousands of items to an array in a loop is bound to perform poorly, because each time an item is appended, the array will be re-created with its size increased by one, all existing items are copied, and then the new item is put in the new free slot.
Any particular reason why you can't simply do something like this?
$traceFilesToLookAt | ForEach-Object {
Get-Content $_.FullName | ForEach-Object {
if ($_.Substring(4, 1) -eq '-' -and $_.Substring(7, 1) -eq '-') {
$line = $_.Split('|', 4)
New-Object PSObject -Property #{
'Time Stamp' = $line[0]
'Memory Usage' = $line[2].Replace(',', '')
}
}
}
} | Export-Csv -Path $csvFullPath -Append -Delimiter ";"
A regular expression match might be an even more elegant approach to extracting timestamp and memory usage from the input files, but I'm going to leave that as an exercise for you.
I'm new to PowerShell and have a script which loops through Active Directory searching for certain computers. I get several variables and then run functions to check things like WMI and registry settings.
In the console, my script runs great and simple Write-Host command prints the data on the screen as I want. I know about Export-Csv when using the pipeline...but I'm not looking to print from the pipeline.
I want to write the variables to a text file, continue the loop, and check the next computer in AD...output the next iteration of the same variables on the next line. Here is my Write-Host:
Write-Host ($computer)","($Speed)","($Regcheck)","($OU)
Output file:
$computer,$Speed,$Regcheck | out-file -filepath C:\temp\scripts\pshell\dump.txt -append -width 200
It gives me the data, but each variable is on its own line. Why? I'd like all the variables on one line with comma separation. Is there a simple way to do this akin to VB writeline? My PowerShell version appears to be 2.0.
Use this:
"$computer, $Speed, $Regcheck" | out-file -filepath C:\temp\scripts\pshell\dump.txt -append -width 200
I usually construct custom objects in these loops, and then add these objects to an array that I can easily manipulate, sort, export to CSV, etc.:
# Construct an out-array to use for data export
$OutArray = #()
# The computer loop you already have
foreach ($server in $serverlist)
{
# Construct an object
$myobj = "" | Select "computer", "Speed", "Regcheck"
# Fill the object
$myobj.computer = $computer
$myobj.speed = $speed
$myobj.regcheck = $regcheck
# Add the object to the out-array
$outarray += $myobj
# Wipe the object just to be sure
$myobj = $null
}
# After the loop, export the array to CSV
$outarray | export-csv "somefile.csv"
You can concatenate an array of values together using PowerShell's `-join' operator. Here is an example:
$FilePath = '{0}\temp\scripts\pshell\dump.txt' -f $env:SystemDrive;
$Computer = 'pc1';
$Speed = 9001;
$RegCheck = $true;
$Computer,$Speed,$RegCheck -join ',' | Out-File -FilePath $FilePath -Append -Width 200;
Output
pc1,9001,True
$computer,$Speed,$Regcheck will create an array, and run out-file ones per variable = they get seperate lines. If you construct a single string using the variables first, it will show up a single line. Like this:
"$computer,$Speed,$Regcheck" | out-file -filepath C:\temp\scripts\pshell\dump.txt -append -width 200
The simple solution is to avoid creating an array before piping to Out-File. Rule #1 of PowerShell is that the comma is a special delimiter, and the default behavior is to create an array. Concatenation is done like this.
$computer + "," + $Speed + "," + $Regcheck | out-file -filepath C:\temp\scripts\pshell\dump.txt -append -width 200
This creates an array of three items.
$computer,$Speed,$Regcheck
FYKJ
100
YES
vs. concatenation of three items separated by commas.
$computer + "," + $Speed + "," + $Regcheck
FYKJ,100,YES
I was lead here in my Google searching. In a show of good faith I have included what I pieced together from parts of this code and other code I've gathered along the way.
# This script is useful if you have attributes or properties that span across several commandlets
# and you wish to export a certain data set but all of the properties you wish to export are not
# included in only one commandlet so you must use more than one to export the data set you want
#
# Created: Joshua Biddle 08/24/2017
# Edited: Joshua Biddle 08/24/2017
#
$A = Get-ADGroupMember "YourGroupName"
# Construct an out-array to use for data export
$Results = #()
foreach ($B in $A)
{
# Construct an object
$myobj = Get-ADuser $B.samAccountName -Properties ScriptPath,Office
# Fill the object
$Properties = #{
samAccountName = $myobj.samAccountName
Name = $myobj.Name
Office = $myobj.Office
ScriptPath = $myobj.ScriptPath
}
# Add the object to the out-array
$Results += New-Object psobject -Property $Properties
# Wipe the object just to be sure
$myobj = $null
}
# After the loop, export the array to CSV
$Results | Select "samAccountName", "Name", "Office", "ScriptPath" | Export-CSV "C:\Temp\YourData.csv"
Cheers