Storing output in CSV file - powershell

I have wriiten the following SQL script to execute using PowerShell.
cls
foreach ($svr in get-content "demo.txt")
{
$con = "server=MC1-PQ10X.RF.LILLY.COM\SQL01;database=mylilly_WSS_Content_INCTSK0014840;Integrated Security=sspi"
$cmd = "SELECT
Docs.DirName + '/' + Docs.LeafName AS 'Item Name',
DocVersions.UIVersion,
(DocVersions.UIVersion/512) as Version_Label, DocVersions.Level, DocVersions.TimeCreated
FROM DocVersions FULL OUTER JOIN Docs ON Docs.Id = DocVersions.Id
-- INNER JOIN Webs On Docs.WebId = Webs.Id
--INNER JOIN Sites ON Webs.SiteId = SItes.Id
WHERE (DirName LIKE '%globalcontentrepository%')
AND (IsCurrentVersion = '0')
AND (DocVersions.Id IN ('$svr'))"
$da = new-object System.Data.SqlClient.SqlDataAdapter ($cmd, $con)
$dt = new-object System.Data.DataTable
$da.fill($dt) |out-null
$dt | Export-Csv music.csv -Encoding ascii -NoTypeInformation
}
The problem I'm facing with the above code is regarding the output. For every $svr this code is creating a new CSV file. The input file is containing around 1000 inputs. My requirement is that all the output should get stored in the same file rather than creating new file.

There are several ways to achieve your goal. One is to append to the output file as #PeterMmm suggested in the comments to your question:
foreach ($svr in Get-Content "demo.txt") {
...
$dt | Export-Csv music.csv -Encoding ascii -NoTypeInformation -Append
}
Note, however, that Export-Csv doesn't support appending prior to PowerShell v3. Also, appending usually has a negative impact on performance, because you have to repeatedly open the output file. It's better to put the export cmdlet outside the loop and write the output file in one go. foreach loops don't play nicely with pipelines, though, so you'd have to assign the loop output to a variable first:
$music = foreach ($svr in Get-Content "demo.txt") {
...
$dt
}
$music | Export-Csv music.csv -Encoding ascii -NoTypeInformation
or run it in a subexpression:
(foreach ($svr in Get-Content "demo.txt") {
...
$dt
}) | Export-Csv music.csv -Encoding ascii -NoTypeInformation
Personally I'd prefer a ForEach-Object loop over a foreach loop, because the former does work well with pipelines:
Get-Content "demo.txt" | ForEach-Object {
...
$dt
} | Export-Csv music.csv -Encoding ascii -NoTypeInformation
Note that you must replace every occurrence of your loop variable $svr with the "current object" variable $_ for this to work, though.

Related

File data not displaying in table form using Powershell Script

I have been grabbing the file data from student_data.dat and trying to display it into tabular form.
The first 3 lines of the file are written like this:
Jamie Zawinski,78.8,81.0,77.3,80.0,80.0,77.0
Adam Douglas,86.2,69.0,77.8,81.0,87.5,88.0
Wallace Steven,66.2,68.0,91.3,78.6,80.3,86.4
I wish to set it up into a table with headers of Student Name, Assignment-1, Assignment-2, etc. I will later be manipulating the data to calculate the class averages for each assignment and the students overall average in the course so far. Each method of setting up the table results in the file being displayed as:
#{Name=Jamie Zawinski; Assignment-1=78.8; Assignment-2=81.0; Assignment-3=77.3; Assignment-4=80.0;
Midterm_Exam=80.0; Final_Exam=77.0} #{Name=Adam Douglas; Assignment-1=86.2; Assignment-2=69.0;
Assignment-3=77.8; Assignment-4=81.0; Midterm_Exam=87.5; Final_Exam=88.0} #{Name=Wallace Steven;
Assignment-1=66.2; Assignment-2=68.0; Assignment-3=91.3; Assignment-4=78.6; Midterm_Exam=80.3;
Final_Exam=86.4}
My coding has looked like this:
$file = Import-Csv C:\**real path**\student_data.dat -Delimiter ',' -Header 'Name', 'Assignment-1','Assignment-2','Assignment-3','Assignment-4','Midterm_Exam','Final_Exam'
Write-Host $file
I tried adding:
foreach ($line in $file){
$data += [pscustomobject]#{
Name = $line.name
Assignment-1 = $line.Assignment-1
Assignment-2 = $line.Assignment-2
}
}
and writing instead:
$filedata = Get-Content ./student_data.dat
$newline = $filedata.Split("`n")
$newline.count
foreach ($l in $newline){
$Names = $l.Split(",")[0].Trim()
$Assignment-1 = $l.Split(",").Trim()
[pscustomObject]#{
Names = $Names;
Assignment-1 = $Assignment-1
}
}
but errors occurred.
First of all, note that you get a more readable output if you use Write-Output instead of Write-Host, which tries to push the entire input into a single string. (Or just remove it altogether, $file is equivalent to Write-Output $file)
As Matthias R. Jessen commented, you are probably looking for the Format-Table command:
$path = "C:\**real path**\student_data.dat"
$header = 'Name', 'Assignment-1','Assignment-2','Assignment-3','Assignment-4','Midterm_Exam','Final_Exam'
$data = Import-Csv $path -Delimiter ',' -Header $header
$data | Format-Table
Note this just "pretty-prints" the data for display in the console. The data is not changed and you should not use this for any kind of file output. And it's not really necessary for working with the data. Once you're done manipulating your data, you can just export it back as CSV:
$outpath = "C:\**real path**\student_data_result.csv"
$data | Export-Csv $outpath -Delimiter "," -NoTypeInformation

How can I count the number of CSV columns when the file has multiline data and no header

My CSV files have no headers and multi line entries like this:
11;"multi line
col12";13;foobar;foobar
21;22;23;24;25
And I'd like to count the number of columns. So 5 in this example. How do I do that?
What I tried:
Import-CSV doesn't work without the header parameter due to duplicate entries on the first line.
(Import-Csv .\bad.csv -Delimiter ";" | get-member -type NoteProperty).count
Adding a header parameter skews the count.
(Import-Csv .\bad.csv -Delimiter ";" -Header (1..99) | get-member -type NoteProperty).count
I had to abort reading the file manually via Get-Content because of all the parsing I would have to handle manually. Escaping characters and multi line entries...
My version of PowerShell is 3 and I have to port my script to version 2 later on.
If you are willing to accept the caveat that this could miscount the number of columns if there are quoted delimiters in string this could be good enough for you.
$path = "c:\temp\test.txt"
$delimiter = ";"
$numberOfColumns = Get-Content $path |
ForEach-Object{($_.split($delimiter)).Count} |
Measure-Object -Maximum |
Select-Object -ExpandProperty Maximum
Import-Csv $path -Header (1..$numberOfColumns) -Delimiter $delimiter
Read in the file with Get-Content and isolate the maximum number of columns by
splitting each line on its delimiter and then using that value to import the CSV. If the file is large you can read in the file once with Get-Content and then use ConvertTo-CSV once you know your column count.
If all lines contain a line break on them the above logic would fail. Still we could temporarily scrub the data by removing the correct line breaks in order to get the accurate count.
$delimiter = ";"
$fileData = (Get-Content $path | Out-String)
$numberOfColumns = ((($fileData -replace "(`"[^;]+?)`r`n",'$1') -split "`r`n" | Select -First 1).split($delimiter)).Count
$fileData | ConvertFrom-Csv -Header (1..$numberOfColumns) -Delimiter $delimiter
What this will do is find lines that end where there is a double quote followed by data that does not contain the delimiter. We also match the newline that follows but drop that same new line in the replacement. If that is done we know that the first line is proper. Use that same line to split and count just like before.
Since Excel knows, let's ask him :
$path = "path\to\bad.csv"
$excel = New-Object -ComObject Excel.Application
$workbook = $excel.Workbooks.Open($path)
$sheet = $workbook.ActiveSheet
$columnIndex = 1
while($sheet.Cells.Item(1, $columnIndex).Text -ne "") {
$columnIndex++
}
"There are $($columnIndex - 1) columns in CSV file $path"
Start-Sleep -Seconds 1
Get-Process excel | Stop-Process -Force
As pointed out by Ansgar Wiechers in comments, there is a much shorter solution :
$path = "path\to\bad.csv"
$excel = New-Object -ComObject Excel.Application
$workbook = $excel.Workbooks.Open($path)
$sheet = $workbook.ActiveSheet
$columnCount = $sheet.UsedRange.Columns.Count
"There are $columnCount columns in CSV file $path"
Start-Sleep -Seconds 1
Get-Process excel | Stop-Process -Force
(I know my way of killing Excel is dirty, but iirc it takes too much code to do so)
I know this is very old, but I came across a similar situation (did not have have rows of varying columns) today and found my own solution so I thought I would share for anyone else coming into this situation. My solution was to use Get-Content for the first row of the CSV and -split on the delimiter (,) to create an array and then return the count of the array. As mentioned in replies above, this will not account for delimiters existing within quotations.
((Get-Content $PathToCsv)[0] -split ",").count
I had the same issue and went with AAgent suggestion.
$CommaCount = ((Get-Content $PathToCsv)[0] -split ",").count
$SemicolonCount = ((Get-Content $PathToCsv)[0] -split ";").count
if ($CommaCount -gt $SemicolonCount){
$CMSlist = Import-Csv ($PathToCsv) –Delimiter “,”
}
else{
$CMSlist = Import-Csv ($PathToCsv) –Delimiter “;”

Powershell .csv merge with column remove

Using the code below I am able to merge several .csv files in 5 seconds.
$getFirstLine = $true
get-childItem "C:\my\dir\*.csv" | foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "C:\my\dir\output_code2.csv" $linesToWrite
}
I would like to take this one step further, preferable using piping to remove several of the columns using a command like:
select DateAndTime,DG1_KW,DG2_KW,WT_KW,HTR1_KW,POSS_Load_KW,INV1_KW,INV2_SOC|Export-csv output_test.csv -Notypeinformation
that being the variables in the header of each file.
How would I modify this code to make this work? The idea here is that I am going to be working with hundreds up to thousands of files.
I have other code which can do this but it is no where near as fast.
for instance using 10 .csv files that are 450kb each. the code below takes 20 seconds to process and spits out a .csv file in 20 seconds removing 48 of the 56 columns leaving the variables I need. If I remove part of the code that trims the columns it still takes 12+ seconds.
# Directory containing csv files, include *.*
$directory = "C:\my\dir\*.*";
# Get the csv files
$csvFiles = Get-ChildItem -Path $directory -Filter *.csv;
#$content = $null;
$content = #();
# Process each file
foreach($csv in $csvFiles)
{
$content += Import-Csv $csv;
}
# Write a datetime stamped csv file
$datetime = Get-Date -Format "yyyyMMddhhmmss";
$content |Export-Csv -Path "C:\my\dir\output_code2_$datetime.csv" -NoTypeInformation;
The code I would like to modify runs those same 10 files in 5 seconds but does not remove the 48 columns.
Any Ideas guys?
Ok, you want an example... Let's say your CSVs always look like this:
Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9,Col10
data1,data2,data3,data4,data5,data6,data7,data8,data9,data10
dataA,dataB,dataC,dataD,dataE,dataF,dataG,dataH,dataI,dataJ
Now let's say you only want Col1, Col2, Col6, Col9, and Col10. You could do a RegEx replace something like:
$Files = get-childItem "C:\my\dir\*.csv" | Select -Expand FullName
ForEach($File in $Files){
If($SkipFirst){
Get-Content $File | Select -Skip 1 | ForEach{$_ -replace "^((?:.*?\,){2})(?:.*\,){3}(.*?\,)(?:(?:.*?\,){2})(.*?,.*?)$", '$1$2$3'} | Add-Content "C:\my\dir\output_code2.csv"
}Else{
Get-Content $File | ForEach{$_ -replace "^((?:.*?\,){2})(?:.*\,){3}(.*?\,)(?:(?:.*?\,){2})(.*?,.*?)$", '$1$2$3'} | Add-Content "C:\my\dir\output_code2.csv"
}
}
That would extract just the columns that I noted above. See https://regex101.com/r/jY4oO6/1 for detailed breakdown of RegEx string. Effective output would be (skipping first line if so dictated):
Col1,Col2,Col6,Col9,Col10
data1,data2,data6,data9,data10
dataA,dataB,dataF,dataI,dataJ

Powershell INI editing

I want to changing some values in an INI file. Unfortunately, I have keys in 2 different sections which share an identical name but need different values. My code uses the Get-IniContent function from PsIni.
Example INI file:
[PosScreen]
BitmapFile=C:\Temp\Random.bmp
Bitmap=1
[ControlScreen]
BitmapFile=C:\Temp\Random.bmp
Bitmap=1
I need to change the above to the following:
[PosScreen]
BitmapFile=C:\Temp\FileC.bmp
Bitmap=1
[ControlScreen]
BitmapFile=C:\Temp\FileD.bmp
Bitmap=1
The PowerShell code I am using seems to work, but it changes every value to "File D". It is obviously parsing everything twice, and the name is the same for each section.
$NewFileC = "C:\Temp\FileC.bmp"
$NewFileD = "C:\Temp\FileD.bmp"
$POSIniContent = Get-IniContent "C:\scripts\Update-EnablerImage\WINSUITE.INI"
$BOIniContent = Get-IniContent "C:\scripts\Update-EnablerImage\WINSUITE.INI"
If ($POSIniContent["PosScreen"]["BitmapFile"] -ne $NewFileC) {
Get-Content "C:\scripts\Update-EnablerImage\WINSUITE.INI" |
ForEach-Object {$_ -replace "BitmapFile=.+" , "BitmapFile=$NewFileC" } |
Set-Content "C:\scripts\Update-EnablerImage\WINSUITE.INI"
}
If ($BOIniContent["ControlScreen"]["BitmapFile"] -ne $NewFileD) {
Get-Content "C:\scripts\Update-EnablerImage\WINSUITE.INI" |
ForEach-Object {$_ -replace "BitmapFile=.+" , "BitmapFile=$NewFileD" } |
Set-Content "C:\scripts\Update-EnablerImage\WINSUITE.INI"
}
My struggle is how to change each one independently. I'm a bit of a scripting newbie, so calling out for some help. Tried using Conditional Logic (ForEach $line in $INIFile, for example), but no luck with that.
You are overcomplicating things. You can use Get-IniContent and Out-IniFile as follows:
$ini = Get-IniContent c:\temp\ini.ini
$ini["posscreen"]["BitmapFile"] = "C:\Temp\FileC.bmp"
$ini | Out-IniFile -FilePath c:\temp\ini2.ini
Note that if you want to overwrite the original file, you must add -Force to the Out-IniFile call.

Output to Pipeline from within IF ELSE statement

UPDATE: I just found out that the client I am running this on is PS V1. I can't use splatting.
I have a script for processing csv files. I don't know ahead of time if the file will have a header or not so I'm prompting the user to either input a header or use the one in the file:
$header = Read-Host 'Input your own header?'
What I want to do is be able to check whether the header variable has been set to decide if I want to use that flag when executing the Import-CSV commandlet.
I have tried a number of things along the lines of:
IF($Header){Import-Csv $File -delimiter $delimiter -Header $header }
ELSE
{Import-Csv $File -delimiter $delimiter} |
Or
IF($Header){Import-Csv $File -delimiter $delimiter -Header $header | %{$_} }
ELSE
{Import-Csv $File -delimiter $delimiter | %{$_}}
The first example results in complaints of an empty pipeline. The second results in the first column of the file just being ouptut to the console and then errors when the file is done processing because the rest of the pipeline is empty.
As always any help would be appreciated.
Thanks,
Bill
Eris already suggested splatting, but I wanted to give a more comprehensive example, using your code.
# Declare a hash containing the parameters you will always need
$csvParams = #{
File = $File;
delimiter = $delimiter;
}
# if the header is specified, add a Header to $csvParams
if ($Header) { $csvParams.Header = $header }
# Call Import-Csv, splatting $csvParams
Import-Csv #csvParams
Splatting is an extremely useful technique. Get-Help about_Splatting for more information about it.
The easiest way is just to use the -OutVariable option on Import-Csv
Import-Csv -Path:$File -Delimiter:$Delimiter -OutVariable:CSVContents will save it in $CSVContents
From the Import-CSV Technet page:
This cmdlet supports the common parameters: -Verbose, -Debug, -ErrorAction, -ErrorVariable, -OutBuffer, and -OutVariable. For more information, see about_CommonParameters (http://go.microsoft.com/fwlink/p/?LinkID=113216).
Another alternative is to use an args hash and "splat" it:
$myArgs = #{
Path = "$HOME\temp\foo"
}
Get-Content #myArgs
Update for Version 1 (untested):
( IF($Header) { Import-Csv $File -delimiter $delimiter -Header $header }
ELSE { Import-Csv $File -delimiter $delimiter} ) |
#More pipeline commands here, example:
Format-Table
Horrible disgusting version (untested):
$ImportCommand = "Import-Csv $File -delimiter $delimiter"
If($header -ne $null) { $ImportCommand += " -header $header" }
Invoke-Expression $ImportCommand |
Format-Table
You second approach is the way I would go. I'm not sure why, what you have shown for your second example, would be failing. Here is what I would do:
filter ProcessCsv {
$_ | ... further processing ...
}
$header = Read-Host 'Input your own header?'
if ($header) {
Import-Csv $file -Delimiter $delimiter -Header $header | ProcessCsv
else {
Import-Csv $file -Delimiter $delimiter | ProcessCsv
}