I have a CSV file with logging information. Date/time info in the file is written in UNIX time-stamp format.
I want to remove all lines older than 5 minutes. I use the following code to do this:
$FiveMinutesAgo=(Get-Date ((get-date).toUniversalTime()) -UFormat +%s).SubString(0,10)-300
$Content = Import-Csv "logfile.csv" | where {$_.DateCompleted -gt $FiveMinutesAgo}
$Content | Export-Csv -delimiter ";" -NoTypeInformation -encoding UTF8 -Path 'logfile.csv'
The CSV file looks like this:
"DateInitiated";"DateStarted";"DateCompleted";"InitiatedBy";"Action";"Status"
"1496659208";"1496659264";"1496752840";"administrator";"Reboot server";"Completed"
No matter whether the five minutes have already passed or not, my CSV file ends up completely empty after executing the script lines above.
I couldn't replicate it in testing (but potentially need the actual source file). I think the issue is that you need to specify both the encoding and delimiter on the Import-CSV as well (as you already have on Export).
Try this:
$FiveMinutesAgo=(Get-Date ((get-date).toUniversalTime()) -UFormat +%s).SubString(0,10)-300
$Content = Import-Csv "test.csv" -Delimiter ';' -Encoding UTF8 | where {$_.DateCompleted -gt $FiveMinutesAgo}
$Content | Export-Csv -delimiter ";" -NoTypeInformation -encoding UTF8 -Path 'logfile.csv'
Related
I'm familiar with exporting data to csv with Powershell, but a particular vendor wants to receive the data in a file that is specific type 'CSV'. If I use the following example command, the data looks fine in the raw csv format (i.e via notepad), where you see it's comma separated ...
,,JVF1033292,test,SL,10700,6804626,745.586,43001843,Test,8/12/2020,8/14/2020,T,5584,,,JPY,0,XTKS,,,,,,0
,,JVF1033293,test,SL,3695,6805179,1362.8457,43001843,Test,8/12/2020,8/14/2020,T,3524,,,JPY,0,XTKS,,,,,,0
... but when the same file is opened in Excel all the data is in one row and therefore is failing on the vendors side. The code I'm using is below.
$Table | ConvertTo-Csv -NoTypeInformation | ForEach-Object {$_-replace "`"", ""} | select -skip 1 | out-file -filepath ("$dir_opr\LVT\Export\CTO\ForJefferies\GMO_Jefferies_Trade_Data_" + $Date+ ".csv")
If I use the below code, then it looks fine in Excel (tabbed correctly), but the raw file is also tabbed and not comma separated which the vendor has issues with.
$Table | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | ForEach-Object {$_-replace "`"", ""} | select -skip 1 | out-file -filepath ("$dir_opr\LVT\Export\CTO\ForJefferies\GMO_Jefferies_Trade_Data_" + $Date+ ".csv")
X JVF1032244 Test BY 450.0000 BYM41J3 10.00000000 43001843 Test 08/11/2020 08/13/2020 T 3.00 JPY 0 XTKS 0.00
X JVF1032245 Test BY 200.0000 BYM41J3 250.00000000 43001843 Test 08/11/2020 08/13/2020
Is it possible to create a comma separated and tabbed raw file, that also is delimited in Excel so not all in one column?
As far as I can tell with testing locally, it might be due to the default encoding that out-file uses - i.e utf8NoBOM. (See the description for the -Encoding parameter for details: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/out-file?view=powershell-7)
Try out-file -filepath $myfile -encoding utf8 - that seems to fix it on my machine.
I have CSV file in below format, when I am editing it in notepad I see following content: please see A - code area
I need something like below, I always have this format when using export-csv from powershell. please see B - code area
I need to use this CSV file in powershell script. I was trying to convert it in excell, and try to use this powershell command:
Import-Csv -Path C:\PS\FileIN.csv | Export-Csv -Path FileOUT.csv -NoTypeInformation UTF8
.I know that I can do it in Notepad++, but I dont want to perform this task manually. Is there any method I can use to convert this CSV file to required format?
# A - Now I see this format
COL1;COL2;COL3;COL4
value1;value2;value3;value4
value5;value6;value7;value8
# B - I need this format"
"COL1","COL2","COL3","COL4"
"value1","value2","value3","value4"
"value5","value6","value7","value8"
Suggested powershell command resolved the problem, additionally i added parameter: -encoded UTF8 at the end od the command.
Import-CSV -Delimiter ';' -Path C:\PS\FileIN.CSV -Encoding UTF8 | Export-CSV -Path FileOut.csv -NoTypeInformation -Enconing UTF8
Add -Encoding UTF8 to your Import-CSV command:
Import-CSV -Delimiter ';' -Path C:\PS\FileIN.CSV -Encoding UTF8 | Export-CSV -Path FileOut.csv -NoTypeInformation
I created a short powershell script to convert csv files from Unicode to UTF-8 encoding. My script outputs new files with the the original file name preceded by UTF8. I'm running into two issues:
I'm trying to only run the powershell script on csv files. Currently the script runs on every file in the directory, including the powershell script (it outputs a new file called UTF8pshell_script if the powershell script was called pshell_script for example). The other methods where I've tried to only run the script on csv files just end up making the script not do anything.
I'm trying to run the script on sub-directories. The first issue is that output files created from csv files in subdirectories have no content inside them whatsoever. If the script is ran in the same directory as the csv file this problem does not arise. This is not crucial but I am also uncertain how to get output files created from those in subdirectories to be outputted in the same subdirectories (currently they are outputted in the main directory where the powershell script is).
as
Get-Content -Encoding Unicode $_ | Out-File -Encoding UTF8
Get-ChildItem -Recurse | ForEach-Object {Get-Content -Encoding Unicode $_ | Out-File -Encoding UTF8 "UTF8$_"}
The desired output is the powershell script running on only csv files, and outputting files to the same subdirectories where the files they were created form are.
Get-ChildItem takes a -Filter parameter, which for files is the simple wildcard pattern. This will allow you to restrict your cmdlet to CSV files only:
Get-ChildItem -Filter *.csv
To process subdirectories, you may also use the -Recurse switch
Get-ChildItem -Filter *.csv -Recurse
Now, I'm never quite sure how $_ changes as you pass different objects through the pipe, so I'm probably not doing the next steps the most efficient way - but it will be clear what I'm trying to do:
Each file object that we find needs to be processed as follows:
Dissect it into a path and a filename: $filepath = $_.PSParentPath; $filename = $_.PSChildName
Load up the CSV: Import-CSV -Path $_
Output the new CSV with the proper encoding: Export-CSV -Path ("{0}\UTF8{1}" -f $filepath,$filename) -Encoding UTF8
So, we put it all together:
Get-ChildItem -Filter *.csv -Recurse -exclude UTF8* | ForEach-Object {
$filepath = $_.PSParentPath
$filename = $_.PSChildName
Import-CSV -Path $_ |
Export-CSV -Encoding UTF8 -Path ("{0}\UTF8{1}" -f $filepath,$filename) -NoTypeInformation
}
The -Exclude UTF8* in the Get-ChildItem ensures that when you create a file, it doesn't get picked up later and re-processed. The -NoTypeInformation on the Export-CSV compensates for a stupidity built in to the cmdlet that causes an extra line with a meaningless object type name at the beginning of the file.
Depending on the original encoding (and presence of a BOM) you might have to specify an encoding also on the input side.
ForEach($Csv in (Get-ChildItem -Filter *.csv -Recurse -Exclude UTF8*)){
(Get-Content $Csv.FullName -raw) |
Set-Content -Path {Join-Path $Csv.Directory ("UTF8"+$Csv.Name)} -Encoding UTF8
}
LotPings beat me to this by 10 minutes with a virtually identical answer, but I'm leaving this for the 'passing an empty file to the pipeline' bit that I have. I also realize after the fact that you don't need a pipeline variable for that same reason, as you only need it if you pass things through the pipeline within the loop.
If all you want to do is change the encoding I would use a ForEach($x in $y){} loop, or a ForEach-Object{} loop with a PipelineVariable on the Get-ChildItem. I'll show that since I think pipeline variables are under used. I would also not read the file and pipe it to something, since if the file is empty you won't create a new file as nothing is passed down the pipeline.
Get-ChildItem *.csv -Recurse -PipelineVariable File | ForEach-Object{
Set-Content -Value (Get-Content $File.FullName -Encoding Unicode) -Path {Join-Path $File.Directory "UTF8$($File.Name)"} -Encoding UTF8
}
if you specify the file extension at the end of Get-ChildItem.
This will get only the files with the .csv extension.
By specifying the File path in Out-File it will send it to the specified directory.
Get-ChildItem -Path C:\folder\*.csv -Recurse | ForEach-Object {Get-Content -Encoding Unicode $_ | Out-File -FilePath C:\Folder -Encoding UTF8 "UTF8$_"}
I have multiple .tsv file that I want to convert to a .csv file using Windows Powershell.
I used
Import-Csv C:\temp\file\users.txt -Delimiter "`t"
But I can't export to a csv file.
Can someone help please?
Use the Export-Csv cmdlet to export the data.
Now, Import-Csv only takes one file at a time, so wrap that in a loop or the ForEach-Object cmdlet:
Get-ChildItem C:\temp\file\ -Filter *.txt |ForEach-Object {
Import-Csv $_.FullName -Delimiter "`t"
} |Export-Csv C:\path\to\output.csv -Delimiter "`t" -NoTypeInformation
My Old Bat file
Copy F:\File.hdr+F:*.csv F:\FinalOutput.csv
the HDR file is a single entry file that has only header data for the CSV files
Is there a way to perform this in PowerShell (to combine all the CSV files into a single file)?
Here is my powershell script that doesn't work
$CSVFolder = 'F:\Input\';
$OutputFile = 'F:\Output\NewOutput.csv';
$CSV= #();
Get-ChildItem -Path $CSVFolder -Filter *.inv | ForEach-Object {
$CSV += #(Import-Csv -Path $CSVFolder\$_)
}
$CSVHeader = Import-Csv 'F:\Input\Headings.hdr'
$CSV = $CSVHeader + $CSV
$CSV | Export-Csv -Path $OutputFile -NoTypeInformation -Force;
I get the list of FileNames that are exported and not the content of the Files.
The script is also modifying the date/time stamp on my INV files. It shouldn't be doing that.
You can skip the whole CSV bit if you just append the files as you would before.
Something like this should work:
# First we create the new file and add the header.
get-content $headerfile | set-content $outputfile
# Then we get the input files, read them out with get-content
# and append them to the output file (add-content).
get-childitem -path $csvfolder *.inv | get-content | add-content $outputfile
The CSV commandlets are handy if you want to be processing the CSV data in your script, but in your case simply appending the files will do the trick. Not bothering with the CSV conversion will be a lot faster as Powershell doesn't have to parse the CSV lines and create PS-objects. It's really fast with pure text though.
Another trick here is how the get-content and add-content are used in the pipeline. Since they are aware of the pipeline you can pass in file objects without having to use a foreach loop. This makes your statements a lot shorter.
How about:
get-childitem *.inv | foreach-object {
import-csv $_ -header (get-content Headings.hdr)
} | export-csv NewOutput.csv -notypeinformation