I have two CSV file with 50 column and more than 10K row. There is a column having date-time value. In some records it is 01/18/2013 18:16:32 and some records is 16/01/2014 17:32.
What I want to convert the column data like this: 01/18/2013 18:16. I want to remove seconds value. I want to do it by PowerShell script.
Sample data:
10/1/2014 13:18
10/1/2014 13:21
15/01/2014 12:03:19
15/01/2014 17:39:27
15/01/2014 18:29:44
17/01/2014 13:33:59
Since you're not going to convert to a sane date format anyway, you can just do a regex replace of that column:
Import-Csv foo.csv |
ForEach-Object {
$_.Date = $_.Date -replace '(\d+:\d+):\d+', '$1'
} |
Export-Csv -NoTypeInformation foo-new.csv
You could also go the route of using date parsing, but that's probably a bit slower:
Import-Csv foo.csv |
ForEach-Object {
$_.Date = [datetime]::Parse($_.Date).ToString('MM/dd/yyyy HH:mm')
} |
Export-Csv -NoTypeInformation foo-new.csv
If you're sure that there are no other timestamps or anything that could look like it elsewhere, you can also just replace everything that looks like it without even parsing as CSV:
$csv = Get-Content foo.csv -ReadCount 0
$csv = $csv -replace '(\d{2}/\d{2}/\d{4} \d{2}:\d{2}):\d{2}', '$1'
$csv | Out-File foo-new.csv
Related
I have a script that extracts data from an XML file and put this into an CSV file with 2 columns.
The file looks like this:
Name, Count
1,34
3,55
15,66
103,99
etc.
So far so good...
My problem is that the program that reads the CSV-file always expect 3 digits in the column "Name".
So the CSV-file need to look like this:
Name, Count
001,34
003,55
015,66
103,99
etc.
How can I do this formatting using "Export-CSV"?
Please help I'm stuck here..
There are several ways to apply the changes to the csv file.
Read/Import to a variable, change name, Export-Csv variable
$Csv = Import-Csv .\Sample.csv
$Csv | ForEach-Object{ $_.Name = $_.Name.PadLeft(3,'0') }
$Csv | Export-Csv .\NewSample.csv -NoTypeInformation
Do the same on the fly with a calculated property reusing the same header/property name.
Import-Csv .\Sample.csv |
Select-Object #{n='Name';e={$_.Name.PadLeft(3,'0')}},Count|
Export-Csv .\NewSample2.csv -NoTypeInformation
Use the -f ( format ) operator with variable
i.e.
[int] $int = 25;
"{0:D3}" -f $int
Here 3 is a number of digits and Output will be :
025
I filtered by date this file data1.csv
2017.11.1,09:55,1.1,1.2,1.3,1.4,1
2017.11.2,09:55,1.5,1.6,1.7,1.8,2
I don't get a header with -NoTypeInformation:
$CutOff = (Get-Date).AddDays(-2)
$filePath = "data1.csv"
$Data = Import-Csv $filePath -Header Date,Time,A,B,C,D,E
$Data2 = $Data | Where-Object {$_.Date -as [datetime] -gt $Cutoff} | convertto-csv -NoTypeInformation -Delimiter "," | % {$_ -replace '"',''}
But when rewriting with Out-File
$Data2 | Out-File "data2.csv" -Encoding utf8 -Force
I get header back as data2.csv contains:
Date,Time,A,B,C,D,E
2017.11.2,09:55,1.5,1.6,1.7,1.8,2
Why do I have Date,Time,A,B,C,D,E ?
-NoTypeInformation is not about the header but the data type of the rows in the file. Remove it to see what shows up. From Microsoft
Omits the type information header from the output. By default, the string in the output contains #TYPE followed by the fully-qualified name of the object type.
Emphasis mine.
CSVs need headers. That is why it is making one. If you don't want to see the header in the output use Select-Object -Skip 1 to remove it.
$Data |
Where-Object {$_.Date -as [datetime] -gt $Cutoff} |
ConvertTo-CSV -NoTypeInformation -Delimiter "," |
Select-Object -Skip 1 |
% {$_ -replace '"'}
I would not pipe Out-File to itself. You could pipe to Set-Content here just as well.
I am guessing this whole process is to keep the source file in the same state just with some lines filtered out based on date. You could skip most of this just by parsing the date out in each line.
$threshold = (Get-Date).AddDays(-2)
$filePath = "c:\temp\bagel.txt"
(Get-Content $filePath) | Where-Object{
$date,$null=$_.Split(",",2)
[datetime]$date -gt $threshold
} | Set-Content $filePath
Now you don't have to worry about PowerShell CSV object structure or output since we act on the raw data of the file itself.
That will take each line of the input file and filter it out if the parsed date does not match the threshold. Change encoding on the input output cmdlets as you see necessary. What $date,$null=$_.Split(",",2) is doing is splitting the line
on the comma into 2 parts. First of which becomes $date and since this is just a filtering condition we dump the rest of the line into $null.
Properly-formed CSV files must have column headers. Your use of -NoTypeInformation in generating the CSV does not affect column headers; instead, it affects whether the PowerShell object type information is included. If you Export-CSV without -NoTypeInformation, the first line of your CSV file will have a line that looks like #TYPE System.PSCustomObject, which you don't want if you're going to open the CSV in a spreadsheet program.
If you subsequently Import-CSV, the headers (Date, Time, A, B, C) are used to create the fields of a PSObject, so that you can refer to them using the standard dot notation (e.g., $CSV[$line].Date).
The ability to specify -Header on Import-CSV is essentially a "hack" to allow the cmdlet to handle files that are comma-separated, but which did not include column headers.
I have a csv file with five columns. The last column being the students grade level. What I want to do is change the value of that column based on the gradelevel. For example if the gradelevel is 12 I want to change that to 2016, 11 to 2017, so on and so forth.
update: I did get it to semi work using the below:
Get-Content users.csv | ForEach-Object -Process {$_ -replace '12','2016'} | Set-Content users1.csv
What happens is if the student id has a 12 in it that gets changed as well to 2016. Example would be 120045 gets change to 20160045
You can import the csv, loop it in a foreach and use the $_ (this) operator and then export ist to csv
Somesthing like:
# import CSV
$list = import-csv P:\ath\to\file.csv -Delimiter ";" -Encoding Standard
# % means foreach
$list | % {
# The grades is the key for each line
# the first line of your csv represent the keys you can use
# e.g. first colum: Name | Grade | Class
# $_.Name | $_.Grade | $_.Class are your keys for every entry (2nd line and above)
# here you can work with your grades an do what you want
if($_.Grade -eq 11){
# set a new value to the grade in the actual line
$_.Grade = 2016
}
}
# now export the new list into csv
export-csv P:\ath\to\new.csv -Delimiter ";" -NoTypeInformation
That should be a basic to work with.
Greetz Eldo.O
Eldo you code worked great. I did have to change the last line from:
export-csv P:\ath\to\new.csv -Delimiter ";" -NoTypeInformation
to
$list | Export-Csv P:\ath\to\new.csv -Delimiter "," -NoTypeInformation
I was also able to add more if statements to accomplish exactly what I needed.
I need to convert a file with this format:
2015.03.27,09:00,1.08764,1.08827,1.08535,1.08747,8941
2015.03.27,10:00,1.08745,1.08893,1.08604,1.08762,7558
to this format
2015.03.27,1.08764,1.08827,1.08535,1.08747,1
2015.03.27,1.08745,1.08893,1.08604,1.08762,1
I started with this code but can't see how to achieve the full transformation:
Import-Csv in.csv -Header Date,Time,O,H,L,C,V | Select-Object Date,O,H,L,C,V | Export-Csv -path out.csv -NoTypeInformation
(Get-Content out.csv) | % {$_ -replace '"', ""} | out-file -FilePath out.csv -Force -Encoding ascii
which outputs
Date,O,H,L,C,V
2015.03.27,1.08745,1.08893,1.08604,1.08762,8941
2015.03.27,1.08763,1.08911,1.08542,1.08901,7558
After that I need to
remove the header (I tried -NoHeader which is not recognized)
replace last column with 1.
How to do that as simply as possible (if possible without looping through each row)
Update : finally I have simplified requirement. I just need to replace last column with constant.
Ok, this could be one massive one-liner... I'm going to do line breaks at the pipes for sanity reasons though.
Import-Csv in.csv -header Date,Time,O,H,L,C,V|select * -ExcludeProperty time|
%{$_.date = [datetime]::ParseExact($_.date,"yyyy.MM.dd",$null).tostring("yyMMdd");$_}|
ConvertTo-Csv -NoTypeInformation|
select -skip 1|
%{$_ -replace '"'}|
Set-Content out.csv -encoding ascii
Basically I import the CSV, exclude the time column, convert the date column to an actual [datetime] object and then convert it back in the desired format. Then I pass the modified object (with the newly formatted date) down the pipe to ConvertTo-CSV, and skip the first line (your headers that you don't want), and then remove the quotes from it, and lastly output to file with Set-Content (faster than Out-File)
Edit: Just saw your update... to do that we'll just change the last column to 1 at the same time we modify the date column by adding $_.v=1;...
%{$_.date = [datetime]::ParseExact($_.date,"yyyy.MM.dd",$null).tostring("yyMMdd");$_.v=1;$_}|
Whole script modified:
Import-Csv in.csv -header Date,Time,O,H,L,C,V|select * -ExcludeProperty time|
%{$_.date = [datetime]::ParseExact($_.date,"yyyy.MM.dd",$null).tostring("yyMMdd");$_.v=1;$_}|
ConvertTo-Csv -NoTypeInformation|
select -skip 1|
%{$_ -replace '"'}|
Set-Content out.csv -encoding ascii
Oh, and this has the added benefit of not having to read the file in, write the file to the drive, read that file in, and then write the file to the drive again.
I'm attempting to format some dates in the first column of a CSV. I would prefer to user something like powershell as I plan to automate this task. Does anyone have any advice on the best way to change the format of the date from something like MM/DD/YYY to YYYY-MM-DD? I've tried something like this:
$date = date -f ('yyyyMMdd')
$HMDA = Import-Csv "C:\HMDA\$date.YieldTableFixed.csv"
ForEach-Object {
$HMDA.Date = [datetime]::ParseExact($HMDA.Date).ToString('YYYY-MM-DD')
} |
Export-Csv -NoTypeInformation C:\HMDA\test.csv
Unfortunately, that didn't seem to do anything but give me a parse error and I can't seem to figure out why that is. Is there a way I can say something like:
ForEach-Object{
$HMDA.A2:$HMDA.A63 = HMDA.$AC.Date.Format('YYYY-MM-DD')
}
Ok, there's some basic errors here, but that's just a matter of not knowing better I think. Now this is hard to answer accurately because you did not give us an example of the incoming date field, so if it has some strange formatting this may throw errors as PowerShell fails to recognize that a string is in fact a date.
First off, if you pipe to a ForEach loop you reference the current object with $_. Such as:
Import-Csv "C:\HMDA\$date.YieldTableFixed.csv" | ForEach-Object {
$_.Date = get-date $_.Date -f 'yyyy-MM-dd'
} | Export-Csv -NoTypeInformation C:\HMDA\test.csv
What would probably be simpler, as I recently learned from somebody else here on SO, would be to use Select, create the updated property on the fly, and then exclude the original property, effectively replacing it with the new one. Something like this:
Import-Csv "C:\HMDA\$date.YieldTableFixed.csv" |
Select *,#{label = 'Date';expression={get-date $_.Date -f 'yyyy-MM-dd'}} -ExcludeProperty Date |
Export-Csv -NoTypeInformation C:\HMDA\test.csv
ParseExact() expects 3 parameters: the date string, a format string, and a format provider (which may be $null). Also, your output format string is incorrect (the year and day format specifiers need to be lowercase), and ForEach-Object reads from a pipeline.
Change this:
$HMDA = Import-Csv "C:\HMDA\$date.YieldTableFixed.csv"
ForEach-Object {
$HMDA.Date = [datetime]::ParseExact($HMDA.Date).ToString('YYYY-MM-DD')
} |
Export-Csv -NoTypeInformation C:\HMDA\test.csv
into this:
Import-Csv 'C:\HMDA\$date.YieldTableFixed.csv' | ForEach-Object {
$_.Date = [DateTime]::ParseExact($_.Date, '*M\/dd\/yyyy', $null).ToString('yyyy-MM-dd')
$_
} | Export-Csv -NoTypeInformation 'C:\HMDA\test.csv'