Powershell removing columns and rows from CSV - powershell
I'm having trouble making some changes to a series of CSV files, all with the same data structure. I'm trying to combine all of the files into one CSV file or one tab delimited text file (don't really mind), however each file needs to have 2 empty rows removed and two of the columns removed, below is an example:
col1,col2,col3,col4,col5,col6 <-remove
col1,col2,col3,col4,col5,col6 <-remove
col1,col2,col3,col4,col5,col6
col1,col2,col3,col4,col5,col6
^ ^
remove remove
End Result:
col1,col2,col4,col6
col1,col2,col4,col6
This is my attempt at doing this (I'm very new to Powershell)
$ListofFiles = "example.csv" #this is an list of all the CSV files
ForEach ($file in $ListofFiles)
{
$content = Get-Content ($file)
$content = $content[2..($content.Count)]
$contentArray = #()
[string[]]$contentArray = $content -split ","
$content = $content[0..2 + 4 + 6]
Add-Content '...\output.txt' $content
}
Where am I going wrong here...
your example file should be read, before foreach to fetch the file list
$ListofFiles = get-content "example.csv"
Inside the foreach you are getting content of mainfile
$content = Get-Content ($ListofFiles)
instead of
$content = Get-Content $file
and for removing rows i will recommend this:
$obj = get-content C:\t.csv | select -Index 0,1,3
for removing columns (column numbers 0,1,3,5):
$obj | %{(($_.split(","))[0,1,3,5]) -join "," } | out-file test.csv -Append
According to the fact the initial files looks like
col1,col2,col3,col4,col5,col6
col1,col2,col3,col4,col5,col6
,,,,,
,,,,,
You can also try this one liner
Import-Csv D:\temp\*.csv -Header 'C1','C2','C3','C4','C5','C6' | where {$_.c1 -ne ''} | select -Property 'C1','C2','C5' | Export-Csv 'd:\temp\final.csv' -NoTypeInformation
According to the fact that you CSVs have all the same structure, you can directly open them providing the header, then remove objects with the missing datas then export all the object in a csv file.
It is sufficient to specify fictitious column names, with a column number that can exceed the number of columns in the file, change where you want and exclude columns that you do not want to take.
gci "c:\yourdirwithcsv" -file -filter *.csv |
%{ Import-Csv $_.FullName -Header C1,C2,C3,C4,C5,C6 |
where C1 -ne '' |
select -ExcludeProperty C3, C4 |
export-csv "c:\temp\merged.csv" -NoTypeInformation
}
Related
need take the first two characters from cloumn 6 and append with PLedger and place end of the row, it should work for all rows in csv file
need take the first two characters from cloumn 6 and append with PLedger and place end of the row, it should work for all rows in csv file. Source file: Row1----- NEW,,2019/11/30,EPBCS,Bonus Accrual,USD,2019/11/06,A,4007,,9999,,,,,,,,,,,,,,,,,,,,,,,,,12637.349999999999,,,Bonus_Accrual_Nov2019,,,Bonus_Accrual_Nov2019,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, Row2----- NEW,,2019/11/30,EPBCS,Bonus Accrual,JPY,2019/11/06,A,2002,9999,,,,,,,,,,,,,,,,,,,,,,,,,320356.8225,,,Bonus_Accrual_Nov2019,,,Bonus_Accrual_Nov2019,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, Expected output file: NEW,,2019/11/30,EPBCS,Bonus Accrual,USD,2019/11/06,A,4007,,9999,,,,,,,,,,,,,,,,,,,,,,,,,12637.349999999999,,,Bonus_Accrual_Nov2019,,,Bonus_Accrual_Nov2019,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,PLedger US,, Row2----- NEW,,2019/11/30,EPBCS,Bonus Accrual,JPY,2019/11/06,A,2002,,9999,,,,,,,,,,,,,,,,,,,,,,,,,320356.8225,,,Bonus_Accrual_Nov2019,,,Bonus_Accrual_Nov2019,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,PLedger JP,,
You can do something like the following: $headers = 1..89 | Foreach-Object { "col{0}" -f $_} $inFile = Import-Csv file.csv -Header $headers foreach ($row in $inFile) { $row.col89 = "PLedger {0}" -f $row.col6.substring(0,2) } # This creates a true CSV output. # Skip to the next line if you want no headers and unqualified text $inFile | Export-Csv output.csv -NoType # Only use this code if you want no quotes and no headers ($inFile | ConvertTo-Csv -NoType | Select-Object -Skip 1) -replace '"' | Set-Content output.csv Since you did not provide headers for your CSV rows, I created them using the format col1 through col89. The foreach loop allows $row to be a reference of the current item in the $inFile array. So when $row is updated, the corresponding item in $inFile is also updated. Export-Csv creates an output file with the manufactured headers.
Powershell Remove spaces in the header only of a csv
First line of csv looks like this spaces are at after Path as well author ,Revision ,Date ,SVNFolder ,Rev,Status,Path I am trying to remove spaces only and rest of the content will be the same . author,Revision,Date,SVNFolder,Rev,Status,Path I tried below Import-CSV .\script.csv | ForEach-Object {$_.Trimend()}
expanding on the comment with an example since it looks like you may be new: $text = get-content .\script.csv $text[0] = $text[0] -replace " ", "" $csv = $text | ConvertFrom-CSV
Note: The solutions below avoid loading the entire CSV file into memory. First, get the header row and fix it by removing all whitespace from it: $header = (Get-Content -TotalCount 1 .\script.csv) -replace '\s+' If you want to rewrite the CSV file to fix its header problem: # Write the corrected header and the remaining lines to the output file. # Note: I'm outputting to a *new* file, to be safe. # If the file fits into memory as a whole, you can enclose # Get-Content ... | Select-Object ... in (...) and write back to the # input file, but note that there's a small risk of data loss, if # writing back gets interrupted. & { $header; Get-Content .\script.csv | Select-Object -Skip 1 } | Set-content -Encoding utf8 .\fixed.csv Note: I've chosen -Encoding utf8 as the example output character encoding; adjust as needed; note that the default is ASCII(!), which can result in data loss. If you just want to import the CSV using the fixed headers: & { $header; Get-Content .\script.csv | Select-Object -Skip 1 } | ConvertFrom-Csv As for what you tried: Import-Csv uses the column names in the header as property names of the custom objects it constructs from the input rows. This property names are locked in at the time of reading the file, and cannot be changed later - unless you explicitly construct new custom objects from the old ones with the property names trimmed. Import-Csv ... | ForEach-Object {$_.Trimend()} Since Import-Csv outputs [pscustomobject] instances, reflected one by one in $_ in the ForEach-Object block, your code tries call .TrimEnd() directly on them, which will fail (because it is only [string] instances that have such a method). Aside from that, as stated, your goal is to trim the property names of these objects, and that cannot be done without constructing new objects.
Read the whole file into an array: $a = Get-Content test.txt Replace the spaces in the first array element ([0]) with empty strings: $a[0] = $a[0] -replace " ", "" Write over the original file: (Don't forget backups!) $a | Set-Content test.txt
$inFilePath = "C:\temp\headerwithspaces.csv" $content = Get-Content $inFilePath $csvColumnNames = ($content | Select-Object -First 1) -Replace '\s','' $csvColumnNames = $csvColumnNames -Replace '\s','' $remainingFile = ($content | Select-Object -Skip 1)
How can I shift column values and add new ones in a CSV
I have to create a new column in my CSV data with PowerShell. There is my code: $csv = Import-Csv .\test1.csv -Delimiter ';' $NewCSVObject = #() foreach ($item in $csv) { $NewCSVObject += $item | Add-Member -name "ref" -value " " -MemberType NoteProperty } $NewCSVObject | export-csv -Path ".\test2.csv" -NoType $csv | Export-CSV -Path ".\test2.csv" -NoTypeInformation -Delimiter ";" -Append When I open the file, the column is here but a the right and I would like to have this at the left like column A. And I don't know if I can export the two object in one line like this (it doesn't work): $csv,$NewCSVObject | Export-CSV -Path ".\test2.csv" -NoTypeInformation -Delimiter ";" -Append The input file (It would have more lines than just the one): A B C D E F G H T-89 T-75 T-22 Y-23 Y-7 Y-71 The current output file: A B C D E F G H Y-23 Y-7 Y-71 ref: ref2: The expected result in the Excel table, display "ref:" and "ref:2" before the product columns: A B C D E F G H ref: T-89 T-75 T-22 ref2: Y-23 Y-7 Y-71
This might be simpler if we just treat the file as a flat text file and save it in a csv format. You could use the csv objects and shift the values into other rows but that is not really necessary. Your approach of adding columns via Add-Member is not accomplishing this goal as it will be adding new columns and would not match your desired output. Export-CSV wants to write to file objects with the same properties as well which you were mixing which gave your unexpected results. This is a verbose way of doing this. You could shorten this easily with something like regular expressions (see below). I opted for this method since it is a little easier to follow what is going on. # Equivelent to Get-Content $filepath. This just shows what I am doing and is a portable solution. $fileContents = "A;B;C;D;E;F;G;H", "T-89;T-75;T-22;Y-23;Y-7;Y-71", "T-89;T-75;T-22;Y-23;Y-7;Y-71" $newFile = "C:\temp\csv.csv" # Write the header to the output file. $fileContents[0] | Set-Content $newFile # Process the rest of the lines. $fileContents | Select-Object -Skip 1 | ForEach-Object{ # Split the line into its elements $splitLine = $_ -split ";" # Rejoin the elements. adding the ref strings (#("ref:") + $splitLine[0..2] + "ref2:" + $splitLine[3..5]) -join ";" } | Add-Content $newFile What the last line is going is concatenating an array. Starts with "ref:" add the first 3 elements of the split line followed by "ref2:" and the remaining elements. That new array is joined on semicolons and sent down the pipe to be outputted to the file. If you are willing to give regex a shot this could be done with less code. $fileContents = Get-Content "C:\source\file\path.csv" $newFile = "C:\new\file\path.csv" $fileContents[0] | Set-Content $newFile ($fileContents | Select-Object -Skip 1) -replace "((?:.*?;){3})(.*)",'ref:;$1ref2:;$2' | Add-Content $newFile What that does is split each line beyond the first on the 3rd semicolon (Explanation). The replacement string is built from the ref strings and the matched content.
You can use Select-Object to specify order. Assuming your headers are A-H (I know that instead of A it should be ref, from the code, but not sure if T-89 etc are your other headers) $NewCSVObject | Select-Object A,B,C,D,E,F,G,H | Export-Csv -Path ".\test2.csv" -NoType
Parse line of text and match with parse of CSV
As a continuation of a script I'm running, working on the following. I have a CSV file that has formatted information, example as follows: File named Import.csv: Name,email,x,y,z \I\RS\T\Name1\c\x,email#jksjks,d,f \I\RS\T\Name2\d\f,email#jsshjs,d,f ... This file is large. I also have another file called Note.txt. Name1 Name2 Name3 ... I'm trying to get the content of Import.csv and for each line in Note.txt if the line in Note.txt matches any line in Import.csv, then copy that line into a CSV with append. Continue adding every other line that is matched. Then this loops on each line of the CSV. I need to find the best way to do it without having it import the CSV multiple times, since it is large. What I got does the opposite though, I think: $Dir = PathToFile $import = Import-Csv $Dir\import.csv $NoteFile = "$Dir\Note.txt" $Note = GC $NoteFile $Name = (($Import.Name).Split("\"))[4] foreach ($j in $import) { foreach ($i in $Note) { $j | where {$Name -eq "$i"} | Export-Csv "$Dir\Result.csv" -NoTypeInfo -Append } } This takes too long and I'm not getting the extraction I need.
This takes too long and I'm not getting the extraction I need. That's because you only assign $name once, outside of the outer foreach loop, so you're basically performing the same X comparisons for each line in the CSV. I would rewrite the nested loops as a single Where-Object filter, using the -contains operator: $Import |Where-Object {$Note -contains $_.Name.Split('\')[4]} |Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
Group the imported data by your distinguishing feature, filter the groups by name, then expand the remaining groups and write the data to the output file: Import-Csv "$Dir\import.csv" | Group-Object { $_.Name.Split('\')[4] } | Where-Object { $Note -contains $_.Name } | Select-Object -Expand Group | Export-Csv "$Dir\Result.csv" -NoType
How to change column position in powershell?
Is there any easy way how to change column position? I'm looking for a way how to move column 1 from the beginning to the and of each row and also I would like to add zero column as a second last column. Please see txt file example below. Thank you for any suggestions. File sample TEXT1,02/10/2015,55.930,57.005,55.600,56.890,1890 TEXT2,02/10/2015,51.060,52.620,50.850,52.510,4935 TEXT3,02/10/2015,50.014,50.74,55.55,52.55,5551 Output: 02/10/2015,55.930,57.005,55.600,56.890,1890,0,TEXT1 02/10/2015,51.060,52.620,50.850,52.510,4935,0,TEXT2 02/10/2015,50.014,50.74,55.55,52.55,5551,0,TEXT3
Another option: #Prepare test file (#' TEXT1,02/10/2015,55.930,57.005,55.600,56.890,1890 TEXT2,02/10/2015,51.060,52.620,50.850,52.510,4935 TEXT3,02/10/2015,50.014,50.74,55.55,52.55,5551 '#).split("`n") | foreach {$_.trim()} | sc testfile.txt #Script starts here $file = 'testfile.txt' (get-content $file -ReadCount 0) | foreach { '{1},{2},{3},{4},{5},{6},0,{0}' -f $_.split(',') } | Set-Content $file #End of script #show results get-content $file 02/10/2015,55.930,57.005,55.600,56.890,1890,0,TEXT1 02/10/2015,51.060,52.620,50.850,52.510,4935,0,TEXT2 02/10/2015,50.014,50.74,55.55,52.55,5551,0,TEXT3
Sure, split on commas, spit the results back minus the first result joined by commas, add a 0, and then add the first result to the end and join the whole thing with commas. Something like: $Input = #" TEXT1,02/10/2015,55.930,57.005,55.600,56.890,1890 TEXT2,02/10/2015,51.060,52.620,50.850,52.510,4935 TEXT3,02/10/2015,50.014,50.74,55.55,52.55,5551 "# -split "`n"|ForEach{$_.trim()} $Input|ForEach{ $split = $_.split(',') ($Split[1..($split.count-1)]-join ','),0,$split[0] -join ',' }
I created file test.txt to contain your sample data. I Assigned each field a name, "one","two","three" etc so that i could select them by name, then just selected and exported back to csv in the order you wanted. First, add the zero to the end, it will end up as second last. gc .\test.txt | %{ "$_,0" } | Out-File test1.txt Then, rearrange order. Import-Csv .\test.txt -Header "one","two","three","four","five","six","seven","eight" | Select-Object -Property two,three,four,five,six,seven,eight,one | Export-Csv test2.txt -NoTypeInformation This will take the output file and get rid of quotes and header line if you would rather not have them. gc .\test2.txt | %{ $_.replace('"','')} | Select-Object -Skip 1 | out-file test3.txt