Suppose I have two csv files. One is
id_number,location_code,category,animal,quantity
12212,3,4,cat,2
29889,7,6,dog,2
98900,
33221,1,8,squirrel,1
the second one is:
98900,2,1,gerbil,1
The second file may have a newline or something at the end (maybe or maybe not, I haven't checked), but only the one line of content. There may be three or four or more different varieties of the "second" file, but each one will have a first element (98900 in this example) that corresponds to an incomplete line in the first file similar to what is in this example.
Is there a way using powershell to automatically merge the line in the second (plus any additional similar) csv file into the matching line(s) of the first file, so that the resulting file is:
12212,3,4,cat,2
29889,7,6,dog,2
98900,2,1,gerbil,1
33221,1,8,squirrel,1
main.csv
id_number,location_code,category,animal,quantity
12212,3,4,cat,2
29889,7,6,dog,2
98900,
33221,1,8,squirrel,1
correction_001.csv
98900,2,1,gerbil,1
merge code used at the commandline, or in the .ps1 file of your choice
$myHeader = #('id_number','location_code','category','animal','quantity')
#Stage all the correction files: last correction in the most recent file wins
$ToFix = #{}
filter Plumbing_Import-Csv($Header){import-csv -LiteralPath $_ -Header $Header}
ls correction*.csv | sort -Property LastWriteTime | Plumbing_Import-Csv $myHeader | %{$ToFix[$_.id_number]=$_}
function myObjPipe($Header){
begin{
function TextTo-CsvField([String]$text){
#text fields which contain comma, double quotes, or new-line are a special case for CSV fields and need to be accounted for
if($text -match '"|,|\n'){return '"'+($text -replace '"','""')+'"'}
return $text
}
function myObjTo-CsvRecord($obj){
return ''+
$obj.id_number +','+
$obj.location_code +','+
$obj.category +','+
(TextTo-CsvField $obj.animal)+','+
$obj.quantity
}
$Header -join ','
}
process{
if($ToFix.Contains($_.id_number)){
$out = $ToFix[$_.id_number]
$ToFix.Remove($_.id_number)
}else{$out = $_}
myObjTo-CsvRecord $out
}
end{
#I assume you'd append any leftover fixes that weren't used
foreach($out in $ToFix.Values){
myObjTo-CsvRecord $out
}
}
}
import-csv main.csv | myObjPipe $myHeader | sc combined.csv -encoding ascii
You could also use ConvertTo-Csv, but my preference is to not have all the extra " cruft.
Edit 1: reduced code redundancy, accounted for \n, fixed appends, and used #OwlsSleeping suggestion about the -Header commandlet parameter
also works with these files:
correction_002.csv
98900,2,1,I Win,1
correction_new.csv
98901,2,1,godzilla,1
correction_too.csv
98902,2,1,gamera,1
98903,2,1,mothra,1
Edit 2: convert gc | ConvertTo-Csv over to Import-Csv to fix the front-end \n issues. Now also works with:
correction_003.csv
29889,7,6,"""bad""
monkey",2
This is a simple solution assuming there's always exactly one match, and you don't care about output order. Change the output path to csv1 to overwrite.
I added headers manually in both input files, but you can specify them in Import-Csv instead if you'd rather avoid changing your files.
[array]$MissingLine = Import-Csv -Path "C:\Users\me\Documents\csv2.csv"
[string]$MissingId = $MissingLine[0].id_number
[array]$BigCsv = Import-Csv -Path "C:\Users\me\Documents\csv1.csv" |
Where-Object {$_.id_number -ne $MissingId}
($BigCsv + $MissingLine) |
Export-Csv -Path "C:\Users\me\Documents\Combined.csv"
I would start with I was looking for answers but I could not find one.
The code is this but it reads in 1 line from the data.txt and after that all lines in message.txt.
The message.txt contains strings and data.txt has these separeted with ';' like this:
Josh Adams;NYC 5th Avenue;18:25
Peter Nordmann;NCY 5th Avenue;20:00
...
foreach ($dataLine in Get-Content .\data.txt){
$data = $dataLine -split ';'
foreach($line in Get-Content .\message.txt){
$line.replace('<name>', $data[0])
$line.replace('<place>', $data[1])
$line.replace('<time>', $data[2])
}
Write-Output $line
}
Just capture your searched text content to a variable, use that variable content to write to the new or the other file, using the normal built-in cmdlet(s)...
Add-Content
or
Out-File
.. using the append options.
Lastly, there are multiple ways to save content to a variable, for use elsewhere and output to the screen.
3 ways to store and display PowerShell Variable simultaneously
# Using -OutVariable parameter
Get-Process a* -OutVariable process
# PowerShell Variable squeezing
($process = Get-Process a*)
# Using Tee-Object Cmdlet
Get-Process a* | Tee-Object -Variable p
I have a file 'abc.txt' that contains below lines.
c:myfilepath\filepath\filepath1\file1.csv
c:myfilepath\filepath\filepath1\file2.csv
c:myfilepath\filepath\filepath1\file2.csv
How to loop through the above file 'abc.txt' and read line by line and create another file called 'xyz.txt' that should contains like below. The file name in the path in 'xyz.txt' should be different, see below (ex. newfile_file1.txt)
c:mynewfile\newfilepath\newfilepath1\newfile_file1.txt (<-This is
corresponding to file1.csv)
c:mynewfile\newfilepath\newfilepath1\newfile_file2.txt
c:mynewfile\newfilepath\newfilepath1\newfile_file2.txt
I've tried using Get-Content to loop through the file but I just get nothing returned. I'm unclear as to where to put the syntax and how to completely construct it.
This should do it (edited to get file names and paths as requested, and dynamic so the paths in the abc-file are used).
$f = Get-Content C:\temp\abc.txt # this is the contents-file
foreach ($r in $f)
{
$r2 = (Split-Path $r).Replace("\", "\new") + '\newfile_' + [io.path]::GetFileNameWithoutExtension($r) + '.txt'
$r2 = $r2.replace(":\", ":\mynewfile\")
Get-Content $r | Out-File -filepath $r2
}
Assuming all of your file paths start with c:myfilepath\filepath\filepath1, then you can just replace the string then Out-File it.
$File1 = get-content E:\abc.txt
$File1 -replace ('c:myfilepath\\filepath\\filepath1\\', 'c:mynewfile\newfilepath\newfilepath1\newfile_') |
Out-File E:\xyz.txt
Note the double backslashes \\ which escape the regex.
I'm trying to find out how to use powershell to find and delete lines without certain string pattern in a set of files. For example, I have the following text file:
111111
22x222
333333
44x444
This needs to be turned into:
22x222
44x444
given that the string pattern of 'x' is not in any of the other lines.
How can I issue such a command in powershell to process a bunch of text files?
thanks.
dir | foreach { $out = cat $_ | select-string x; $out | set-content $_ }
The dir command lists the files in the current directory; the foreach goes through each file; cat reads the file and pipes into select-string; select-string finds the lines that contains the specific pattern, which in this case is "x"; the result of select-string is stored in $out; and finally, $out is written to the same file with set-content.
We need the temporary variable $out because you cannot read and write the same file at the same time.
This will process all txt files from the working directory. Each file content is checked and only lines that have 'x' in them are allowed to pass on. The result is written back to the file.
Get-ChildItem *.txt | ForEach-Object{
$content = Get-Content $_.FullName | Where-Object {$_ -match 'x'}
$content | Out-File $_.FullName
}
I have a file called one_to_many.txt. In the file is the data:
a,aaa
b,bbb
c,ccc
I want to use powershell to create 3 files a.txt, b.txt and c.txt
File a.txt to contain data "aaa", b to contain "bbb ...
I'm new to Powershell and can't work out how to do it. Thanks.
One possible solution:
Get-Content one_to_many.txt | Foreach-Object {
$fileName, $content = $_ -split ','
$content | Set-Content "$fileName.txt"
}
Get-Content reads the file and returns lines, one by one. Each line is piped to Foreach-Object where it can be accessed as $_.
You need to split the line by comma, so invoke operator -split which returns in this case array of 2 items. The assignment $filename, $content = .. causes that content of first item in array (from -split) is assigned to $filename and the rest to `$content.
Then simply store that content in the file via Set-Content