Powershell output file missing lines from input file - powershell

Why would the output file produced by the following script not contain all of the lines that were contained in the original file? I'm doing replacement logic on a line by line level, I'm not explicitely removing any lines though.
[regex]$r = "\$\%\$##(.+)\$\%\$##";
(Get-Content $inputFile) |
Foreach-Object {
$line = $_;
$find = $r.matches($line);
if ($find[0].Success) {
foreach ($match in $find) {
$found = $match.value
$replace = $found -replace "[^A-Za-z0-9\s]", "";
$line.Replace($found, $replace);
}
}
} |
Set-Content $outputFile
Input File
Output File

You're only outputting content to the pipe if it finds a match, at this line:
$line.Replace($found, $replace)
If there was not a match found, then you need to output the line without doing any replacement:
[regex]$r = "\$\%\$##(.+)\$\%\$##";
(Get-Content $inputFile) |
Foreach-Object {
$line = $_;
$find = $r.matches($line);
if ($find[0].Success) {
foreach ($match in $find) {
$found = $match.value
$replace = $found -replace "[^A-Za-z0-9\s]", "";
$line.Replace($found, $replace);
}
}
Else { $line }
} |
Set-Content $outputFile

Related

Blank result for String concatenation in foreach

There might not be the best way but what is wrong with this code. Why do i get blank result from $str in the foreach loop, whereas if it try to concatenate individual cells i get the right result.
$csv = import-csv -Path "C:\Users\abc\csv4script\TEST.csv" -Header 'IPs'
$str = ''
$x = 1
foreach ($cell in $csv.){
if ($x -le 4000){
$str = $str + ", " + $cell.IPs
if ($x -eq 4000){
$x = 1}
$str = ''
}
$x = $x + 1
}
$str
# $str = $str + $csv[1].IPs + ", " + $csv[2].IPs
$str
It doesn't have a value because you misplaced a '}' character.
$x = 1}
Move it to after $str = ''
$x = 1
$str = ''}
Also as Thomas pointed out, remove the period after $csv variable in your foreach statement, it should be like: foreach($cell in $csv)
Also if you want to increment a value,
$x++ is quicker and easier to read than $x = $x + 1
Lastly, as you have it, if your list has more than 4000 IPs the $str variable will empty out.

How to count occurrence of each word on each line?

I'm stuck as I have to use Powershell at work. I've attached my code and results so far below.
$data = Get-Content "/Users/mikeshobes/Documents/Powershell/nfl.csv"
write-host $data.count total lines read from file
foreach ($line in $data)
{
write-host $line
}
13 total lines read from file
1,Tom Brady,NE,QB,93,142,65.5,47.3,"1,137",8,379,7,3,55,38.7,48,17,3,9,97.7
2,Matt Ryan,ATL,QB,70,98,71.4,32.7,"1,014",10.3,338,9,0,54,55.1,73T,13,2,8,135.3
3,Aaron Rodgers,GB,QB,80,128,62.5,42.7,"1,004",7.8,334.7,9,2,53,41.4,42T,17,1,10,103.8
4,Ben Roethlisberger,PIT,QB,64,96,66.7,32,735,7.7,245,3,4,35,36.5,62T,8,3,2,82.6
5,Russell Wilson,SEA,QB,40,60,66.7,30,449,7.5,224.5,4,2,22,36.7,42,5,2,6,97.2
6,Dak Prescott,DAL,QB,24,38,63.2,38,302,7.9,302,3,1,16,42.1,40T,3,1,2,103.2
7,Eli Manning,NYG,QB,23,44,52.3,44,299,6.8,299,1,1,12,27.3,51,3,2,2,72.1
8,Matt Moore,MIA,QB,29,36,80.6,36,289,8,289,1,1,16,44.4,37,3,0,5,97.8
9,Matthew Stafford,DET,QB,18,32,56.3,32,205,6.4,205,0,0,10,31.3,30,3,0,3,75.7
10,Alex Smith,KC,QB,20,34,58.8,34,172,5.1,172,1,1,9,26.5,24,3,0,1,69.7
11,Brock Osweiler,HOU,QB,14,25,56,25,168,6.7,168,1,0,9,36,38,1,0,0,90.1
12,Connor Cook,OAK,QB,18,45,40,45,161,3.6,161,1,3,11,24.4,20,1,0,3,30
13,Julian Edelman,NE,WR,0,1,0,0.3,0,0,0,0,0,0,0,--,0,0,0,39.6
if what you want to get is the amount of times every word in a line is in that line, first you need to split the line in words an then get the amount of times that word is in that line.
I think a snippet of code explain it better:
For ($i = 0; $i -lt $Data.Length; $i++) {
$Line = $Data[$i]
Foreach ($Word in $Line.Split(' ,')) {
Write-Host ('Line {0} contains the word: "{1}" {2} time(s)' -f ($i + 1), $Word, (($Line -split $Word).Count-1))
}
}

How do edit the last occurrence of a particular string in powershell

My text file contains G-Code with the code "G94" appearing 5 times at different line numbers.
G94
G94
G94
G94
G94
I need to change the last occurrence of "G94" to
G94
/M16
but I keep getting no edit at all.
I'm trying this:
$text = get-content C:\cncm\g94.txt
$i = 1
$replace = 5 #Replace the 5th match
ForEach ( $match in ($text | select-String "G94" -AllMatches).matches)
{
$index = $match.Index
if ( $i -eq $replace )
{
$text.Remove($index,"G94".length).Insert($index,"G94 n /M16")
}
$i++
}
What am I missing?
$text is an array of strings, how are you calling Remove() without getting an exception? First because Remove() only takes one parameter, second because you can't remove from a fixed length array.
I'm thinking:
$text = get-content C:\cncm\g94.txt
$fifthMatch = ($text | select-string "G94" -AllMatches)[4]
$line = $text[$fifthMatch.LineNumber]
$line = $line.Remove($fifthMatch.index,"G94".length).Insert($fifthMatch.index,"G94 `n /M16")
$text[$fifthMatch.LineNumber] = $line
$text | out-file c:\cncm\g942.txt
Use regexp with negative lookahead on a string that contains the entire file.
Replacing last occurrence in the entire file - (?s) DOTALL mode allows .* to span across new line characters:
$text = [IO.File]::ReadAllText('C:\cncm\g94.txt')
$text = $text -replace '(?s)G94(?!.*?G94)', "G94`n/M16"
Replacing last occurrence in every line - (?m) MULTILINE mode:
$text = [IO.File]::ReadAllText('C:\cncm\g94.txt')
$text = $text -replace '(?m)G94(?!.*?G94)', "G94`n/M16"

Why is my string still empty?

I'm wondering why my string is still empty when just hitting enter after this code:
$file = <>;
if ($file eq "") {
$file = "test.txt";
}
print "$file";
If I type in anything, it is presented by the print command, but when I just hit enter, nothing is printed out. What I want is for perl to understand when the user inputs nothing and automatically edit the string to, in this case, test.txt. What am I doing wrong?
Thanks!
Because "enter" is not equal to nothing -- it's equal to \n.
$file = <>;
if ($file eq "") {
$file = "test.txt";
}
print length($file);
Run this, hit ENTER, and watch as you get -- 1!
Try:
$file = <>;
if ($file eq "\n") {
$file = "test.txt";
}
print "$file";
Bear in mind that \n isn't portable across systems. What you really want is something like:
$file = <>;
if ($file =~ /^\s*$/) {
$file = "test.txt";
}
print $file;
to match on whitespace.
To get the result you want you have to chomp your line.
use strict;
use warnings;
chomp(my $file = <>); #remove newline. $file will have empty string if only a newline was entered.
if ($file eq "") {
$file = "test.txt";
}
print "$file\n";

Powershell / Perl : Merging multiple CSV files into one?

I have the following CSV files, I want to merge these into a single CSV
01.csv
apples,48,12,7
pear,17,16,2
orange,22,6,1
02.csv
apples,51,8,6
grape,87,42,12
pear,22,3,7
03.csv
apples,11,12,13
grape,81,5,8
pear,11,5,6
04.csv
apples,14,12,8
orange,5,7,9
Desired output:
apples,48,12,7,51,8,6,11,12,13,14,12,8
grape,,,87,42,12,81,5,8,,,
pear,17,16,2,22,3,7,11,5,6,,,
orange,22,6,1,,,,,,5,7,9
Can anyone provide guidance on how to achieve this? Preferably using Powershell but open to alternatives like Perl if that's easier.
Thanks Pantik, your code's output is close to what I want:
apples,48,12,7,51,8,6,11,12,13,14,12,8
grape,87,42,12,81,5,8
orange,22,6,1,5,7,9
pear,17,16,2,22,3,7,11,5,6
Unfortunately I need "placeholder" commas in place for when the entry is NOT present in a CSV file, e.g. orange,22,6,1,,,,,,5,7,9 rather than orange,22,6,1,5,7,9
UPDATE: I would like these parsed in order of the filenames, e.g.:
$myFiles = #(gci *.csv) | sort Name
foreach ($file in $myFiles){
regards
ted
Here is my Perl version:
use strict;
use warnings;
my $filenum = 0;
my ( %fruits, %data );
foreach my $file ( sort glob("*.csv") ) {
$filenum++;
open my $fh, "<", $file or die $!;
while ( my $line = <$fh> ) {
chomp $line;
my ( $fruit, #values ) = split /,/, $line;
$fruits{$fruit} = 1;
$data{$filenum}{$fruit} = \#values;
}
close $fh;
}
foreach my $fruit ( sort keys %fruits ) {
print $fruit, ",", join( ",", map { $data{$_}{$fruit} ? #{ $data{$_}{$fruit} } : ",," } 1 .. $filenum ), "\n";
}
Which gives me:
apples,48,12,7,51,8,6,11,12,13,14,12,8
grape,,,,87,42,12,81,5,8,,,
orange,22,6,1,,,,,,,5,7,9
pear,17,16,2,22,3,7,11,5,6,,,
So do you have a typo for grape or i have misunderstood something?
Ok, gangabass solution works, and is cooler than mine, but I'll add mine anyway. It is slightly stricter, and preserves a data structure that can be used as well. So, enjoy. ;)
use strict;
use warnings;
opendir my $dir, '.' or die $!;
my #csv = grep (/^\d+\.csv$/i, readdir $dir);
closedir $dir;
# sorting numerically based on leading digits in filename
#csv = sort {($a=~/^(\d+)/)[0] <=> ($b=~/^(\d+)/)[0]} #csv;
my %data;
# To print empty records we first need to know all the names
for my $file (#csv) {
open my $fh, '<', $file or die $!;
while (<$fh>) {
if (m/^([^,]+),/) {
#{ $data{$1} } = ();
}
}
close $fh;
}
# Now we can fill in values
for my $file (#csv) {
open my $fh, '<', $file or die $!;
my %tmp;
while (<$fh>) {
chomp;
next if (/^\s*$/);
my ($tag,#values) = split (/,/);
$tmp{$tag} = \#values;
}
for my $key (keys %data) {
unless (defined $tmp{$key}) {
# Fill in empty values
#{$tmp{$key}} = ("","","");
}
push #{ $data{$key} }, #{ $tmp{$key} };
}
}
&myreport;
sub myreport {
for my $key (sort keys %data) {
print "$key," . (join ',', #{$data{$key}}), "\n";
}
}
Powershell:
$produce = "apples","grape","orange","pear"
$produce_hash = #{}
$produce | foreach-object {$produce_hash[$_] = #(,$_)}
$myFiles = #(gci *.csv) | sort Name
foreach ($file in $myFiles){
$file_hash = #{}
$produce | foreach-object {$file_hash[$_] = #($null,$null,$null)}
get-content $file | foreach-object{
$line = $_.split(",")
$file_hash[$line[0]] = $line[1..3]
}
$produce | foreach-object {
$produce_hash[$_] += $file_hash[$_]
}
}
$ofs = ","
$out = #()
$produce | foreach-object {
$out += [string]$produce_hash[$_]
}
$out | out-file "outputfile.csv"
gc outputfile.csv
apples,48,12,7,51,8,6,11,12,13,14,12,8
grape,,,,87,42,12,81,5,8,,,
orange,22,6,1,,,,,,,5,7,9
pear,17,16,2,22,3,7,11,5,6,,,
Should be easy to modify for additional items. Just add them to the $produce array.
Second Powershell solution (as requested)
$produce = #()
$produce_hash = #{}
$file_count = -1
$myFiles = #(gci 0*.csv) | sort Name
foreach ($file in $myFiles){
$file_count ++
$file_hash = #{}
get-content $file | foreach-object{
$line = $_.split(",")
if ($produce -contains $line[0]){
$file_hash[$line[0]] += $line[1..3]
}
else {
$produce += $line[0]
$file_hash[$line[0]] = #(,$line[0]) + (#($null) * 3 * $file_count) + $line[1..3]
}
}
$produce | foreach-object {
if ($file_hash[$_]){$produce_hash[$_] += $file_hash[$_]}
else {$produce_hash[$_] += #(,$null) * 3}
}
}
$ofs = ","
$out = #()
$produce_hash.keys | foreach-object {
$out += [string]$produce_hash[$_]
}
$out | out-file "outputfile.csv"
gc outputfile.csv
apples,48,12,7,51,8,6,11,12,13,14,12,8
grape,,,,87,42,12,81,5,8,,,
orange,22,6,1,,,,,,,5,7,9
pear,17,16,2,22,3,7,11,5,6,,,
you have to parse the files, I don't see easier way hot to do it
solution in powershell:
UPDATE: ok, adjusted a bit - hopefully understandable
$items = #{}
$colCount = 0 # total amount of columns
# loop through all files
foreach ($file in (gci *.csv | sort Name))
{
$content = Get-Content $file
$itemsToAdd = 0; # columns added by this file
foreach ($line in $content)
{
if ($line -match "^(?<group>\w+),(?<value>.*)")
{
$group = $matches["group"]
if (-not $items.ContainsKey($group))
{ # in case the row doesn't exists add and fill with empty columns
$items.Add($group, #())
for($i = 0; $i -lt $colCount; $i++) { $items[$group] += "" }
}
# add new values to correct row
$matches["value"].Split(",") | foreach { $items[$group] += $_ }
$itemsToAdd = ($matches["value"].Split(",") | measure).Count # saves col count
}
}
# in case that file didn't contain some row, add empty cols for those rows
$colCount += $itemsToAdd
$toAddEmpty = #()
$items.Keys | ? { (($items[$_] | measure).Count -lt $colCount) } | foreach { $toAddEmpty += $_ }
foreach ($key in $toAddEmpty)
{
for($i = 0; $i -lt $itemsToAdd; $i++) { $items[$key] += "" }
}
}
# output
Remove-Item "output.csv" -ea 0
foreach ($key in $items.Keys)
{
"$key,{0}" -f [string]::Join(",", $items[$key]) | Add-Content "output.csv"
}
Output:
apples,48,12,7,51,8,6,11,12,13,14,12,8
grape,,,,87,42,12,81,5,8,,,
orange,22,6,1,,,,,,,5,7,9
pear,17,16,2,22,3,7,11,5,6,,,
Here is a more consise way how to do it. However, it still doesn't add the commas when the item is missing.
Get-ChildItem D:\temp\a\ *.csv |
Get-Content |
ForEach-Object -begin { $result=#{} } -process {
$name, $otherCols = $_ -split '(?<=\w+),'
if (!$result[$name]) { $result[$name] = #() }
$result[$name] += $otherCols
} -end {
$result.GetEnumerator() | % {
"{0},{1}" -f $_.Key, ($_.Value -join ",")
}
} | Sort