Discovering duplicate lines - perl

Question

I've got a file of CSS elements, and I'm trying to check for any duplicate CSS elements,.. then output the lines that show the dupe lines.
###Test
###ABC
###test
##.hello
##.ABC
##.test
bob.com###Test
~qwerty.com###Test
~more.com##.ABC
###Test & ##.ABC already exists in the list, and I'd like a way to output the lines that are used in the file, basically duplication checking (case sensitive). So using the above list, I would generate something like this..
Line 1: ###Test
Line 7: bob.com###Test
Line 8: ~qwerty.com###Test
Line 5: ##.ABC
Line 9: ~more.com##.ABC
Something in bash, or maybe perl?
Thanks :)

Answer 1

I've been challenged by your problem, so I wrote you a script. Hope you liked it. :)
#!/usr/bin/perl
use strict;
use warnings;
sub loadf($);
{
my #file = loadf("style.css");
my #inner = #file;
my $l0 = 0; my $l1 = 0; my $l2 = 0; my $dc = 0; my $tc;
foreach my $line (#file) {
$l1++;
$line =~ s/^\s+//;
$line =~ s/\s+$//;
foreach my $iline (#inner) {
$l2++;
$iline =~ s/^\s+//;
$iline =~ s/\s+$//;
next if ($iline eq $line);
if ($iline =~ /\b$line\b/) {
$dc++;
if ($dc > 0) {
if ($l0 == 0) {
print "Line " . $l1 . ": " . $line . "\n";
$l0++;
}
print "Line " . $l2 . ": " . $iline . "\n";
}
}
}
print "\n" unless($dc == 0);
$dc = 0; $l0 = 0; $l2 = 0;
}
}
sub loadf($) {
my #file = ( );
open(FILE, $_[0] . "\n") or die("Couldn't Open " . $_[0] . "\n");
#file = <FILE>;
close(FILE);
return #file;
}
__END__
This does exactly what you need. And sorry if it's a bit messy.

Answer 2

This seems to work:
sort -t '#' -k 2 inputfile
It groups them by the part after the # characters:
##.ABC
~more.com##.ABC
###ABC
##.hello
##.test
###test
bob.com###Test
~qwerty.com###Test
###Test
If you only want to see the unique values:
sort -t '#' -k 2 -u inputfile
Result:
##.ABC
###ABC
##.hello
##.test
###test
###Test
This pretty closely duplicates the example output in the question (it relies on some possibly GNU-specific features):
cat -n inputfile |
sed 's/^ *\([0-9]\)/Line \1:/' |
sort -t '#' -k 2 |
awk -F '#+' '{if (! seen[$2]) { \
if ( count > 1) printf "%s\n", lines; \
count = 0; \
lines = "" \
}; \
seen[$2] = 1; \
lines = lines "\n" $0; ++count}
END {if (count > 1) print lines}'
Result:
Line 5: ##.ABC
Line 9: ~more.com##.ABC
Line 1: ###Test
Line 7: bob.com###Test
Line 8: ~qwerty.com###Test

Answer 3

I'd recommend using the uniq function if you can install MoreUtils:
how-do-i-print-unique-elements-in-perl-array

Answer 4

Here is one way to do it, which is fairly easy to extend to multiple files if need be.
With this file find_dups.pl:
use warnings;
use strict;
my #lines;
while (<>) { # read input lines
s/^\s+//; s/\s+$//; # trim whitespace
push #lines, {data => $_, line => $.} if $_ # store useful data
}
#lines = sort {length $$a{data} <=> length $$b{data}} #lines; # shortest first
while (#lines) {
my ($line, #found) = shift #lines;
my $re = qr/\Q$$line{data}\E$/; # search token
#lines = grep { # extract matches from #lines
not $$_{data} =~ $re && push #found, $_
} #lines;
if (#found) { # write the report
print "line $$_{line}: $$_{data}\n" for $line, #found;
print "\n";
}
}
then perl find_dups.pl input.css prints:
line 5: ##.ABC
line 9: ~more.com##.ABC
line 1: ###Test
line 7: bob.com###Test
line 8: ~qwerty.com###Test