Perl Most effective way for scanning for a particular web server http banner? - perl

Question

So basically I'm trying to scan web servers that run for example version apache 2.2.4 on their web server, what's the best way of doing this?
Scan for IP's range from blah blah to blah blah, with port 80 open + web server enabled then just make a script that loads ips and checks to see if they have the server banner i want.
Or what's an alternative faster way?
Basically I'm trying to make a script like ShodanHQ.
I'm trying to get a large amount of web servers running a certain version, can anybody give me a direction, thanks hope i was clear.

Answer 1

For doing Internet-wide surveys like Shodan or Scans.io, you need very-high-bandwidth access, legal approval (or at least a blind eye turned) from your ISP, and likely an asynchronous scanner like Zmap or masscan. Nmap is a decent alternative with the --min-rate argument. Anything using the default TCP stack on your OS (e.g. curl, netcat, or Perl solutions) will not be able to keep up with the high packet volume and number of targets required.
If, however, you want to scan a smaller network (say a /16 with 65K addresses), then Nmap is up to the job, requires less setup than the asynchronous scanners (since they require firewall settings to prevent the native TCP stack from responding to returned probes), and is widely available. You could get reasonable performance with this command:
sudo nmap -v -T5 -PS80 -p80 -sS --script http-server-header -oA scan-results-%D%T 10.10.0.0/16
This breaks down to:
-v - verbose output
-T5 - Fastest timing options. This may be too much for some networks; try -T4 if you suspect lost results.
-PS80 - Only consider hosts that respond on port 80 (open or closed).
-p80 - Scan port 80 on alive hosts
-sS - Use Nmap's half-open SYN scan, which has the best timing performance
--script http-server-header - This script will grab the Server header from a basic GET request. Alternatively you could use http-headers to get all headers, or use -sV --version-light to do basic version detection from probe responses.
-oA scan-results-%D%T - Output 3 formats into separate timestamped files. You can process results with one of the many tools that imports Nmap XML output.

Answer 2

You could use curl and sed:
curl -sI 'http://192.0.2.1' | sed -n 's/^Server:[[:blank:]]*//p'
Call it from perl with:
perl -e '$server=`curl -sI 'http://192.0.2.1' | sed -n 's/^Server:[[:blank:]]*//p'`; print $server'
The -I option in curl prints the http headers using a HEAD request.