How can I show the steps time of wget?
you see my snapshot, in my PC, I use wget to download a http page.
you see there have DNS resolve, TCP connection, and the HTTP response.
how can I show every step time?
so I can know every step using time:
(DNS resolve 3ms)
(TCP connection 5ms)
(HTTP response 10ms)
There is no specific way of getting this from wget. But you can use the answer to this question to get something close of what you want :
Is there a Unix utility to prepend timestamps to stdin?
You have to be aware that output of wget is not going to stdout but to errors' ouput and you have to redirect it : with 2>&1
So it gives something like this :
------$ wget http://www.google.com 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2020-08-04 22:55:46 --2020-08-04 22:55:46-- http://www.google.com/
2020-08-04 22:55:46 Résolution de www.google.com (www.google.com)… 2a00:1450:4007:80c::2004, 216.58.201.228
2020-08-04 22:55:46 Connexion à www.google.com (www.google.com)|2a00:1450:4007:80c::2004|:80… connecté.
2020-08-04 22:55:46 requête HTTP transmise, en attente de la réponse… 200 OK
2020-08-04 22:55:46 Taille : non indiqué [text/html]
2020-08-04 22:55:46 Enregistre : «index.html.10»
2020-08-04 22:55:46
2020-08-04 22:55:46 0K .......... . 5,74M=0,002s
2020-08-04 22:55:46
2020-08-04 22:55:46 2020-08-04 22:55:46 (5,74 MB/s) - «index.html.10» enregistré [11886]
2020-08-04 22:55:46
Related
I'm trying to prepend the current time to a few columns pulled from each line of a growing text file, and print to the screen.
tail -f dumpfile.txt|awk '{ "date +%T"|getline curtime;print curtime, $2, $7 }'
A sample input line from the file would be:
N 7056 65433 flags 0dbc a000 OK 0
Above command prints each line from tail, however it executes date just once, time that the command line is launched:
Desired Actual
13:10:05 7050 OK 13:10:04 7050 OK
13:10:05 7051 OK 13:10:04 7051 OK
13:10:06 7052 OK 13:10:04 7052 OK
13:10:06 7053 OK 13:10:04 7053 OK
13:10:07 7054 OK 13:10:04 7054 OK
13:10:07 7055 OK 13:10:04 7055 OK
13:10:08 7056 OK 13:10:04 7056 OK
How can I rewrite this so that the date command is executed each time a new line comes in?
Thanks.
The first getline exhausts date's output (as it's just a single line), subsequent calls to getline fail and leave curtime unchanged. You need to close the file descriptor associated with date +%T every time you read from it, e.g:
tail -f dumpfile.txt | awk 'BEGIN{cmd="date +%T"} {cmd|getline curtime; print curtime,$2,$7; close(cmd)}'
With GNU awk:
| awk '{print strftime("%H:%M:%S", systime()), $2, $7}'
Output (example):
16:58:35 7056 OK
See: 9.1.5 Time Functions
I have two web servers running with One load balancer with Haproxy. I need to block IP's that are coming to my load balancer more than often. How do I check all the incoming IP's? Is there a log?
If you want to see the established connections on a Linux server, use this command (via SSH):
netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -n 10
If you want to log more verbose HAProxy activity, use this setting in haproxy.cfg:
log 127.0.0.1 local0 info
You can view the more verbose output in /var/log/haproxy_0.log
You should try this :
echo 'Client IP: '.$_SERVER["REMOTE_ADDR"];
echo 'Client IP: '.$_SERVER["HTTP_CLIENT_IP"];
These commands displays loadbalancer's IP. More at : https://serverfault.com/a/331909
I have some data from an Nmap Scan. It looks like this.
Nmap scan report for 10.16.17.34
Host is up (0.011s latency).
Not shown: 65530 closed ports
PORT STATE SERVICE
22/tcp open ssh
23/tcp open telnet
80/tcp open http
| http-headers:
| Date: THU, 30 AUG 2012 22:46:11 GMT
| Expires: THU, 30 AUG 2012 22:46:11 GMT
| Content-type: text/html
|
|_ (Request type: GET)
443/tcp open https
| ssl-enum-ciphers:
| SSLv3
| Ciphers (11)
| TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA - unknown strength
| TLS_RSA_EXPORT1024_WITH_RC4_56_SHA - unknown strength
| TLS_RSA_EXPORT_WITH_DES40_CBC_SHA - unknown strength
| TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 - unknown strength
| TLS_RSA_EXPORT_WITH_RC4_40_MD5 - unknown strength
| TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_RSA_WITH_AES_256_CBC_SHA - unknown strength
| TLS_RSA_WITH_DES_CBC_SHA - unknown strength
| TLS_RSA_WITH_RC4_128_MD5 - unknown strength
| TLS_RSA_WITH_RC4_128_SHA - strong
| TLSv1.0
| Ciphers (10)
| TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA - unknown strength
| TLS_RSA_EXPORT1024_WITH_RC4_56_SHA - unknown strength
| TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 - unknown strength
| TLS_RSA_EXPORT_WITH_RC4_40_MD5 - unknown strength
| TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_RSA_WITH_AES_256_CBC_SHA - unknown strength
| TLS_RSA_WITH_DES_CBC_SHA - unknown strength
| TLS_RSA_WITH_RC4_128_MD5 - unknown strength
| TLS_RSA_WITH_RC4_128_SHA - strong
| Compressors (1)
| NULL
|_ Least strength = unknown strength
2023/tcp open xinuexpansion3
Nmap scan report for 10.16.40.0
Host is up (0.00062s latency).
All 65535 scanned ports on 10.16.40.0 are closed
Nmap scan report for 10.16.40.1
Host is up (0.00071s latency).
All 65535 scanned ports on 10.16.40.1 are closed
What I am attempting to do is to either use Awk, Sed or Grep or something else to extract any section that starts with Nmap Scan and ends in a blank new line and has ssl-enum-ciphers in it. I figured out with Awk how to print each section but I can't get it to check for the ssl line. I'm out of my league with this.
Thanks
What you have is blank-line separated records. You can use awk to check for your ssl-enum-ciphers:
awk -v RS='' '/ssl-enum-ciphers/' file.txt
This will check that the record doesn't contain the phrase 'host down':
awk -v RS='' '/ssl-enum-ciphers/ && !/host down/' file.txt
You could make this more stringent by changing the field separator to a newline character:
awk 'BEGIN { RS=""; FS="\n" } /ssl-enum-ciphers/ && $1 !~ /host down/' file.txt
Add some newlines between records:
awk 'BEGIN { RS=""; FS="\n" } /ssl-enum-ciphers/ && $1 !~ /host down/ { printf "%s\n\n", $0 }' file.txt
Processing Nmap text output is tricky and fraught with dangers, since it can change from version to version. For parsing Nmap output, use the XML output with the -oX or -oA arguments. Then use an XML parsing library or utility to extract the information you need.
For your example, use xmlstarlet to extract the host element that contains a script element with the id attribute set to "ssl-enum-ciphers". This example will output the IP address of the target, followed by the output from the ssl-enum-ciphers script:
xmlstarlet sel -t -m '//script[#id="ssl-enum-ciphers"]' \
-v '../../../address[#addrtype="ipv4"]/#addr' -v '#output' output.xml
In the next release of Nmap, script output itself will be further broken into XML structures, making it easier to do things like output a list of only the weak ciphers in use.
Okay, I want to have Nmap scan an IP range for computers with a certain port open (port 80 in this case) and have it output all the IP's it finds into a text file, stored in this format:
192.168.0.1
192.168.0.185
192.168.0.192
192.168.0.195
So to output the file, I tried using this command:
nmap -sT -p 80 -ttl 40 192.168.0.0-255 -oG - | grep "80/open" > output.txt
Where "output.txt" is the output file that contains the results. So a line of output.txt looks
like this:
Host: 192.168.0.1 () Ports: 80/open/tcp//http///
So I basically want it only to output the IP address with port 80 open, and nothing else.
I want it to not output the "Host: " or the "()" and "Ports: 80/open/tcp//http///" lines. So is there anyway I can have Nmap not put that stuff into the output file? Or make it only
output the IP addresses? I tried looking at the map page, it was of little help. And I looked all over the Internet and that wasn't very useful either. So does anyone know how I can do this? Thanks
Awk is your friend!
$ nmap -sT -p 80 192.168.0.0/24 -oG - | awk '/ 80\/open/{print $2}' > output.txt
This will find lines with port 80 open (notice the space before 80, if you plan to scan more than the one port!), and print field 2, splitting on whitespace. Another way to do it would be:
$ nmap -sT -p 80 --open 192.168.0.0/24 -oG - | awk '$4=="Ports:"{print $2}' > output.txt
This one uses the --open argument to Nmap to only produce output for hosts with open ports. The awk command checks that this is a "Ports" line, not a "Status" line (which may only show up when using -v, but I'm not positive) before printing the IP address.
Note that it is usually in your best interests to save the scan results to a file, to avoid needing to repeat the scan if you decide to extract some different information. If you choose to do this, I would recommend using the XML output (-oX), since there are lots of analysis tools that have parsers built for it already.
Having nmap produce exactly what you want would indeed be nice. But as a more general solution:
$ nmap ... | grep ... | tr '/' ' ' | awk '{ print $2,$5; }
192.168.0.1 80
Or maybe:
nmap ... | grep ... | tr '/' ' ' | cut -d' ' -f2,8
I found a script called scanreport.sh very useful. Although its not necessary, you could just use awk as suggested, but thought it might be of interest.
It gives the ability to output the nmap results nicely by service or port (with highlighting). It uses the grep-able output from nmap (-oG) after a quick tidy from grep -v ^# nmapoutput.txt > report.txt
Example
nmap -sS 192.168.1.22 -oG /directory/of/choice/results.txt
grep -v ^# results.txt > report.txt
./scanreport.sh -f report.txt
Host: 192.168.1.22 ()
22 open tcp ssh OpenSSH 5.3p1 Debian 3ubuntu4 (protocol 2.0)
80 open tcp http Apache httpd 2.2.14 ((Ubuntu))
./scanreport.sh -f report.txt -p 80
Host: 192.168.1.22 ()
80 open tcp http Apache httpd 2.2.14 ((Ubuntu))
./scanreport.sh -f report.txt -s ssh
Host: 192.168.1.22 ()
22 open tcp ssh OpenSSH 5.3p1 Debian 3ubuntu4 (protocol 2.0)
Plenty of stuff on google about it but here a link to one ref.
./scanreport.sh
I'm using regex to parse NMAP output. I want the ip addresses which are up with the corresponding ports open. Now I've a very naive method of doing that:
awk '/^Scanning .....................ports]/ {print substr ($2,1,15);}' results.txt
awk '/^[0-9][0-9]/ {print substr($1,1,4);}' results.txt | awk -f awkcode.awk
where awkcode.awk contains the code to extract numbers out of the substring.
The first line prints all the ips that are up and 2nd gives me the ports. My problem is that I want them mapped to each other. Is there any way to do that? Even a sed script would do.
You will probably find using the "Grepable" output format to be easier to parse:
nmap -oG - -v -A 192.168.0.1-254
Sample output:
Host: 192.168.1.1 (foo) Status: Up
Host: 192.168.1.1 (foo) Ports: 22/open/tcp//ssh//OpenSSH 5.1p1 Debian 6ubuntu2 (protocol 2.0)/, 80/open/tcp//http//Apache httpd 2.2.12 ((Ubuntu))/, 139/open/tcp//netbios-ssn//Samba smbd 3.X (workgroup: BAR)/, 445/open/tcp//netbios-ssn//Samba smbd 3.X (workgroup: BAR)/, 7100/open/tcp//font-service//X.Org X Font Server/ Ignored State: closed (995)
Or if you have an XML parser, use the XML output format:
nmap -oX - -v -A 192.168.0.1-254
Sample output:
<?xml version="1.0" ?>
<?xml-stylesheet href="file:///usr/share/nmap/nmap.xsl" type="text/xsl"?>
<!-- Nmap 5.00 scan initiated Sun Jun 13 08:11:32 2010 as: nmap -oX - -v -A 192.168.1.1-254 -->
<nmaprun scanner="nmap" args="nmap -oX - -v -A 192.168.1.1-254" start="1276434692" startstr="Sun Jun 13 08:11:32 2010" version="5.00" xmloutputversion="1.03">
...
...
<host starttime="1276434692" endtime="1276434775"><status state="up" reason="syn-ack"/>
<address addr="192.168.1.1" addrtype="ipv4" />
<hostnames><hostname name="foo" type="PTR" /></hostnames>
<ports><extraports state="closed" count="995">
<extrareasons reason="conn-refused" count="995"/>
</extraports>
<port protocol="tcp" portid="22"><state state="open" reason="syn-ack" reason_ttl="0"/><service name="ssh" product="OpenSSH" version="5.1p1 Debian 6ubuntu2" extrainfo="protocol 2.0" ostype="Linux" method="probed" conf="10" /><script id="ssh-hostkey" output="1024 1a:2b:4d:5e:6f:00:f1:e2:d3:c4:b5:a6:e2:f3:fe (DSA)
2048 fa:eb:dc:cd:be:af:a0:75:65:8a:52:7d:11:22:33:44 (RSA)" /></port>