tshark - how to avoid to select null or empty http.referer fileds - tshark

On Tshark v. 3.0.5., I am trying to run these commands in order to select URLs typed directly from the user. So I need to exclude http.referer filed that are empty (on null).
tshark -Y "http.request == 1 and **!http.referer**" -T fields -e frame.time -e http.referer -e http.host -r traffic.pcap > no_referer.txt
bash: !http.referer: event not found
tshark -Y "http.request == 1 and **http.referer == ""**" -T fields -e frame.time -e http.referer -e http.host -r traffic.pcap > no_referer.txt
Running as user "root" and group "root". This could be dangerous.
tshark: Unexpected end of filter string.
Do you have any idea about how can I select this data on tshark?

Bash will expand ** in your display filter to something you probably don't want because you are using double quotes. You can use single quotes to ensure that bash doesn't change the contents of the display filter.
Per wireshark http docs, http.referer is a string, so checking against empty values is checking against "". So use http.referer and !http.referer == "" to get packets that have this field, but also where the field is not empty.
tshark -Y 'http.request == 1 and http.referrer and !http.referer == ""' ...
Note: field != value and !field == value are different, and the latter is preferred. dftest can be used to demonstrate why this is.

Related

How to find out tshark boolean filter is empty?

I have a tshark with the following usage:
tshark -r my_pcap_file.pcap -Y 'my and boolean statement'
Now, I want to write a capture file with -w , But before any writing file, I have to find out my tshark is empty or not.
How to can I do it?

How to export some data from Firebird database with FBExport?

I'm trying to export some data from a Firebird database, with FBExport to a CSV file.
The problem is I have two different errors. I spent a few hours to try different combinations:
Unknown switch -
Switches must begin with -
The command I tried:
fbexport -Sc -Q -F h:\AABBCC\export.csv -B -D h:\AABBCC\XXYYZZ.FDB -U "MMNNOO" -P "PPQQRR" -X "select PATIENTS.IPP, PATIENTS.NOM, PATIENTS.NOM_MARITAL, PATIENTS.NOM_USUEL, PATIENTS.PRENOMS, dmc.NUM_DOSSIER_PAPIER, dmc.NUM_DOSSIER_PAPIER_2, dmc.NUM_DOSSIER_PAPIER_3 from PATIENTS join dmc on PATIENTS.ipp = dmc.CODE where dmc.NUM_DOSSIER_PAPIER is not null or dmc.NUM_DOSSIER_PAPIER_2 is not null or dmc.NUM_DOSSIER_PAPIER_3 is not null;"
I absolutely don't understand what FBExport needs.
How can I export the data?
For me, the following command line works:
fbexport -Sc -D employee -U sysdba -P masterkey -F C:\Temp\export.csv -Q "select * from employee"
The problem in your original command was that you had a bare -Q, which caused the following -F to be interpreted as the argument of -Q, which then lead to h:\AABBCC\export.csv to be interpreted as an option, which then produced an error because it doesn't start with a -.
In addition, your command also had the following problems:
-B defines an alternative separator character for the produced CSV. It expects a separator character or TAB or \t for a tab. So, in similar vein as the previous problem, this would cause -D to be interpreted as an argument of -B, which then leads to h:\AABBCC\XXYYZZ.FDB to be interpreted as an option (without -).
-X is a primary option (like -S), to execute the query specified by -Q, instead of exporting (saving, -S). It doesn't accept a query text as argument, so the query text is also interpreted as an option (without -). This occurrence of -X should have been -Q.

Tshark filtering http/https get request

i need to see only HTTP GET requests HOST once (only the webpage), only from defined source ip, not other information/data.
The first request to that url.
For example:
The first packet what goes from client to server.
GET / HTTP/1.1\r\n
What filters should i add? I have tried few, but still get too much information/data...
Is there any possibilities to look HTTPS first request packet too? To see where the client is sending the request?
If you're on a Un*x platform, you could try something like:
tshark -r file.pcap -Y 'ip.src == 1.2.3.4 and http.request.method == "GET"' -T fields -e http.request.method -e http.request.version -e http.request.uri | head -n 1
... or maybe you want to use http.request.full_uri instead of http.request.uri?
If you're on Windows, you may need to install Cygwin coreutils in order to use head, and you may have to quote things a bit differently, e.g.:
tshark -r file.pcap -Y "ip.src == 1.2.3.4 and http.request.method == \"GET\"" -T fields -e http.request.method -e http.request.version -e http.request.uri | head -n 1
For https, you'll need to decrypt the SSL. You can read how to do that on the Wireshark SSL wiki page.

What are tshark's default output fields?

I'm trying to expand tshark's output. On the first round I haven't found simple solution, only that one can extract a field by the -e option, so the following command outputs
frame number
time from beginning of capture
source ip address
destination ip address
http request uri
and the http content lenght, which I want to add to the default output.
tshark -T fields -e frame.number -e frame.time_relative -e ip.src -e ip.dst http.request.uri -e http.content_length
My problem is, that I can't find the default output field names or an option that leaves them and append the desired fields to it.
It's not mandatory, but would be nice to know : )
This is not exactly the default output but comes quite close:
tshark -T fields -e frame.number -e frame.time_delta -e _ws.col.Source -e _ws.col.Destination -e _ws.col.Protocol -e ip.len -e _ws.col.Info

right tool to filter the UUID from the output of blkid program (using grep, cut, or awk, e.t.c)

I want to filter the output of the blkid to get the UUID.
The output of blkid looks like
CASE 1:-
$ blkid
/dev/sda2: LABEL="A" UUID="4CC9-0015"
/dev/sda3: LABEL="B" UUID="70CF-169F"
/dev/sda1: LABEL=" NTFS_partition" UUID="3830C24D30C21234"
In somecases the output of blkid looks like
CASE 2:-
$ blkid
/dev/sda1: UUID="d7ec380e-2521-4fe5-bd8e-b7c02ce41601" TYPE="ext4"
/dev/sda2: UUID="fc54f19a-8ec7-418b-8eca-fbc1af34e57f" TYPE="ext4"
/dev/sda3: UUID="6f218da5-3ba3-4647-a44d-a7be19a64e7a" TYPE="swap"
I want to filter out the UUID.
Using the combination of grep and cut it can be done as
/sbin/blkid | /bin/grep 'sda1' | /bin/grep -o -E 'UUID="[a-zA-Z|0-9|\-]*' | /bin/cut -c 7-
I have tried using awk , grep and cut as below for filtering the UUID
$ /sbin/blkid | /bin/grep 'sda1' | /usr/bin/awk '{print $2}' | /bin/sed 's/\"//g' | cut -c 7-
7ec380e-2521-4fe5-bd8e-b7c02ce41601
The above command(which uses awk) is not reliable since sometimes an extra field such as LABEL may be present in the output of the blkid program as shown in the above output.
What is the best way to create a command using awk which works reliably?
Please post if any other elegant method exits for the job using bin and core utils. I dont want to use perl or python since this has to be run on busybox.
NOTE:-I am using busybox blkid to which /dev/sda1 can not be passed as the args(the version i am using does not support it) hence the grep to filter the line.
UPDATE :- added the CASE 2: -output to show that field position can not be relied upon.
Why are you making it so complex?
Try this:
# blkid -s UUID -o value
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
fc54f19a-8ec7-418b-8eca-fbc1af34e57f
6f218da5-3ba3-4647-a44d-a7be19a64e7a
Or this:
# blkid -s UUID -o value /dev/sda1
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
Install proper blkid package if you don't have it:
sudo apt-get install util-linux
sudo yum install util-linux
For all the UUID's, you can do :
$ blkid | sed -n 's/.*UUID=\"\([^\"]*\)\".*/\1/p'
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
fc54f19a-8ec7-418b-8eca-fbc1af34e57f
6f218da5-3ba3-4647-a44d-a7be19a64e7a
Say, only for a specific sda1:
$ blkid | sed -n '/sda1/s/.*UUID=\"\([^\"]*\)\".*/\1/p'
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
The sed command tries to group the contents present within the double quotes after the UUID keyword, and replaces the entire line with the token.
Here's a short awk solution:
blkid | awk 'BEGIN{FS="[=\"]"} {print $(NF-1)}'
Output:
4CC9-0015
70CF-169F
3830C24D30C21234
Explanation:
BEGIN{FS="[=\"]"} : Use = and " as delimiters
{print $(NF-1)}: NF stands of Number of Fields; here we print the 2nd to last field
This is based on the consistent structure of blkid output: UUID in quotes is at the end of each line.
Alternatively:
blkid | awk 'BEGIN{FS="="} {print $NF}' | sed 's/"//g'
data.txt
/dev/sda2: LABEL="A" UUID="4CC9-0015"
/dev/sda3: LABEL="B" UUID="70CF-169F"
/dev/sda1: LABEL=" NTFS_partition" UUID="3830C24D30C21234"
awk and sed combination
cat data.txt | awk 'BEGIN{FS="UUID";RS="\n"} {print $2}' | sed -e 's/=//' -e 's/"//g'
Explanation:
Set the Field Separator to the string 'UUID', $2 will give the rest output
use sed then to remove the = and " as shown where -e is a switch so that you can give multiple sed commands/expression in one.
All occurrences of " are removed using the ending g option i.e. global.
The question has a "e.t.c" so I'm going to assume python is one of the options ;)
#!/usr/bin/env python3
import subprocess, re, json
# get blkid output
blkid = subprocess.check_output(["blkid"]).decode('utf-8')
devices = []
for line in [x for x in blkid.split('\n') if x]:
parameters = line.split()
for idx, parameter in enumerate(parameters):
if idx is 0:
devices.append({"DEVICE": re.sub(r':$','',parameter)})
continue
key_and_value = parameter.split('=')
devices[-1].update({
key_and_value[0]: re.sub(r'"','',key_and_value[1])
})
uuids = [{dev['DEVICE']: dev['UUID']} for dev in devices if 'UUID' in dev.keys()]
print(json.dumps(uuids, indent=4, sort_keys=True))
Although, this is probably overkill and quite a few error handling/optimization is missing from this script XD
I assume you're using busybox in an initramfs and you are waiting for your e.g. USB drive with the rootfs on it to become available.
You could use the following awk script (busybox awk compliant).
# cat get-ruuid.awk
BEGIN {
ruuid=ENVIRON["RUUID"]
}
/^\/dev\/sd[a-z]/ {
if (index($0, tolower(ruuid)) || index($0, toupper(ruuid))) {
split($1, parts, ":")
printf("%s\n", parts[1])
exit(0) # Return success and stop further scanning.
}
}
END {
exit(1) # If we reach the end, it means RUUID was not found.
}
Call it as follows from e.g. the init script; this is not the most ideal way.
# The UUID of your root partition
export RUUID="<put proper uuid value here>"
for x in 1, 2, 3, 4, 5 ; do
mdev -s
found=$(blkid | awk -f ./get-ruuid.awk)
test -z $found || break; # If no longer zero length, break the loop.
sleep 1
done
But if this is the only reason why you would want to have an initramfs, I would use the 'root=PARTUUID=... waitroot' Linux kernel command line option. Check the kernel docs and sources.
Get the proper PARTUUID (NOT UUID) of your root partition with the blkid command.