Nfdump nfcapd file - binary to csv file conversion - netflow
Given a nfcapd file produced in nfdump nfcapd.2017 which is in the default binary format
How would I create a version of this file in csv format using nfdump?
I tried using nfdump -r nfcapd.2017 -w newfile -o csv but that doesn't seem to work
The -w option is for writing in binary nfdump format (or actually nfcapd format). Simply omit it to output in CSV:
nfdump -r nfcapd.2017 -o csv
ts,te,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,in,out,sas,das,smk,dmk,dtos,dir,nh,nhb,svln,dvln,ismc,odmc,idmc,osmc,mpls1,mpls2,mpls3,mpls4,mpls5,mpls6,mpls7,mpls8,mpls9,mpls10,cl,sl,al,ra,eng,exid,tr
2018-01-16 16:33:14,2018-01-16 16:33:14,0.003,192.168.2.204,224.0.0.251,5353,5353,UDP,......,0,0,2,691,0,0,0,0,0,0,0,0,0,0,0.0.0.0,0.0.0.0,0,0,00:00:00:00:00:00,00:00:00:00:00:00,00:00:00:00:00:00,00:00:00:00:00:00,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0, 0.000, 0.000, 0.000,0.0.0.0,0/0,1,1970-01-01 01:00:00.000
2018-01-16 16:33:14,2018-01-16 16:33:14,0.000,192.168.2.204,192.168.2.70,55925,50767,UDP,......,0,0,1,546,0,0,0,0,0,0,0,0,0,0,0.0.0.0,0.0.0.0,0,0,00:00:00:00:00:00,00:00:00:00:00:00,00:00:00:00:00:00,00:00:00:00:00:00,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0,0-0-0, 0.000, 0.000, 0.000,0.0.0.0,0/0,1,1970-01-01 01:00:00.000
...
And redirect the output to get a CSV file:
nfdump -r nfcapd.2017 -o csv > nfcapd.2017.csv
Related
file encoding to utf-8 from iso-8859
input file with 50k plus unix file command shows it as ISO-8859 text, with very long lines input record causing issue MONTRéAL when i use iconv command like below nothing changes, record is as-is **iconv -f ISO-8859-1 -t UTF-8 input.txt -o output.txt** when i copy the specific record in question using sed command, file is created as utf-8 and the record looks good **MONTRéAL** **sed -n '41696p' input.txt > output.txt** when i copy from 1 through 41696 with the same sed command, record didn't change **sed -n '1,41696p' input.txt > output.txt** how do I copy the file from iso-8859 to utf-8 with proper characterset??
Converting only non utf-8 files to utf-8
I have a set of md files, some of them are utf-8 encoded, and others are not (windows-1256 actually).
I want to convert only non-utf-8 files to utf-8.
The following script can partly do the job:
for file in *.md;
do
iconv -f windows-1256 -t utf-8 "$file" -o "${file%.md}.🆕.md";
done
I still need to exclude the original utf-8 files from this process, (maybe using file command?). Try the following command to understand what I mean:
file --mime-encoding *
Notice that although file command isn't smart enough to detect the right character set of non-utf-8 files, It's enough in this case that it can distinguish between utf-8 and non-utf-8 files.
Thanks in advance for help.
You can use for example an if statement:
if file --mime-encoding "$file" | grep -v -q utf-8 ; then
iconv -f windows-1256 -t utf-8 "$file" -o "${file%.md}.🆕.md";
fi
If grep doesn't find a match, it returns a status code indicating failure. The if statement tests the status code
how to parse out file name and import the data into csv files using shellscript?
I have a Matlab program output some data into filenames for example a filename look like Temperature_10_Volumn_5.dat The file is empty inside, but I need to parse out 10 and 5. I have 1000 files like this. What would be the easiest shellscript to parse out the numbers and write them to a csv file such that Matlab can read the csv file and plot the graph? Thanks!
Here is the script that will print all your values in a single file with values separated by ";"
Usage is :
parse_file.sh /home/user/your_dir_to_files output.csv
#!/bin/bash
#title :parse_file.sh
#description :parse file name into a single file
#author :Bertrand Martel
#date :13/08/2015
file_list=`ls $1`
IFS=$'\n' #line delimiter
#empty your output file
cp /dev/null "$2"
for i in $file_list; do
#get temperature between _ and _
temperature=`echo $i | awk -v FS="([_]|[_])" '{print $2}'`
#remove everything before Volumn
second_part=`echo $i | sed 's/.*Volumn//'`
#get volume between _ and .
volume=`echo $second_part | awk -v FS="([_]|[.])" '{print $2}'`
new_line="$temperature;$volume"
#append to file
echo $new_line >> "$2"
done
cat "$2"
I created a gist with the file https://gist.github.com/bertrandmartel/fd68c0373af35eaba934
I assume you only have the so called files in you directory output
How can I extract multiple .gz log files in command line
I have a years worth of log files that are all in .gz files. Is there a command I can use to extract these all at once into their current directory? I tried unzip *.gz but doesn't work. Any other suggestions?
shell sciprt?
#!/bin/ksh
TEMPFILE=tempksh_$$.tmp #create a file name
> $TEMPFILE #create a file w/ name
ls -l | grep '.*\.gz$' \ #make dynamic shell script
| awk '{printf "unzip %s;\n", $9;}' \ #with unzip cmd for each item
>> $TEMPFILE #and write to TEMPFILE
chmod 755 $TEMPFILE #give run permissions
./$TEMPFILE #and run it
rm -f $TEMPFILE #clean up
Untested but i think you get the idea....
Actually a little fiddling and gets far simpler...
set -A ARR *.gz;
for i in ${characters[#]}; do `unzip $i`; done;
unset ARR;
For googles sake, since it took me here, it's as simple as this: gzip -dc access.log.*.gz > access.log
As noted in a comment, you want to use gunzip, not gzip. unzip is for .zip files. gzip is for .gz files. Two completely different formats. gunzip *.gz or: gzip -d *.gz That will delete the .gz files after successfully decompressing them. If you'd like to keep all of the original .gz files, then: gzip -dk *.gz
How do I use afconvert to convert all the files in a directory from wav to caf?
I have a directory with about 50 wav files that I need to convert to caf, because AudioServicesCreateSystemSoundID() returns an error for some of them (but not all). Here's an example of the command I've used successfully for a single file: afconvert -f caff -d LEI16#44100 -c 1 whistle.wav whistle.caf How do I do this quickly - not one-by-one for each file?
Similar approach for bash: for i in *.wav; do afconvert -f caff -d LEI16#44100 -c 1 $i ${i%.wav}.caf; done
On Windows, use the %~ni syntax. for %i in (*.wav) do afconvert -f caff -d LEI16#44100 -c 1 %i %~ni.caf
for file in *.wav; do afconvert -f caff -d LEI16#44100 -c 1 "$file"; done Hit the Return key directly after done. Simple :)
For the people who are using OSX and are a bit afraid of Terminal scripts I created a little application with Automator, this application converts the files you select. Download here
found this: ## ## Shell script to batch convert all files in a directory to caf sound format for iPhone ## Place this shell script a directory with sound files and run it: 'sh converttocaf.sh' ## Any comments to 'support#ezone.com' ## for f in *; do if [ "$f" != "converttocaf.sh" ] then /usr/bin/afconvert -f caff -d LEI16 "$f" echo "$f converted" fi done Customize the afconvert options then save it as a text file called 'converttocaf.sh', place it in the directory of files you want to convert and run it from Terminal. It works with files with spaces in their names.
If you are dealing with filenames that contain spaces on Linux, try the following code:
SAVEIFS=$IFS
IFS=$(echo -en "\n\b"); for i in *.wav; do afconvert -f caff -d LEI16#44100 -c 1 $i ${i%.wav}.caf; done
Python script on OSX. Default data format of the .caf is ima4. Default directory is '.'
Make a file called wav2caf.py, put it in your path, make it executable, fill it with:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys
import os
import argparse
import glob
def main():
# handle command line args
parser = argparse.ArgumentParser(description='A program that converts .wav to .caf files.', formatter_class=argparse.RawTextHelpFormatter)
parser.add_help = True
parser.add_argument('-f', '--folder', dest='folder', type=str, default='.', help='folder of files to convert')
parser.add_argument('-d', '--data', dest='data', type=str, default='ima4', help='data format of .caf')
args = parser.parse_args()
# process files in folder
os.chdir(args.folder)
for filename in glob.glob("*.wav"):
name, ext = os.path.splitext(filename)
command = 'afconvert -f caff -d ' + args.data + ' ' + filename + ' ' + name + '.caf'
os.system(command)
if __name__ == "__main__":
main()
Converts all .wav to .caf in current directory:
wav2caf.py
Converts all .wav to .caf in specified directory:
wav2caf.py -f Desktop/wavs/
Converts all .wav to .caf with data type 'aac ':
wav2caf.py -d 'aac '