sed search and replace specific column only - sed

Im taking weather day from wunderground.com and I and then trimming down the data for use on gnuplot. Im having trouble replacing the second column only data from number to months abbreviations. Only interested in the second column.
I want to go from this;
>2013 08 02 23 37 00 73.3
>2013 08 02 23 42 00 73.4
>2013 08 02 23 45 00 73.3
>2013 08 02 23 47 00 73.1
>2013 08 02 23 52 00 73.1
>2013 08 02 23 57 00 73.1
To this:
>2013 AUG 02 23 37 00 73.3
>2013 AUG 02 23 42 00 73.4
>2013 AUG 02 23 45 00 73.3
>2013 AUG 02 23 47 00 73.1
>2013 AUG 02 23 52 00 73.1
>2013 AUG 02 23 57 00 73.1
i am trying to use sed to change the numbers into the correct month and i keep getting this. I only want the correct sed expression to execute not all of them. This is the command i am trying to use.
sed -e 's/01/JAN/' -e 's/02/FEB/' -e 's/03/MAR/' -e 's/04/APR/' -e 's/05/MAY/' -e 's/06/JUN/' -e 's/07/JUL/' -e 's/08/AUG/' -e 's/09/SEP/' -e 's/10/OCT/' -e 's/11/NOV/' -e 's/12/DEC/'
How would i go about this.

This might work for you (GNU sed):
sed -nri 'G;s/$/01JAN02FEB03MAR04APR05MAY06JUN07JUL08AUG09SEP10OCT11NOV12DEC/;s/ (..)(.*)\1(...)/ \3\2/;P' file
This adds a lookup table to the end of each line and substitutes the key for the value.

I would use awk for this:
$ awk 'BEGIN{split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",a)} {$2=a[$2+0]}1' a
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
To update the field with the new content, just redirect and then move:
awk .... file > temp_file && mv temp_file file
Explanation
What we do is to give awk a list of strings with the months names. Once we convert it into an array, a[1] will be Jan, a[2] Feb and so on. So then it is just a matter of replacing the 2nd field with a[2nd field].
BEGIN{split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",a)} fetches the data and inserts into the a[] array.
{$2=a[$2+0]} sets the 2nd field as a[2nd field]. The $2+0 is done to convert 08 to 8.
Finally 1 evaluates as true and makes awk perform its default action: {print $0}.

Workaround that works for this problem (since your first column is very predictable) but not the general question:
sed -E -e 's/^([0-9]{4}) 01/\1 JAN/' -e 's/^([0-9]{4}) 02/\1 FEB/' etc.
awk has a sub function that could get unwieldy for many options you have here.
Perl scripts might be the best way to go.

$ awk '{$2=substr("JanFebMarAprMayJunJulAugSepOctNovDec",(3*$2)-2,3)}1' file
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
Since it came up in a comment:
The idiomatic awk way to map from a month number to a name is:
number = (match("JanFebMarAprMayJunJulAugSepOctNovDec",<name>)+2)/3
and the above is just the natural inverse of that:
name = substr("JanFebMarAprMayJunJulAugSepOctNovDec",(3*<number>)-2,3)
Like with anything in awk there's various ways to get the output you want but IMHO the symmetry here makes it an attractive solution:
awk 'BEGIN{
months = "JanFebMarAprMayJunJulAugSepOctNovDec"
name = "Jul"
number = (match(months,name)+2)/3
print name " -> " number
name = substr(months,(3*number)-2,3)
print number " -> " name
}'
Jul -> 7
7 -> Jul
Notice that the script uses the same definition for months no matter which direction the conversion is being done and it's a similar math calculation in both directions.
Nothing wrong with doing it this way too of course:
awk 'BEGIN{
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",num2name)
for (number in num2name) {
name2num[num2name[number]] = number
}
name = "Jul"
number = name2num[name]
print name " -> " number
name = num2name[number]
print number " -> " name
}'
Jul -> 7
7 -> Jul
Just a few more lines of code, nbd.

Using gnu awk's function strftime() and mktime()
awk '{$2=strftime("%b",mktime("2014 " $2 " 1 1 0 0"))}1' file
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
Explanation
mktime("2014 " $2 " 1 1 0 0") fake the epoch time, use the column 2 as month
strftime("%b",mktime("2014 " $2 " 1 1 0 0")) convert the epoch back to the date, with %b, export the abbreviated month name (Jan, Feb, etc)
The benefit with this awk:
It is shorter, of course. Second, you can control/adjust the format in strftime() to export any date format you like.
For example, if change to full month name %B. You needn't rewrite the code.
awk '{$2=strftime("%B",mktime("2014 " $2 " 1 1 0 0"))}1' file

Related

Feed hexdump to sed

I have a file called samples.txt with 374371 lines that was created like this:
hexdump -e '8/1 "%02x " "\n"' samples.bin > samples.txt
and it looks like this (deprecated):
1a 03 1a 03 4a 03 57 03
4b 03 44 03 1e 03 09 04
10 03 19 03 40 03 ae 03
1e 03 26 03 33 03 ad 03
10 03 84 03 43 03 62 03
e0 03 16 03 34 03 c4 03
f8 02 3b 03 53 03 61 03
10 03 15 03 42 03 58 03
23 03 1f 03 57 03 62 03
17 03 73 03 31 03 33 03
14 03 ff 02 30 03 51 03
5f 03 42 03 47 03 7e 03
ba 03 26 03 35 03 a0 03
08 03 33 03 36 03 c8 03
0c 03 38 03 97 03 44 03
and the following sed command:
echo " ch0 ch1 ch2 ch3"; cat -n samples.txt | sed 's/\s*\([0-9]*\)\s*\([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\)/Sample \1:\t0x\2\3 0x\4\5 0x\6\7 0x\8\9/' | head -n 15
gives me output that looks like this:
ch0 ch1 ch2 ch3
Sample 1: 0x1a03 0x1a03 0x4a03 0x5703
Sample 2: 0x4b03 0x4403 0x1e03 0x0904
Sample 3: 0x1003 0x1903 0x4003 0xae03
Sample 4: 0x1e03 0x2603 0x3303 0xad03
Sample 5: 0x1003 0x8403 0x4303 0x6203
Sample 6: 0xe003 0x1603 0x3403 0xc403
Sample 7: 0xf802 0x3b03 0x5303 0x6103
Sample 8: 0x1003 0x1503 0x4203 0x5803
Sample 9: 0x2303 0x1f03 0x5703 0x6203
Sample 10: 0x1703 0x7303 0x3103 0x3303
Sample 11: 0x1403 0xff02 0x3003 0x5103
Sample 12: 0x5f03 0x4203 0x4703 0x7e03
Sample 13: 0xba03 0x2603 0x3503 0xa003
Sample 14: 0x0803 0x3303 0x3603 0xc803
Sample 15: 0x0c03 0x3803 0x9703 0x4403
It would seem to me, redirecting the output of hexdump to a temporary samples.txt file is inefficient, because for one you have to delete it at the end, and for two, perhaps using sed on the data in memory is faster than it would be on disk. How can I skip the samples.txt file and get the same results?
I think the following should work, if I understand your intentions correctly.
hexdump -e '8/1 "%02x " "\n"' samples.bin | cat -n | sed '....
Alternative with xxd and coreutils:
{
# Print header
printf "\t\tCh0 Ch1 Ch2 Ch3\n"
# Create hexdump of data
xxd -p samples.bin |
# Use 16 bit samples
fold -w4 |
# Add hex indicator
sed 's/^/0x/' |
# Data contains 4 interleaved channels
xargs -n4 |
# Number each sample
nl -n ln |
sed 's/ /:/; s/^/Sample /'
} > samples.out
Content of samples.out:
Ch0 Ch1 Ch2 Ch3
Sample 1: 0x1a03 0x1a03 0x4a03 0x5703
Sample 2: 0x4b03 0x4403 0x1e03 0x0904
Sample 3: 0x1003 0x1903 0x4003 0xae03
Sample 4: 0x1e03 0x2603 0x3303 0xad03
Sample 5: 0x1003 0x8403 0x4303 0x6203
Sample 6: 0xe003 0x1603 0x3403 0xc403
Sample 7: 0xf802 0x3b03 0x5303 0x6103
Sample 8: 0x1003 0x1503 0x4203 0x5803
Sample 9: 0x2303 0x1f03 0x5703 0x6203
Sample 10: 0x1703 0x7303 0x3103 0x3303
Sample 11: 0x1403 0xff02 0x3003 0x5103
Sample 12: 0x5f03 0x4203 0x4703 0x7e03
Sample 13: 0xba03 0x2603 0x3503 0xa003
Sample 14: 0x0803 0x3303 0x3603 0xc803
Sample 15: 0x0c0a 0x3803 0x9703 0x4403

Problems with empty cell in a large matrix: MATLAB

I have a large matrix of 102730 rows in the form of text file (sample text file is attached) with some header files in it. The first column show year, the next the month, followed by the day, and value1, value2 and value 3. Some of the cells are missing/empty. I want to fill these empty cells with NaN, so that they don't interefere with the next value.
This is the input matrix:
1970 01 13 21.0 6.1 06 000.0
1970 01 14 22.4 8.1 03 000.0
1970 01 15 21.2 8.1 04 000.0
1970 01 16 22.6 9.1 04 000.0
1970 01 17 22.8 9.1 02 000.0
1970 01 18 22.9 8.9 07 000.0
1970 01 19 23.8 10.8 04 000.0
1970 01 20 21.8 12.1 10 010.5
1970 01 21 19.8 06 012.9
1970 01 22 15.3 8.5 07 000.0
1974 06 28 39.2 25.6 03 000.0
1974 06 29 41.2 30.5 05 000.0
1974 06 30 40.3 31.2 07 000.0
1974 07 01 41.3 31.5 12 000.0
1974 07 02 43.3 31.3 20 000.0
1974 07 03 41.2 16 041.6
1974 07 04 34.3 21.4 14 054.5
1974 07 05 33.1 23.8 05 000.0
1974 07 06 36.2 28.9 06 000.0
1975 04 18 36.6 20.8 12 000.0
1975 04 19 37.4 21.1 05 000.0
1975 04 20 39.9 27.0 07 000.0
1975 04 21 39.5 27.3 09 000.0
1975 04 22
1975 04 23 39.5 27.1 08 000.0
1975 04 24 37.7 26.0 10 000.0
1975 04 25 38.7 27.2 15 000.0
The desired output matrix:
1970 01 13 21.0 6.1 06 000.0
1970 01 14 22.4 8.1 03 000.0
1970 01 15 21.2 8.1 04 000.0
1970 01 16 22.6 9.1 04 000.0
1970 01 17 22.8 9.1 02 000.0
1970 01 18 22.9 8.9 07 000.0
1970 01 19 23.8 10.8 04 000.0
1970 01 20 21.8 12.1 10 010.5
1970 01 21 19.8 Nan 06 012.9
1970 01 22 15.3 8.5 07 000.0
1974 06 28 39.2 25.6 03 000.0
1974 06 29 41.2 30.5 05 000.0
1974 06 30 40.3 31.2 07 000.0
1974 07 01 41.3 31.5 12 000.0
1974 07 02 43.3 31.3 20 000.0
1974 07 03 41.2 Nan 16 041.6
1974 07 04 34.3 21.4 14 054.5
1974 07 05 33.1 23.8 05 000.0
1974 07 06 36.2 28.9 06 000.0
1975 04 18 36.6 20.8 12 000.0
1975 04 19 37.4 21.1 05 000.0
1975 04 20 39.9 27.0 07 000.0
1975 04 21 39.5 27.3 09 000.0
1975 04 22 Nan Nan Nan Nan
1975 04 23 39.5 27.1 08 000.0
1975 04 24 37.7 26.0 10 000.0
1975 04 25 38.7 27.2 15 000.0
As an attempt, first I tried with this:
T = readtable('sample.txt') ;
Above code didn't work since it meshed up and gave the wrong number of columns when there 2 digits before the decimal. Secondly, I found this link: Creating new matrix from cell with some empty cells disregarding empty cells
The foll. code snippet may be useful from this link, but I don't know how to read the data directly from the text pad inorder to apply this code & subsequent retrieval process:
inds = ~cellfun('isempty', elem); %elem to be replaced as sample
I also find out the method to detect empty cells here: How do I detect empty cells in a cell array?
but I couldn't figure out how to read the data from a text file considering these empty cells.
Could anyone please help?
Since R2019a, you can simply use readmatrix:
>> myMat = readmatrix('sample.txt')
From the docs:
For delimited text files, the importing function converts empty fields in the file to either NaN (for a numeric variable) or an empty character vector (for a text variable). All lines in the text file must have the same number of delimiters. The importing function ignores insignificant white space in the file.
For previous releases, you can use detectImportOptions object when calling readtable:
% Detect options.
>> opts = detectImportOptions('sample.txt');
% Read table.
>> myTable = readtable('sample.txt',opts);
% Visualise last rows of table.
>> tail(myTable)
ans =
8×7 table
Var1 Var2 Var3 Var4 Var5 Var6 Var7
____ ____ ____ ____ ____ ____ ____
1975 4 18 36.6 20.8 12 0
1975 4 19 37.4 21.1 5 0
1975 4 20 39.9 27 7 0
1975 4 21 39.5 27.3 9 0
1975 4 22 NaN NaN NaN NaN
1975 4 23 39.5 27.1 8 0
1975 4 24 37.7 26 10 0
1975 4 25 38.7 27.2 15 0
For your text file, detectImportOptions is filling missing values with NaN :
>> opts.VariableOptions
If the desired output is a matrix, you can then use table2array:
>> myMat = table2array(myTable)

A Script To Convert Numbers

I was hoping someone could assist me in making a linux script that can take a number and convert it to this format.
Here is a few of examples:
number = 753082360700386 Converted = 7A 35 80 32 06 07 30 68
number = 653082360700387 Converted = 6A 35 80 32 06 07 30 78
number = 453082360700389 Converted = 4A 35 80 32 06 07 30 98
The number will always have one A and last pair of numbers
swapped from 89 to 98 and next set swapped from 03 to 30
going from right to left and adding spaces between each pair

APDU: "Conditions of use not satisfied" (69 85) while calculate signature

With a smart card Gemalto (IAS ECC), I would to calculate a signature by using private key stored on smart card. For this, I use APDU commands:
// Verify PIN
00 20 00 01 04 31 32 33 34
-> 90 00
// Create a context for security operation
00 22 41 B6 06 84 01 84 80 01 12
-> 90 00
// Set the hash of the document
00 2A 90 A0 14 HASH OF DOCUMENT
-> 69 85
// Calculating the signature
00 2A 9E 9A 80
-> 69 85
My problem is the following: the las two commands return the error code "69 85", meaning "Conditions of use not satisfied".
I have already tried several solutions, but I obtain always the same error. How to resolve it? What does this code can mean?
After some tests, I discovered something interesting. When I replace cla "00" by "10", smart card returns a different response:
// Create a context for security operation
00 22 41 B6 06 84 01 84 80 01 12
// Verify PIN
00 20 00 01 04 31 32 33 34
// Calculating the signature (I replace "00" by "10")
10 2A 9E 9A 23 30 21 30 09 06 05 2B 0E 03 02 1A 05 00 04 14 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 12 13 14 15
I don't know if it's the good solution because smart card returns "90 00". But, it would return the content of my signature!
Thank you for your help!
Best regards
You are getting SW 6985 for
// Set the hash of the document
00 2A 90 A0 14 HASH OF DOCUMENT
-> 69 85
Since you have not set the correct context in current security environment.
Let me explain this below
First you performed VERIFY PIN command which was successful
// Verify PIN
00 20 00 01 04 31 32 33 34
-> 90 00
Then you performed MSE SET command,Where you set the security context.For this you have to understood how SE works(Please refer to section 3.5 fron IAS ECC v1.01).
At the time of personalisation, the Personaliser agent create SDO(Secure Data Object) inside the card.The reference to this SDO are mentioned in SE(Security Environment) in form of CRT(Control reference template).
// Create a context for security operation
00 22 41 B6 06 84 01 84 80 01 12
-> 90 00
Generally speaking, MSE SET command will always return SW 900 even if the SDO reference is wrong. Since it only return SW 6A80 when the template is wrong not when the reference is wrong.(The SDO reference is passed in tag 84)
After that you performed PSO HASH command
// Set the hash of the document
00 2A 90 A0 14 HASH OF DOCUMENT
-> 69 85
where the card return SW 6985(Condition of use not satisfied), This indicate the algorithm and SDO reference used for calculating Hash may wrong. Which is probably happening since the SDO reference which was sent during the time of MSE SET command is not available
Detecting error coming from MSE SET could be tricky since it return SW 9000.
For these type of situation you have to check the personalisation file carefully and need to match the MSE SET command with regard to SDO reference and supported ALGOs.
It may be useful to put the default context (e.g., cryptographic algorithms or
security operations) into the current SE in order to have few exchanges of MSE set commands.

How can I tell what Scala version a .class file was compiled with?

How can I tell what Scala version a .class file was compiled with?
I suppose the information is stored in the "pickled" part of the .class file, according to the 2008 "Reflecting Scala" rapport, from Yohann Coppel, under the supervision of Prof. Martin Odersky.
During the compilation process (represented on fig. 2), the Scala compiler generates two types of data.
The first one is some classic Java bytecode, which can be read and executed by a standard Java virtual machine.
The second one is what is called “Pickled data”, and represents the basic structure of the original source file.
This information is enclosed in a .class file.
The Java bytecode specification allows the compiler to “define and emit class files containing new attributes in the attributes tables of class file structures”. These attributes are silently ignored by JVMs if they do not recognize them.
The Scala compiler generates pickled data for about any data structure in a Scala program, called symbols in the pickler context.
Symbols are stored linearly with the format shown on Fig. 3.
The tag represents the type of data stored,
then the length gives the length of the following data block.
The data block can contain multiple information,
such as the name of a symbol.
ScalaSig = "ScalaSig" Version Symtab
Version = Major_Nat Minor_Nat <====
Symtab = numberOfEntries_Nat {Entry}
The ScalaSig attribute definition.
A more complete definition can be found in the scala.tools.nsc.symtab.PickleFormat source file (now scala.reflect.internal.pickling.PickleFormat).
You can also see how to read the Pickled data in scala.tools.nsc.util.ShowPickled.
This page shows a script (not tested) which will display the pickled data:
#!/bin/sh
#
# Shows the pickled scala data in a classfile.
if [ $# == 0 ] ; then
echo "Usage: $0 [--bare] [-cp classpath] <class*>"
exit 1
fi
TOOLSDIR=`dirname $0`
CPOF="$TOOLSDIR/cpof"
PACK="$TOOLSDIR/../build/pack/lib"
QUICK="$TOOLSDIR/../build/quick/classes"
STARR="$TOOLSDIR/../lib"
CP=""
if [ -f "${PACK}/scala-library.jar" ] ; then
CP=`${TOOLSDIR}/packcp`
elif [ -d "${QUICK}/library" ] ; then
CP=`${TOOLSDIR}/quickcp`
else
CP=`${TOOLSDIR}/starrcp`
fi
if [ "$1" == "-cp" ] ; then
shift
CP="${1}:${CP}"
shift
fi
java -cp "$CP" scala.tools.nsc.util.ShowPickled $*
You can see the Scala Major/Minor version in the class file if you use javap with the verbose option. For example, the following is shown for a file compiled using scala 2.8.0 final:
javap -private -verbose T
Compiled from "SomeTest.scala"
public interface T
SourceFile: "SomeTest.scala"
ScalaSig: length = 0x3
05 00 00
RuntimeVisibleAnnotations: length = 0xB
00 01 00 06 00 01 00 07 73 00 08
minor version: 0
major version: 49
Constant pool:
const #1 = Asciz SourceFile;
const #2 = Asciz SomeTest.scala;
const #3 = Asciz s;
const #4 = Asciz ()Ljava/lang/String;;
const #5 = Asciz ScalaSig;
//etc etc...
while the following is the output of a file compiled using scala 2.7.7:
javap -verbose T2
Compiled from "SomeTest2.scala"
public interface T2
SourceFile: "SomeTest2.scala"
ScalaSig: length = 0x87
04 01 1B 06 08 01 02 FFFFFF84 FFFFFF90 FFFFFF80 FFFFFF91 00 05 02 02 54
32 0A 01 03 01 07 3C 65 6D 70 74 79 3E 03 00 13
02 00 06 10 02 07 0C 0D 01 08 0A 02 09 0A 01 04
6C 61 6E 67 0A 01 0B 01 04 6A 61 76 61 09 02 0D
08 02 06 4F 62 6A 65 63 74 08 05 0F 00 FFFFFF86 00 10
01 01 73 15 01 11 10 02 12 18 0E 02 13 16 0D 01
14 0A 01 15 01 05 73 63 61 6C 61 09 02 17 14 01
06 50 72 65 64 65 66 09 02 19 1A 02 06 53 74 72
69 6E 67 0A 02 17 14
minor version: 0
major version: 49
Constant pool:
const #1 = Asciz SourceFile;
const #2 = Asciz SomeTest2.scala;
//etc etc...
The first two bytes of the ScalaSig constant entry should represent the scala Major/Minor version, I believe, which are defined in PickleFormat. The 2.7.7 version of PickleFormat can be found here, and shows that the major/minor version differs from the 2.8.0 version.
I checked the 2.7.1 version of this class as well, but here the Major/Minor version is the same as the 2.7.7 one, so you may not be able to distinguish between minor scala versions by using this method.
Most probably you could parse the .class file and read the version from an attribute attached from the scala compiler onto the class file.
To learn more about the existance of such an attribute you might start at the sources of the scala compiler ( http://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/src/compiler/scala/tools/nsc/backend/jvm/GenJVM.scala ).
To learn how to parse a .class file you might read in the spec ( http://jcp.org/aboutJava/communityprocess/final/jsr202/index.html ).
The example code I posted here ( Java Illegal class modifiers Exception code 0x209 ) might help at the implementation, too .
FWIW, Here's a version of VonC's script that sets the classpath to scala-library.jar and scala-compiler.jar
Tested under cygwin and linux, with scala 2.11.8 and 2.12.1,
Should work under OSX.
Doesn't seem to like --bare argument, however.
(requires scala to be in your PATH.)
#!/bin/bash
# Shows the pickled scala data in a classfile.
if [ $# == 0 ] ; then
echo "Usage: $0 [--bare] [-cp classpath] <class*>"
exit 1
fi
unset JAVA_TOOL_OPTIONS
[ -z "$SCALA_HOME" ] && SCALA_HOME=$(which scala | sed -e 's#/bin/scala##')
export OSTYPE=$(uname | tr '[A-Z]' '[a-z]' | sed -e 's#[_0-9].*##')
case $OSTYPE in
cygwin) SEP=";" ;;
*) SEP=":" ;;
esac
CP="${SCALA_HOME}/lib/scala-library.jar${SEP}${SCALA_HOME}/lib/scala-compiler.jar${SEP}${SCALA_HOME}/lib/scala-reflect.jar"
if [ "$1" == "-cp" ] ; then
shift
CP="${1}${SEP}${CP}"
shift
fi
java -cp "$CP" scala.tools.nsc.util.ShowPickled $*