Feed hexdump to sed - sed

I have a file called samples.txt with 374371 lines that was created like this:
hexdump -e '8/1 "%02x " "\n"' samples.bin > samples.txt
and it looks like this (deprecated):
1a 03 1a 03 4a 03 57 03
4b 03 44 03 1e 03 09 04
10 03 19 03 40 03 ae 03
1e 03 26 03 33 03 ad 03
10 03 84 03 43 03 62 03
e0 03 16 03 34 03 c4 03
f8 02 3b 03 53 03 61 03
10 03 15 03 42 03 58 03
23 03 1f 03 57 03 62 03
17 03 73 03 31 03 33 03
14 03 ff 02 30 03 51 03
5f 03 42 03 47 03 7e 03
ba 03 26 03 35 03 a0 03
08 03 33 03 36 03 c8 03
0c 03 38 03 97 03 44 03
and the following sed command:
echo " ch0 ch1 ch2 ch3"; cat -n samples.txt | sed 's/\s*\([0-9]*\)\s*\([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\) \([0-9a-f]*\)/Sample \1:\t0x\2\3 0x\4\5 0x\6\7 0x\8\9/' | head -n 15
gives me output that looks like this:
ch0 ch1 ch2 ch3
Sample 1: 0x1a03 0x1a03 0x4a03 0x5703
Sample 2: 0x4b03 0x4403 0x1e03 0x0904
Sample 3: 0x1003 0x1903 0x4003 0xae03
Sample 4: 0x1e03 0x2603 0x3303 0xad03
Sample 5: 0x1003 0x8403 0x4303 0x6203
Sample 6: 0xe003 0x1603 0x3403 0xc403
Sample 7: 0xf802 0x3b03 0x5303 0x6103
Sample 8: 0x1003 0x1503 0x4203 0x5803
Sample 9: 0x2303 0x1f03 0x5703 0x6203
Sample 10: 0x1703 0x7303 0x3103 0x3303
Sample 11: 0x1403 0xff02 0x3003 0x5103
Sample 12: 0x5f03 0x4203 0x4703 0x7e03
Sample 13: 0xba03 0x2603 0x3503 0xa003
Sample 14: 0x0803 0x3303 0x3603 0xc803
Sample 15: 0x0c03 0x3803 0x9703 0x4403
It would seem to me, redirecting the output of hexdump to a temporary samples.txt file is inefficient, because for one you have to delete it at the end, and for two, perhaps using sed on the data in memory is faster than it would be on disk. How can I skip the samples.txt file and get the same results?

I think the following should work, if I understand your intentions correctly.
hexdump -e '8/1 "%02x " "\n"' samples.bin | cat -n | sed '....

Alternative with xxd and coreutils:
{
# Print header
printf "\t\tCh0 Ch1 Ch2 Ch3\n"
# Create hexdump of data
xxd -p samples.bin |
# Use 16 bit samples
fold -w4 |
# Add hex indicator
sed 's/^/0x/' |
# Data contains 4 interleaved channels
xargs -n4 |
# Number each sample
nl -n ln |
sed 's/ /:/; s/^/Sample /'
} > samples.out
Content of samples.out:
Ch0 Ch1 Ch2 Ch3
Sample 1: 0x1a03 0x1a03 0x4a03 0x5703
Sample 2: 0x4b03 0x4403 0x1e03 0x0904
Sample 3: 0x1003 0x1903 0x4003 0xae03
Sample 4: 0x1e03 0x2603 0x3303 0xad03
Sample 5: 0x1003 0x8403 0x4303 0x6203
Sample 6: 0xe003 0x1603 0x3403 0xc403
Sample 7: 0xf802 0x3b03 0x5303 0x6103
Sample 8: 0x1003 0x1503 0x4203 0x5803
Sample 9: 0x2303 0x1f03 0x5703 0x6203
Sample 10: 0x1703 0x7303 0x3103 0x3303
Sample 11: 0x1403 0xff02 0x3003 0x5103
Sample 12: 0x5f03 0x4203 0x4703 0x7e03
Sample 13: 0xba03 0x2603 0x3503 0xa003
Sample 14: 0x0803 0x3303 0x3603 0xc803
Sample 15: 0x0c0a 0x3803 0x9703 0x4403

Related

Address of segment descriptor

All values ​​are in hexadecimal number system. On Pentium in protected mode, registers have the following value: LDTR = 06000000, GDTR = 08000000, CR3 = 10000000, DS = 14, CS = 0034 CR0 = 00000001.
If the instruction (e.g. MOV AL, [2A66] accesses the logical address 2A66, what physical address does it access? At what address is the segment descriptor located? Current memory status, looking at absolute addresses is:
........
06000000 CD 20 FF 9F 00 9A EE FE 1D F0 4F 03 22 05 8A 03
06000010 22 05 17 03 22 93 0D 04 01 01 01 00 02 FF FF FF
.........
08000000 CA 20 FF 9F 00 9A E3 FE 1D F2 4F 08 23 05 8A 07
08000010 26 05 19 03 22 05 0D 04 01 02 01 00 02 FF FA FF
.........
10000020 3A 56 21 40 2A 38 42 18 2A 56 42 40 8E 48 42 18
10000030 2A 36 42 40 9A 48 42 18 7A 56 42 20 8E 48 42 18
10000040 23 60 42 40 4E A8 42 18 5A 56 42 40 8E 48 42 18
.........
40426860 C6 06 23 99 00 80 3E 1D 96 00 74 03 E9 99 00 E8
40426870 A6 01 E8 FF 03 75 19 80 3E C4 98 00 34 00 AD 0A
40426880 13 96 00 BA E9 89 75 03 E9 17 01 C6 06 1F 99 01
40426890 B8 00 6C BE 08 98 BB 21
.........
C6011D70 C6 06 23 99 00 80 3E 1D 96 00 74 03 E9 99 00 E8
C6011D80 A6 01 E8 FF 03 75 19 80 3E C4 98 00 34 00 AD 0A
C6011D90 13 96 00 BA E9 89 75 03 E9 17 01 C6 06 1F 99 01
Could you give me some guidelines what is the problem here and what I need to know to solve it? Operating systems and registry is new to me, so I don't know what I'm supposed to do here. I don't know even where should I start.

How to decode hex into separate bits in Perl?

A byte stored as two hex numbers contains a set of flags. I need to extract these flags, as 0's and 1's. Iterating over entries in a file with:
foreach(<>)
{
#line = split(/ /,$_);
$torR = !!((hex $line[4]) & 0x3); # bit 0 or 1
$torY = !!((hex $line[4]) & 0x4); # bit 2
$torG = !!((hex $line[4]) & 0x8); # bit 3
print "$torR,$torY,$torG\n";
}
run on data file:
796.129 [1f/01] len:7< 02 01 D5 01 8B 0A 8E
796.224 [1f/01] len:7< 02 01 D4 03 09 A9 B8
796.320 [1f/01] len:7< 00 01 D4 03 07 49 5A
796.415 [1f/01] len:7< 00 01 D4 00 11 A0 EE
796.515 [1f/01] len:7< 00 01 D4 00 00 31 4C
796.627 [1f/01] len:7< 02 01 D4 01 89 C1 FD
796.724 [1f/01] len:7< 02 01 D3 03 06 39 FD
796.820 [1f/01] len:7< 08 01 D4 03 08 40 6F
796.915 [1f/01] len:7< 08 01 D5 00 13 3D A4
797.015 [1f/01] len:7< 08 01 D4 00 00 34 04
Actual Result -
1,,
1,,
,,
,,
,,
1,,
1,,
,,1
,,1
,,1
Desired result:
1,0,0
1,0,0
0,0,0
0,0,0
0,0,0
1,0,0
1,0,0
0,0,1
0,0,1
0,0,1
Seems like 'false' gets stored as empty string instead of '0'.
Is there a neat trick to get this right at once, or do I need to convert empty strings to zeros "manually"?
If you want the true/false values to be numeric, you need to coerce them to be numeric:
$torR = 0 + !!((hex $line[4]) & 0x3); # bit 0 or 1
$torY = 0 + !!((hex $line[4]) & 0x4); # bit 2
$torG = 0 + !!((hex $line[4]) & 0x8); # bit 3
Keep in mind that the empty string '' is also a false value.
On the other hand, I might be inclined to write that as:
my (#ryg) = map 0 + !!((hex $line[4]) & $_), 0x3, 0x4, 0x5;
print join(', ', #ryg), "\n";
In addition, you would probably benefit from not using plain numbers in your program. Consider, for example, having a %FLAG structure that gives names to these constants, and a %COL structure that gives names to the columns you are interested in. Using the data you posted:
use Const::Fast;
const my %FLAG => (
torR => 0x3,
torY => 0x4,
torG => 0x5,
);
const my %COL => (
# ...
tor => 4,
);
while (my $line = <DATA>) {
my #line = split ' ', $line;
my %set_flags = map +($_ => 0 + !!((hex $line[$COL{tor}]) & $FLAG{$_})), qw(torR torY torG);
print join(', ', #set_flags{qw(torR torY torG)}), "\n";
}
__DATA__
796.129 [1f/01] len:7< 02 01 D5 01 8B 0A 8E
796.224 [1f/01] len:7< 02 01 D4 03 09 A9 B8
796.320 [1f/01] len:7< 00 01 D4 03 07 49 5A
796.415 [1f/01] len:7< 00 01 D4 00 11 A0 EE
796.515 [1f/01] len:7< 00 01 D4 00 00 31 4C
796.627 [1f/01] len:7< 02 01 D4 01 89 C1 FD
796.724 [1f/01] len:7< 02 01 D3 03 06 39 FD
796.820 [1f/01] len:7< 08 01 D4 03 08 40 6F
796.915 [1f/01] len:7< 08 01 D5 00 13 3D A4
797.015 [1f/01] len:7< 08 01 D4 00 00 34 04
I think I would use split and unpack to turn each value into an array of zeroes and ones, and then examine them individually
use strict;
use warnings 'all';
for my $val ( qw/ 02 02 00 00 00 01 01 08 08 08 / ) {
my #bits = split //, unpack 'b8', chr hex $val;
my $torR = $bits[0] || $bits[1] ? 1 : 0;
my $torY = $bits[2] ? 1 : 0;
my $torG = $bits[3] ? 1 : 0;
print "$torR,$torY,$torG\n";
}
output
1,0,0
1,0,0
0,0,0
0,0,0
0,0,0
1,0,0
1,0,0
0,0,1
0,0,1
0,0,1
Or here's a way using the Bit::Vector which produces the same result
use strict;
use warnings 'all';
use Bit::Vector;
for my $val ( qw/ 02 02 00 00 00 01 01 08 08 08 / ) {
my $vec = Bit::Vector->new_Hex(8, $val);
my $torR = $vec->Chunk_Read(2, 0) ? 1 : 0;
my $torY = $vec->Chunk_Read(1, 2) ? 1 : 0;
my $torG = $vec->Chunk_Read(1, 3) ? 1 : 0;
print "$torR,$torY,$torG\n";
}

sed search and replace specific column only

Im taking weather day from wunderground.com and I and then trimming down the data for use on gnuplot. Im having trouble replacing the second column only data from number to months abbreviations. Only interested in the second column.
I want to go from this;
>2013 08 02 23 37 00 73.3
>2013 08 02 23 42 00 73.4
>2013 08 02 23 45 00 73.3
>2013 08 02 23 47 00 73.1
>2013 08 02 23 52 00 73.1
>2013 08 02 23 57 00 73.1
To this:
>2013 AUG 02 23 37 00 73.3
>2013 AUG 02 23 42 00 73.4
>2013 AUG 02 23 45 00 73.3
>2013 AUG 02 23 47 00 73.1
>2013 AUG 02 23 52 00 73.1
>2013 AUG 02 23 57 00 73.1
i am trying to use sed to change the numbers into the correct month and i keep getting this. I only want the correct sed expression to execute not all of them. This is the command i am trying to use.
sed -e 's/01/JAN/' -e 's/02/FEB/' -e 's/03/MAR/' -e 's/04/APR/' -e 's/05/MAY/' -e 's/06/JUN/' -e 's/07/JUL/' -e 's/08/AUG/' -e 's/09/SEP/' -e 's/10/OCT/' -e 's/11/NOV/' -e 's/12/DEC/'
How would i go about this.
This might work for you (GNU sed):
sed -nri 'G;s/$/01JAN02FEB03MAR04APR05MAY06JUN07JUL08AUG09SEP10OCT11NOV12DEC/;s/ (..)(.*)\1(...)/ \3\2/;P' file
This adds a lookup table to the end of each line and substitutes the key for the value.
I would use awk for this:
$ awk 'BEGIN{split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",a)} {$2=a[$2+0]}1' a
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
To update the field with the new content, just redirect and then move:
awk .... file > temp_file && mv temp_file file
Explanation
What we do is to give awk a list of strings with the months names. Once we convert it into an array, a[1] will be Jan, a[2] Feb and so on. So then it is just a matter of replacing the 2nd field with a[2nd field].
BEGIN{split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",a)} fetches the data and inserts into the a[] array.
{$2=a[$2+0]} sets the 2nd field as a[2nd field]. The $2+0 is done to convert 08 to 8.
Finally 1 evaluates as true and makes awk perform its default action: {print $0}.
Workaround that works for this problem (since your first column is very predictable) but not the general question:
sed -E -e 's/^([0-9]{4}) 01/\1 JAN/' -e 's/^([0-9]{4}) 02/\1 FEB/' etc.
awk has a sub function that could get unwieldy for many options you have here.
Perl scripts might be the best way to go.
$ awk '{$2=substr("JanFebMarAprMayJunJulAugSepOctNovDec",(3*$2)-2,3)}1' file
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
Since it came up in a comment:
The idiomatic awk way to map from a month number to a name is:
number = (match("JanFebMarAprMayJunJulAugSepOctNovDec",<name>)+2)/3
and the above is just the natural inverse of that:
name = substr("JanFebMarAprMayJunJulAugSepOctNovDec",(3*<number>)-2,3)
Like with anything in awk there's various ways to get the output you want but IMHO the symmetry here makes it an attractive solution:
awk 'BEGIN{
months = "JanFebMarAprMayJunJulAugSepOctNovDec"
name = "Jul"
number = (match(months,name)+2)/3
print name " -> " number
name = substr(months,(3*number)-2,3)
print number " -> " name
}'
Jul -> 7
7 -> Jul
Notice that the script uses the same definition for months no matter which direction the conversion is being done and it's a similar math calculation in both directions.
Nothing wrong with doing it this way too of course:
awk 'BEGIN{
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",num2name)
for (number in num2name) {
name2num[num2name[number]] = number
}
name = "Jul"
number = name2num[name]
print name " -> " number
name = num2name[number]
print number " -> " name
}'
Jul -> 7
7 -> Jul
Just a few more lines of code, nbd.
Using gnu awk's function strftime() and mktime()
awk '{$2=strftime("%b",mktime("2014 " $2 " 1 1 0 0"))}1' file
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
Explanation
mktime("2014 " $2 " 1 1 0 0") fake the epoch time, use the column 2 as month
strftime("%b",mktime("2014 " $2 " 1 1 0 0")) convert the epoch back to the date, with %b, export the abbreviated month name (Jan, Feb, etc)
The benefit with this awk:
It is shorter, of course. Second, you can control/adjust the format in strftime() to export any date format you like.
For example, if change to full month name %B. You needn't rewrite the code.
awk '{$2=strftime("%B",mktime("2014 " $2 " 1 1 0 0"))}1' file

determining hash function used in digital signature

I have a digital signature (RSA - PKCS#1). After decrypting it with the RSA public key I get the following 128 bytes
00 01 ff ff ff .. ff 00 30 31 30 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 77 51 1b f4 d7 17 d7 ad 8c 2d e5 89 2a ca e0 6d a3 c0 7d 13 4d d7 b8 01 14 87 03 00 69 e4 9b b3
PKCS#1 padding removed, 51 bytes left:
30 31 30 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 77 51 1b f4 d7 17 d7 ad 8c 2d e5 89 2a ca e0 6d a3 c0 7d 13 4d d7 b8 01 14 87 03 00 69 e4 9b b3
I would like two things about this:
Is it possible to determine the hash function used? Encoded algorithm ID should be prepended to the actual body of the digest, is it possible to tell what algorithm it is from the raw bytes?
Where does the actual digest start (how long the head / digest is)?
This appears to be EMSA-PKCS1-v1_5 as described in RFC 3447, which means that after removing the header and padding, you have a DER encoding of an AlgorithmIdentifier followed by the hash value itself.
From the RFC:
For the six hash functions mentioned in Appendix B.1, the DER
encoding T of the DigestInfo value is equal to the following:
[...]
SHA-256: (0x)30 31 30 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 || H.
So in your example, the hash value is the SHA-256 hash starting 77511bf4d7....

Webbit websocket ws:// connection works but wss:// handshake fails silently without any error?

Upgraded Webbit to 0.4.6 to use the new SSL support but immediately realized that all wss:// handshakes are failing silently and I don't have any errors to show for it. Chrome only reports a "success" for a response without a HTTP code or any other headers. I check server logs and it doesn't even register an "open" event.
The catch here is that any ws:// connection works great. So what could be possible problems and how can I get an error out of it? Could it be something wrong with the java keystore and SSL handshake?
Edit
I was able to find an openSSL command for a test handshake. Here's the output:
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
Edit 2
I realized I could debug this further
CONNECTED(0000016C)
SSL_connect:before/connect initialization
write to 0x1f57750 [0x1f6a730] (210 bytes => 210 (0xD2))
0000 - 16 03 01 00 cd 01 00 00-c9 03 01 4f 6b 8d 68 63 ...........Ok.hc
0010 - 99 06 08 30 93 2a 42 88-f8 f1 c4 c5 dc 89 71 0b ...0.*B.......q.
0020 - b6 04 42 4e 11 79 b4 76-6c f7 66 00 00 5c c0 14 ..BN.y.vl.f..\..
0030 - c0 0a 00 39 00 38 00 88-00 87 c0 0f c0 05 00 35 ...9.8.........5
0040 - 00 84 c0 12 c0 08 00 16-00 13 c0 0d c0 03 00 0a ................
0050 - c0 13 c0 09 00 33 00 32-00 9a 00 99 00 45 00 44 .....3.2.....E.D
0060 - c0 0e c0 04 00 2f 00 96-00 41 00 07 c0 11 c0 07 ...../...A......
0070 - c0 0c c0 02 00 05 00 04-00 15 00 12 00 09 00 14 ................
0080 - 00 11 00 08 00 06 00 03-00 ff 01 00 00 44 00 0b .............D..
0090 - 00 04 03 00 01 02 00 0a-00 34 00 32 00 01 00 02 .........4.2....
00a0 - 00 03 00 04 00 05 00 06-00 07 00 08 00 09 00 0a ................
00b0 - 00 0b 00 0c 00 0d 00 0e-00 0f 00 10 00 11 00 12 ................
00c0 - 00 13 00 14 00 15 00 16-00 17 00 18 00 19 00 23 ...............#
00d2 - <SPACES/NULS>
SSL_connect:SSLv2/v3 write client hello A
read from 0x1f57750 [0x1f6fc90] (7 bytes => 0 (0x0))
12488:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:.\ssl\s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 210 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
Edit 3
Ok I've nailed the problem to Webbit initialization, but it doesn't throw any errors so I could use some input to getting getResourceAsStream functioning properly. Here's how the server is initialized:
def startWebSocketServer(webSocketHandler:PartialFunction[WebSocketEvent, Unit]) {
val webServer = WebServers.createWebServer(port)
try {
webServer.setupSsl(getClass.getResourceAsStream("/keystore"), "webbit")
webServer.add("/", new WebSocketEventAdapter(webSocketHandler))
webServer.start
} catch {
case e => e.printStackTrace()
}
}
Unfortunately setupSsl won't output any information, and I've tried both what I thought would be the path and inserting a fake path. In either case, I can't get an error. How on earth would I properly locate the path? Thanks!
The OMFG Answer
In a hysterical twist of fate, I found the problem. This particular issue took up 48 hours of my time, but the cause was not even code related and a funny miscommunication.
So as it turns out, another developer had copied our websocket code into a new file he was working on for development. All this time we were trying to debug code in a file that wasn't even executing at run-time. So upon further investigation we scrolled to the bottom of a very long and different file, and found the webbit init code and excuted it perfectly.
Moral of the story: don't commit an incomplete file to the master branch and point everyone there for debugging ;)