Kdb+ data fomat when writing to a file - kdb

Question

I'm trying to understand what happen when someone try to write to a file the next way:
q)h:hopen `:out
q)h (1 2;3)
3i
q)hclose h
q)read1 `:out
0x07000200000001000000000000000200000000000000f90300000000000000
this is not the same as a binary representation:
q)-8!(1 2;3)
0x010000002d00000000000200000007000200000001000000000000000200000000000000f90300000000000000
in which data format (1 2;3) was written into :out?
is there a way to read data from it back? - like -9!-8!(1 2;3)
how set/get related to these binary data formats? - do they use another 3-rd different binary data format?

Answer 1

Technically you can read the object if you have a little prior knowledge about the object, such that you can fabricate a header:
q)read1`:out
0x07000200000001000000000000000200000000000000f90300000000000000
q)-9!read1`:out
'badmsg
[0] -9!read1`:out
^
q)-9!{0x01,0x000000,(reverse 0x0 vs `int$count[x]+1+3+4+1+1+4),0x00,0x00,(reverse 0x0 vs 2i),x}read1`:out
1 2
3
The header here is comprised of:
0x01 - little endian
0x000000 - filler
message length (count of raw `x` plus the header additions)
0x00 - type (generic list = type 0) ... you have to know this in advance
0x00 - attributes .... none here, you would have to know this
length of list .... here we knew it was a 2-item list
x - the raw bytecode of the object without header
As rianoc pointed out, there are better ways to write such objects such that they can be more easily read without requiring advanced knowledge

Answer 2

-8! returns the IPC byte representation.
On disk the binary format is different. To read this data use get.
To stream data to a log file you need to create the empty file first
https://code.kx.com/q/kb/replay-log/#replaying-log-files
q)`:out set () /This important part adds a header to the file to set the type
`:out
q)h:hopen `:out
q)h (1 2;3)
600i
q)hclose h
q)get `:out
1 2
3
Note if you wish to write your item as a single element to the list then use enlist
q)`:out set ()
`:out
q)h:hopen `:out
q)h enlist (1 2;3)
616i
q)hclose h
q)get `:out
1 2 3
Binary and text data can also be written to a file which is what you are doing.
https://code.kx.com/q/ref/hopen/#files
The intention is to write specific pieces of data
q)h 0x2324 /Write some binary
q)h "some text\n" /Write some text
In your code the binary representation of the raw KX object does get written but no header is added (it is neither IPC nor disk format). So when you read back the data it cannot be interpreted correctly by either -9! or get.
Valid file binary created with `:out set () has a file header:
(Can be read by get)
q)read1 `:out
0xff0100000000000000000200000007000200000001000000000000000200000000000000f90300000000000000
Valid IPC binary with IPC header:
(Can be read by -9!)
q)-8!(1 2;3)
0x010000002d00000000000200000007000200000001000000000000000200000000000000f90300000000000000
Your raw object in binary - no header present to enable interpretation
q)read1 `:out
0x07000200000001000000000000000200000000000000f90300000000000000