kdb: guid encoding in c results in invalid serialization - kdb

I'm trying to manipulate guid from C++. Whenever I attempt to serialize a guid, I get a null pointer.
U g={0};
auto k = ku(g);
auto p = ::b9(2, k);
First two lines are straight from the manual for creating a null guid. This will result in p == 0.
Really what I was attempting to do was creating a list of guid and then serializing:
k = ktn(UU, 3)
kU(k)[0] = <an instance of U with the g bytes initialized>
kU(k)[1] = <an instance of U with the g bytes initialized>
kU(k)[2] = <an instance of U with the g bytes initialized>
That did not work when attempting to serialize.

I believe you should be using 3 as the first argument to b9. For example:
jmcmurray#homer ~/c $ more test.c
#include"k.h"
K f(K x)
{
K k = ktn(UU,3);I j=0;
for(j=0;j<3;j++){
U g={0};I i=0;
for(i=j;i<j+16;i++){
g.g[i] = (unsigned char)i;
}
kU(k)[0] = g;
}
return b9(3,k);
}
jmcmurray#homer ~/c $ gcc -shared -fPIC -DKXVER=3 test.c -o test.so
jmcmurray#homer ~/c $ q
KDB+ 3.5 2017.11.30 Copyright (C) 1993-2017 Kx Systems
l64/ 8()core 16048MB jmcmurray homer.aquaq.co.uk 192.168.1.57 EXPIRE 2019.06.30 AquaQ #52428
q)f:`:./test 2:(`f;1)
q)f[]
0x010000003e000000020003000000000002030405060708090a0b0c0d0e0f00ae67af727f000..
q)-9!f[]
00000203-0405-0607-0809-0a0b0c0d0e0f 001868af-727f-0000-6062-67af727f0000 a0a..
q)
Here I am able to return a serialised list of GUIDs from my shared object & deserialize on the q side. When I tried with 2 as in your example I got a 'type error when running the function in q.
According to https://code.kx.com/q/interfaces/capiref/#b9-serialize 3 means
unenumerate, compress, allow serialization of timespan and timestamp
2 is the same without "compress". So I guess you must compress GUIDs?

Related

Lsode throwing INTDY-- T (=R1) ILLEGAL and invalid input detected

I have my function:
function [result] = my_func(x,y)
result = y^2*(1-3*x)-3*y;
endfunction
Also, my vector with Ts, my function address and my initial variable x_0
load file_with_ts
# Add my limits as I also want to calculate those
# (all values in file_with_ts are within those limits.)
t_points = [-1, file_with_ts, 2]
myfunc = str2func("my_func")
x_0 = 0.9142
I am trying to execute the following line:
lsode_d1 = lsode(myfunc, x_0, t_points)
And expecting a result, but getting the following error:
INTDY-- T (=R1) ILLEGAL
In above message, R1 = 0.7987082301475D+00
T NOT IN INTERVAL TCUR - HU (= R1) TO TCUR (=R2)
In above, R1 = 0.8091168896311D+00 R2 = 0.8280400838323D+00
LSODE-- TROUBLE FROM INTDY. ITASK = I1, TOUT = R1
In above message, I1 = 1
In above message, R1 = 0.7987082301475D+00
error: lsode: invalid input detected (see printed message)
error: called from
main at line 20 column 10
Also, the variable sizes are:
x_0 -> 1x1
t_points -> 1x153
myfunc -> 1x1
I tried transposing the t_points vector
using #my_func instead of the str2func function
I tried adding multiple variables as the starting point (instead of x_0 I entered [x_0; x_1])
Tried changing my function header from my_func(x, y) to my_func(y, x)
Read the documentation and confirmed that my_func allows x to be a vector and returns a vector (whenever x is a vector).
EDIT: T points is the following 1x153 matrix (with -1 and 2 added to the beggining and the end respectively):
-4.9451e-01
-4.9139e-01
-4.7649e-01
-4.8026e-01
-4.6177e-01
-4.5412e-01
-4.4789e-01
-4.2746e-01
-4.1859e-01
-4.0983e-01
-4.0667e-01
-3.8436e-01
-3.7825e-01
-3.7150e-01
-3.5989e-01
-3.5131e-01
-3.4875e-01
-3.3143e-01
-3.2416e-01
-3.1490e-01
-3.0578e-01
-2.9267e-01
-2.9001e-01
-2.6518e-01
-2.5740e-01
-2.5010e-01
-2.4017e-01
-2.3399e-01
-2.1491e-01
-2.1067e-01
-2.0357e-01
-1.8324e-01
-1.8112e-01
-1.7295e-01
-1.6147e-01
-1.5424e-01
-1.4560e-01
-1.1737e-01
-1.1172e-01
-1.0846e-01
-1.0629e-01
-9.4327e-02
-8.0883e-02
-6.6043e-02
-6.6660e-02
-6.1649e-02
-4.7245e-02
-2.8332e-02
-1.8043e-02
-7.7416e-03
-6.5142e-04
1.0918e-02
1.7619e-02
3.4310e-02
3.3192e-02
5.2275e-02
5.5756e-02
6.8326e-02
8.2764e-02
9.5195e-02
9.4412e-02
1.1630e-01
1.2330e-01
1.2966e-01
1.3902e-01
1.4891e-01
1.5848e-01
1.7012e-01
1.8026e-01
1.9413e-01
2.0763e-01
2.1233e-01
2.1895e-01
2.3313e-01
2.4092e-01
2.4485e-01
2.6475e-01
2.7154e-01
2.8068e-01
2.9258e-01
3.0131e-01
3.0529e-01
3.1919e-01
3.2927e-01
3.3734e-01
3.5841e-01
3.5562e-01
3.6758e-01
3.7644e-01
3.8413e-01
3.9904e-01
4.0863e-01
4.2765e-01
4.2875e-01
4.3468e-01
4.5802e-01
4.6617e-01
4.6885e-01
4.7247e-01
4.8778e-01
4.9922e-01
5.1138e-01
5.1869e-01
5.3222e-01
5.4196e-01
5.4375e-01
5.5526e-01
5.6629e-01
5.7746e-01
5.8840e-01
6.0006e-01
5.9485e-01
6.1771e-01
6.3621e-01
6.3467e-01
6.5467e-01
6.6175e-01
6.6985e-01
6.8091e-01
6.8217e-01
6.9958e-01
7.1802e-01
7.2049e-01
7.3021e-01
7.3633e-01
7.4985e-01
7.6116e-01
7.7213e-01
7.7814e-01
7.8882e-01
8.1012e-01
7.9871e-01
8.3115e-01
8.3169e-01
8.4500e-01
8.4168e-01
8.5705e-01
8.6861e-01
8.8211e-01
8.8165e-01
9.0236e-01
9.0394e-01
9.2033e-01
9.3326e-01
9.4164e-01
9.5541e-01
9.6503e-01
9.6675e-01
9.8129e-01
9.8528e-01
9.9339e-01
Credits to Lutz Lehmann and PierU.
The problem lied in the array t_points not being a monotonous array. Adding a sort(t_points) before doing any calculations fixed the error.

Importing .mat file in Fortran - Segmentation Fault error

I am writing this F90 program to compute a function in fortran which takes the input from a .mat file and save the results in another .mat file.
I followed this answer to get the code compiled and correctly linked. This is my makefile command:
gfortran -g -fcheck=all binhkorn_mat.F90 -I/usr/local/MATLAB/R2015b/extern/include/ -L/usr/local/MATLAB/R2015b/bin/glnxa64 -cpp -o binhkorn_mat -lmat -lmx -Wl,-rpath /usr/local/MATLAB/R2015b/bin/glnxa64/
The output file is apparently correctly compiled, but then once I run the program the following SF appears (I'm working on LINUX Ubuntu 14.04 LTS):
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x7FD650063F27
#1 0x7FD6500644F4
#2 0x7FD64FCBCD3F
#3 0x7FD64FDD7AF6
#4 0x400A3F in binhkorn_mat at binhkorn_mat.F90:17 (discriminator 2)
./binhkorn_mat: Segmentation fault
I can't figure out if there's an error with the compiler or if I did something wrong with the pointers/functions definitions. Here's the code (binhkorn_mat.F90):
#include "fintrf.h"
PROGRAM binhkorn_mat
IMPLICIT NONE
mwPointer matOpen, matGetVariable, matPutVariable
mwPointer mpin, mpX, mpout, mpcf
INTEGER :: i,j
REAL*8, DIMENSION(2) :: x
REAL*8, DIMENSION(4) :: cf
!input/output through .mat f
mpin = matOpen('X.mat', 'u')
mpX = matGetVariable(mpin, 'X')
CALL mxCopyPtrToReal8(mpX, x, 2)
CALL matClose(mpin)
!fitness functions
cf(1) = ((x(1)-2)**2 + (x(2)-1)**2 + 2)
cf(2) = (9*x(1) + (x(2)-1)**2)
!constraints
cf(3) = x(1)*x(1) + x(2)*x(2) - 225
cf(4) = x(1) - 3*x(2) + 10
!output file created
CALL mxCopyReal8ToPtr(cf, mpcf, 4)
mpout = matOpen('cf.mat', 'w')
mpcf = matPutVariable(mpout, 'cf', mpcf)
CALL matClose(mpout)
END PROGRAM
The X.mat file is correctly created by an external Matlab script and contains a variable named X which is a 2-element row vector.
I basically misunderstood how to use of many functions. The pointers i supplied as input to some of them were not the correct ones. I post here the working solution:
#include "fintrf.h"
PROGRAM binhkorn_mat
IMPLICIT NONE
mwPointer matOpen, matGetVariable!, matPutVariable
mwPointer mxGetData, mxGetNumberOfElements, mxCreateNumericArray
mwPointer mpin, mpX, mpout, mpcf
mwSize ndim
mwSize dims(2)
INTEGER :: s
INTEGER*4 mxClassIDFromClassName
CHARACTER (LEN = 6) :: classname
REAL*8, DIMENSION(2) :: x
REAL*8, DIMENSION(4) :: cf
!input/output through .mat f
mpin = matOpen('X.mat', 'r')
mpX = matGetVariable(mpin, 'X')
CALL mxCopyPtrToReal8(mxGetData(mpX), x, mxGetNumberOfElements(mpX))
!CALL matClose(mpin)
!fitness functions
cf(1) = ((x(1)-2)**2 + (x(2)-1)**2 + 2)
cf(2) = (9*x(1) + (x(2)-1)**2)
!constraints
cf(3) = x(1)*x(1) + x(2)*x(2) - 225
cf(4) = x(1) - 3*x(2) + 10
!output .mat file created and filled
s = size(cf)
ndim = 2
classname = 'double'
dims(1) = 1
dims(2) = s
mpcf = mxCreateNumericArray(ndim, dims, mxClassIDFromClassName(classname), 0)
CALL mxCopyReal8ToPtr(cf, mxGetData(mpcf), mxGetNumberOfElements(mpcf))
mpout = matOpen('cf.mat', 'w')
CALL matPutVariable(mpout, 'cf', mpcf)
!CALL matClose(mpout)
END PROGRAM

"LapackError: Parameter a has non-native byte order in lapack_lite.dgesdd" when importing from Matlab files

After importing this data file from Matlab with scipy.io.loadmat, things appeared to work fine until we tried to calculate the conditioning number of one of the matrixes within.
Here's the minimum amount of code that reproduces for us:
import scipy
import numpy
stuff = scipy.io.loadmat("dati-esercizio1.mat")
numpy.linalg.cond(stuff["A"])
Here's the extended stacktrace courtesy of iPython:
In [3]: numpy.linalg.cond(A)
---------------------------------------------------------------------------
LapackError Traceback (most recent call last)
/snip/<ipython-input-3-15d9ef00a605> in <module>()
----> 1 numpy.linalg.cond(A)
/snip/python2.7/site-packages/numpy/linalg/linalg.py in cond(x, p)
1409 x = asarray(x) # in case we have a matrix
1410 if p is None:
-> 1411 s = svd(x,compute_uv=False)
1412 return s[0]/s[-1]
1413 else:
/snip/python2.7/site-packages/numpy/linalg/linalg.py in svd(a, full_matrices, compute_uv)
1313 work = zeros((lwork,), t)
1314 results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt,
-> 1315 work, -1, iwork, 0)
1316 lwork = int(work[0])
1317 work = zeros((lwork,), t)
LapackError: Parameter a has non-native byte order in lapack_lite.dgesdd
All obvious ideas (like flattening and reshaping the matrix or recreating the matrix from scratch reassigning it element by element) failed. How can I want to massage the data, then, in order to make it more agreeable with numpy?
It's a bug, fixed some time ago: https://github.com/numpy/numpy/pull/235
Workaround:
np.linalg.cond(stuff['A'].newbyteorder('='))
This works for me:
In [33]: stuff = loadmat('dati-esercizio1.mat')
In [34]: a = stuff['A']
In [35]: try: np.linalg.cond(a)
....: except: print "Fail!"
Fail!
In [36]: b = np.array(a, dtype='>d')
In [37]: np.linalg.cond(b)
Out[37]: 62493201976.673141
In [38]: np.all(a == b) # Verify they hold the same data.
Out[38]: True
Apparently it's something wrong with the byte order (endianness?) of each number in the resulting ndarray and not just with the ndarray object itself.
Something like this but more elegant should do the trick:
n, m = A.shape()
B = numpy.empty_like(A)
for i in xrange(n):
for j in xrange(m):
B[i,j] = float(A[i,j])
del A
B = A
print numpy.linalg.cond(A) # 62493210091.354507
(For some reason an in-place replacement still gives that error - so there's something wrong with the byte order of the whole object, too.)

Length of data to hash for PGP

I have finally managed to verify some simple PGP signed message blocks. However, I discovered that for some reason, my implementation limits me to verifying data that is 9-16 bytes long. no less. no more.
is there some instruction somewhere (RFC4880 or elsewhere) that specifies how to deal with plaintext data of any length? maybe there is some sort of padding i missed? pkcs1?
I am pretty sure i formatted the data to hash properly, since the instructions in RFC 4880 sec 5.2.4 say for text documents, just replace all \n with \r\n and add a trailer. since my test values were single lines of data, nothing had to be replaced
all of these values are in base 10 unless otherwise noted:
// DSA public key values
p = 175466718616740411615640156350265486163809613514213656685227237159351776260193236923030228927905671867677337184318134702903960237546408302010360724274436019639502405323187799029742776686067449287558904042137172927936686590837020160292525250748155580652384740664931255981772117478967314777932252547256795892071
q = 809260232002608708872165272150356204306578772713
g = 127751900783328740354741342100721884490035793278553520238434722215554870393020469115393573782393994875216405838455564598493958342322790638050051759023658096740912555025710033120777570527002197424160086000659457154926758682221072408093235236853997248304424303705425567765059722098677806247252106481642577996274
y = 172935968966072909036304664996424500241381878537444332146572958203083745609400290814117451480512268901233962890933482206538294509037615827035398352528065134903071886710296983781453184598843331365336270501467458073523376152406987560592548479865116940266729198119357206749848310472131186772143408998928864559411
not working:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
abcd
-----BEGIN PGP SIGNATURE-----
Version: BCPG v1.39
iFsEARECABsFAk/tB28UHGFiYyA8bWFrY21AYWFhLmNvbT4ACgkQMFIlRc933Ya2
RwCfdMyI08Iz0rDXVHOPlGA3s5Y9j/8An2He7+hHjWfGJNoOJT7gAxqJaoLo
=I2rT
-----END PGP SIGNATURE-----
data hashed (in hex): 6162636404011102001b05024fed076f141c616263203c6d616b636d406161612e636f6d3e04ff00000021
r = 666804200764671083282351405489424949903645052927
s = 558743769080942454889260816818443017172325925608
w = 702955297882281869313155599553522395227576660460 // s^-1 mod q
u1 = 190417717173929082607343542521304347388874234334
u2 = 306786785479358548892951170619047936651163362761
v = g^u1 * y^u2 % p % q = 737052148656331043521702886300418501784667890334
v != r
working:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
0123456789abcdef
-----BEGIN PGP SIGNATURE-----
Version: BCPG v1.39
iFsEARECABsFAk/tCE0UHGFiYyA8bWFrY21AYWFhLmNvbT4ACgkQMFIlRc933YYG
IQCfercgPsXFnah6otgQdEMbv9OeCgIAnRIyOLirbqSlBugBT6Ex/Adz4+7L
=bzab
-----END PGP SIGNATURE-----
data hashed (in hex): 3031323334353637383961626364656604011102001b05024fed084d141c616263203c6d616b636d406161612e636f6d3e04ff00000021
r = 700580719365380086754774917458461236187098909186
s = 103881812262595813943381509986903840453887782603
w = 178510125628083028184051840492924307896586330444 // s^-1 mod q
u1 = 78831508775508876446567239486098677466912246622
u2 = 572875590470993668032596348682349224460207395691
v = g^u1 * y^u2 % p % q = 700580719365380086754774917458461236187098909186
v == r
what data did i not include in the hash / what did i do wrong?
EDIT: here is the public key as requested, even though the relevant values have been posted already)
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: BCPG v1.39
mQGiBE5B0h8RBAD533Z5bK1IpBx02QyQL0QoJE4uFRIMGDiwXuwmZzVl+R7Vlurd
GRLsCCbE6vOOh7XQVZGzLEBy9WNzZ9m+EbCfSVAYkjS6FhLws6hG6irrnS+b3JBf
gFJ8vNGF9Z7bhx+7y7NBk0IMyWkGnUkcnav73t5FQUI2faEBN4c/yAGJZwCgjcB7
3akWk9XVWvTCsiMXxpyvkukEALXsvB6cOoFEtQq9cQHjP63fBlvD94dhhMiM0cH6
hW9JotxdK+cxFGG9ZIWgoN2PWbMJka/H4W5EL6tS+YiNAR7I1Ozkt6X16GjnQUzZ
MlSpleK+KiKVN2anRaPEoOIinHrE3ZXd6QlJ/4+OJn4IVWmSEaJpFf4QNgvEu4rh
xinyBAD2RNzREOA+wpnFZ4lDt9NZXmXdxQME/l0J9XcvWhpGsxA/MATQKImy7N49
7GT/M38F+TrpBobag1O3buE99fOLyws4Tbc+sZMdHxoiGZDAIRNQS2rv475E6ktj
7vd5CYvOkA6+8sX1+hPcNlkHtHB1OFkJRsYp6k0zkyC9adjBM7QTYWJjIDxtYWtj
bUBhYWEuY29tPohGBBMRAgAGBQJOQdIfAAoJEDBSJUXPd92GRSQAoItbtbToOg7a
/hcg2sA/aBEQNwuxAKCGR69vmSoCWoBP5waPk0UsjM3BSbjMBE5B0h8QAgCUlP7A
lfO4XuKGVCs4NvyBpd0KA0m0wjndOHRNSIz44x24vLfTO0GrueWjPMqRRLHO8zLJ
S/BXO/BHo6ypjN87Af0VPV1hcq20MEW2iujh3hBwthNwBWhtKdPXOndJGZaB7lsh
LJuWv9z6WyDNXj/SBEiV1gnPm0ELeg8Syhy5pCjMAf9QHehP2eCFqfEwTAnaOlA6
CU+rYHKPZaI9NUwCA7qD2d93/l08/+ZtFvejZW1RWrJ8qfLDRtlPgRzigoF/CXbR
iEYEGBECAAYFAk5B0h8ACgkQMFIlRc933YZRrACfUnWTjHHN+QsEEoJrwRvFmvzj
bR4An24pTpeeN+I6R59O/sdmYsAhjULX
=sStS
-----END PGP PUBLIC KEY BLOCK-----
Haven't got enough time to look up the details, but I would guess that you're applying (or not applying) padding correctly. That would cause the right result to come up for some input lengths, but not for others.
I guess I'll look into this more, but I wanted to get something in under the bounty wire :)
Edit: Ok, found an error. Not sure why you're getting it, but if it's fixed, then the right answer comes out. In your not-working example, you calculate w (s^-1 mod q) as
w = 702955297882281869313155599553522395227576660460 // s^-1 mod q
but I get
w = 702955297882281869313155599553522395227576660458
off by 2! Really, really close values though. And it can be shown that mine is right:
s * your_w mod q = 308227306159276200906356361486529830038073078504
s * my_w mod q = 1
If you plug in this w value, you then get
u1 = 536931432138658080437983667536052790245747416035
u2 = 591698847955233800072578903940910445457030802333
v = (g^u1 * y^u2) % p % q = 666804200764671083282351405489424949903645052927
r == v
Hope that helps.

g++ unicode variable name

I am trying to use unicode variable names in g++.
It does not appear to work.
Does g++ not support unicode variable names, ... or is there some subset of unicode (from which I'm not testing in).
Thanks!
You have to specify the -fextended-identifiers flag when compiling, you also have to use \uXXXX or \uXXXXXXXX for unicode(atleast in gcc it's unicode)
Identifiers (variable/class names etc) in g++ can't be of utf-8/utf-16 or whatever encoding,
they have to be:
identifier:
nondigit
identifier nondigit
identifier digit
a nondigit is
nondigit: one of
universalcharactername
_ a b c d e f g h i j k l m n o p q r s t u v w x y z
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
and a universalcharactername is
universalcharactername:
\UXXXXXXXX
\uXXXX
Thus, if you save your source file as UTF-8, you cannot have a variable like e.g.:
int høyde = 10;
it had to be written like:
int h\u00F8yde = 10;
(which imo would beat the whole purpose - so just stick with a-z)
A one-line patch to the cpp preprocessor allows UTF-8 input. Details for gcc are given at
https://www.raspberrypi.org/forums/viewtopic.php?p=802657
however, since the preprocessor is shared, the same patch should work for g++ as well. In particular, the patch needed, as of gcc-5.2 is
diff -cNr gcc-5.2.0/libcpp/charset.c gcc-5.2.0-ejo/libcpp/charset.c
*** gcc-5.2.0/libcpp/charset.c Mon Jan 5 04:33:28 2015
--- gcc-5.2.0-ejo/libcpp/charset.c Wed Aug 12 14:34:23 2015
***************
*** 1711,1717 ****
struct _cpp_strbuf to;
unsigned char *buffer;
! input_cset = init_iconv_desc (pfile, SOURCE_CHARSET, input_charset);
if (input_cset.func == convert_no_conversion)
{
to.text = input;
--- 1711,1717 ----
struct _cpp_strbuf to;
unsigned char *buffer;
! input_cset = init_iconv_desc (pfile, "C99", input_charset);
if (input_cset.func == convert_no_conversion)
{
to.text = input;
Note that for the above patch to work, a recent version of iconv needs to be installed that supports C99 conversions. Type iconv --list to verify this, otherwise, you can install a new version of iconv along with gcc as described in the link above. Change the configure command to
$ ../gcc-5.2.0/configure -v --disable-multilib \
--with-libiconv-prefix=/usr/local/gcc-5.2 \
--prefix=/usr/local/gcc-5.2 \
--enable-languages="c,c++"
if you are building for x86 and want to include the c++ compiler as well.