I have a dataset that consists of IP addresses packets transferred etc. Octave is converting the IP addresses into float values. How do I import IP addresses as it is and what data type is to be used (String?)?
How do you import them in octave? GNU Octave has plenty of functions to load/save data. It depends how your IP adresses (IPv4 or IPv6?) are stored in your file which function is the easiest/best for you.
For example if you have a file called "ips.txt".
192.168.10.4
8.8.8.8
14.32.244.8
You could use this to get a cell:
octave:1> f = fopen("ips.txt", "r");
octave:2> l = textscan(f, "%s");
octave:3> fclose(f);
octave:4>
octave:4> l{1}
ans =
{
[1,1] = 192.168.10.4
[2,1] = 8.8.8.8
[3,1] = 14.32.244.8
}
octave:5>
But perhaps char(fread(..)) or fgetl may be better for you dependent what you want to do later with the imported ips...
addition:
Because you commented that you have IP addresses are inside a list of floats and not a fixed scheme (a fixed scheme would be for example: "The wanted ips are at the beginning of the line, the 4.th column or somethink like that which could be processed with awk) I also add a possibility with regexp:
I created this file ips.txt:
192.168.10.4 some text 3.14 8.8.8.8
other 123.44 14.32.244.8
4.667.2 12.943 127.0.0.1 hello world
And load it with this script using regexp
f = fopen("ips.txt", "r");
x = char(fread(f));
fclose(f);
[S, E, TE, M, T, NM, SP] = regexp (x', '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}');
which gives you for M:
M =
{
[1,1] = 192.168.10.4
[1,2] = 8.8.8.8
[1,3] = 14.32.244.8
[1,4] = 127.0.0.1
}
-- Andy
Related
I am looking for the most simple protocol to program a remote function call, e.g. from Matlab to Julia.
[out1, out2, ...] = jlcall(socket, fname, arg1, arg2, ...);
fname is a string, all other input and output variables are numerical arrays (or other structures known to both sides) on Linux, Windows as option.
Client: open connection, block, pack and transmit
Server: receive, unpack, process, pack and transmit back
Client: receive, unpack, close connection and continue
The solutions I've seen (tcp, zmq) were built with old versions and do no longer work.
Protocol could (should?) be limited to do the pack/transmit - receive/unpack work.
UPDATE
Here is what I have come up with using pipes:
function result = jlcall(varargin)
% result = jlcall('fname', arg1, arg2, ...)
% call a Julia function with arguments from Matlab
if nargin == 0 % demo
task = {'foo', 2, 3}; % demo fun, defined in jsoncall.jl
else
task = varargin;
end
% create pipe and write function and parameter(s) to pipe
pipename = tempname;
pipe = java.io.FileOutputStream(pipename);
pipe.write(uint8(jsonencode(task)));
pipe.close;
% run Julia and read result back
system(sprintf('julia jsoncall.jl %s', unixpath(pipename)))
fid = fopen(pipename, 'r');
c = fread(fid);
result = jsondecode(char(c'));
fclose(fid);
function path_unix = unixpath(path_pc)
%convert path to unix version
path_unix = path_pc;
path_unix(strfind(path_unix,'\'))='/';
# jsoncall.jl
using JSON3 # get JSON3.jl from repository first
function foo(a,b) # demo function
a+b, a*b
end
jsonfile = ARGS[1] # called as > julia jsoncall.jl <json_cmdfile>
io = open(jsonfile, "r") # open IOStream for read
data = read(io) # read UTF8 data from stream
close(io) # close stream
cmd = JSON3.read(String(data)) # unpack stream into [fun, farg] array
fun = Symbol(cmd[1]) # first element is Julia function name,
result = #eval $fun(cmd[2:end]...) # others are function arguments
io = open(jsonfile, "w") # open IOStream for write
write(io, JSON3.write(result)) # (over-)write result back to stream
close(io) # close stream
Open points:
my first use of pipes/streams
output formatting: where Julia outputs a tuple of two, Matlab creates an an nx2 array.
replace json by msgpack for performance, might help with type formatting as well.
Your comments are welcome!
Here is a stripped down way of doing it. If you are going to vary your functions and arguments, a REST as in the comments server is going to be more flexible and less likely to pose a security risk (as you are eval()ing arbitrary code in some cases).
#server code
using Sockets
const port = 6001
const addr = ip"127.0.0.1"
const server = listen(addr, port)
while true
try
#info "Server on $port awaiting request..."
sock = accept(server)
#info "Server connected."
msg = strip(readline(sock))
#info "got message $msg"
fstr, argstr = split(msg, limit=2)
x = parse(Float64, argstr) # or other taint checks here...
ans = eval(Meta.parse(fstr * "($x)"))
#info "server answer: $ans"
write(sock, "$ans\n")
catch y
#info "exiting on condition: $y"
end
end
# client code
using Sockets
port = 6001
server = ip"127.0.0.1"
sock = connect(server, port)
#info "Client connected to $server"
func = "sinpi"
x = 0.5
#info "starting send"
write(sock, "$func $x\n")
flush(sock)
#info "flushed send"
msg = strip(readline(sock)) # read one line of input and \n, remove \n
ans = parse(Float64, msg)
println("answer is $ans")
close(sock)
I am having a weird "ValueError: mean must be 1 dimensional" when I am trying to build a Hierarchical GL-LVM model. Basically I'm trying to reproduce this paper: Hierarchical Gaussian Process Latent Variable Models using GPflow.
Therefore I implemented my own new model as follow:
class myGPLVM(gpflow.models.BayesianModel):
def __init__(self, data, latent_data, x_data_mean, kernel):
super().__init__()
print("GPLVM")
self.kernel0 = kernel[0]
self.kernel1 = kernel[1]
self.mean_function = Zero()
self.likelihood0 = gpflow.likelihoods.Gaussian(1.0)
self.likelihood1 = gpflow.likelihoods.Gaussian(1.0)
# make some parameters
self.data = (gpflow.Parameter(x_data_mean), gpflow.Parameter(latent_data), data)
def hierarchy_ll(self):
x, h, y = self.data
K = self.kernel0(x)
num_data = x.shape[0]
k_diag = tf.linalg.diag_part(K)
s_diag = tf.fill([num_data], self.likelihood0.variance)
ks = tf.linalg.set_diag(K, k_diag + s_diag)
L = tf.linalg.cholesky(ks)
m = self.mean_function(x)
return multivariate_normal(h, m, L)
def log_likelihood(self):
"""
Computes the log likelihood.
.. math::
\log p(Y | \theta).
"""
x, h, y = self.data
K = self.kernel1(h)
num_data = h.shape[0]
k_diag = tf.linalg.diag_part(K)
s_diag = tf.fill([num_data], self.likelihood1.variance)
ks = tf.linalg.set_diag(K, k_diag + s_diag)
L = tf.linalg.cholesky(ks)
m = self.mean_function(h)
# [R,] log-likelihoods for each independent dimension of Y
log_prob = multivariate_normal(y, m, L). # <- trows the error!
log_prob_h = self.hierarchy_ll()
log_likelihood = tf.reduce_sum(log_prob) + tf.reduce_sum(log_prob_h)
return log_likelihood
The model seems to work with a toy example:
from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=40, centers=3, n_features=12, random_state=2)
Y = tf.convert_to_tensor(X, dtype=default_float())
but fails and trough me the error when I am trying with a bvh file (the one from the paper actually). I also used Lawrence's code to read my bvh from mocap which I modified to fit python3
Anyway, it's been few a days and I am out of ideas. I tried multiple way to force my mean array "m" to be of one dimensional but nothing worked. I also tried with the "three_phase_oil_flow" dataset from the first GPLVM paper which works as well.
Therefore, I would assume that my model is correct, or at least I got some optimisation going on, and would think that perhaps the bvh reader could be the cause. But the data seems all fine to me... Especially I don't understand why when forcing multivariate function like:
m = np.zeros((np.shape(m)[0], 1))
log_prob = multivariate_normal(y, m, L)
or even with the gpflow Zero function
m = Zero(h)
log_prob = multivariate_normal(y, m, L)
it still trows me the error. Any help will be highly appreciated.
edited thanks to: Artem Artemev
The rest of the code if anyone wants to try to reproduce:
https://github.com/michaelStettler/h-GPLVM
error flow:
(venv) MacBookMichael2:stackOverflow michaelstettler$ python3 HGPLVM.py
(199, 96)
shape Y (199, 3, 38)
2020-01-26 17:00:48.104029: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-26 17:00:48.113609: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8dd5ff5410 executing computations on platform Host. Devices:
2020-01-26 17:00:48.113627: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
shape Y (199, 38)
Number of points: 199 and Number of dimensions: 38
shape x_mean_latent (199, 8)
shape x_mean_init (199, 2)
HGPLVM
gpr_data (199, 2) (199, 8) (199, 38)
2020-01-26 17:00:48.139003: W tensorflow/python/util/util.cc:299] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
shape m (199, 1)
Traceback (most recent call last):
File "HGPLVM.py", line 131, in <module>
_ = opt.minimize(closure, method="bfgs", variables=model.trainable_variables, options=dict(maxiter=maxiter))
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/gpflow/optimizers/scipy.py", line 60, in minimize
**scipy_kwargs)
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/scipy/optimize/_minimize.py", line 594, in minimize
return _minimize_bfgs(fun, x0, args, jac, callback, **options)
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 998, in _minimize_bfgs
gfk = myfprime(x0)
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 327, in function_wrapper
return function(*(wrapper_args + args))
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 73, in derivative
self(x, *args)
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 65, in __call__
fg = self.fun(x, *args)
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/gpflow/optimizers/scipy.py", line 72, in _eval
loss, grads = _compute_loss_and_gradients(closure, variables)
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/gpflow/optimizers/scipy.py", line 116, in _compute_loss_and_gradients
loss = loss_cb()
File "HGPLVM.py", line 127, in closure
return - model.log_marginal_likelihood()
File "/Users/michaelstettler/PycharmProjects/GPflow/venv/lib/python3.6/site-packages/gpflow/models/model.py", line 45, in log_marginal_likelihood
return self.log_likelihood(*args, **kwargs) + self.log_prior()
File "HGPLVM.py", line 62, in log_likelihood
log_prob = multivariate_normal(y, m, L)
File "mtrand.pyx", line 3729, in numpy.random.mtrand.RandomState.multivariate_normal
ValueError: mean must be 1 dimensional
I would recommend posting a working MWE code. I have tried to use your code snippets, but it gives me errors.
I don't have issues with multivariate_normal function. If you have localised the issue correctly you can debug TF2.0 more thoroughly and find the place that causes that exception. Here is the code which I'm running:
In [2]: from sklearn.datasets.samples_generator import make_blobs
...: X, y = make_blobs(n_samples=40, centers=3, n_features=12, random_state=2)
In [10]: m = np.zeros((np.shape(y)[0], 1))
In [11]: m.shape
Out[11]: (40, 1)
In [12]: y.shape
Out[12]: (40,)
In [13]: L = np.eye(m.shape[0])
In [15]: gpflow.logdensities.multivariate_normal(y, m, L)
Out[15]:
<tf.Tensor: shape=(40,), dtype=float64, numpy=
array([ -56.75754133, ...])>
I am working on a robotic arm.
M106 is turn on the fan
M17 is stepper on
M18 is stepper off
G1 X... Y.. X.. is the coordinates of movement
the port is correct, the terminal prints the hello hi there...
However the robotic arm is not moving, I totally have no clue why is this happening.
Is it there is some problem with my code?
import serial
import struct
def gcode_encode(gcode):
gcode += '\r\n'
return struct.pack(f'<{len(gcode)}s', gcode.encode(encoding='utf-8'))
print("hello")
# ser = serial.Serial('COM7', 9600, timeout=0, parity=serial.PARITY_EVEN, rtscts=1)
ser = serial.Serial()
ser.port = 'COM7'
ser.baudrate = 9600
ser.timeout = 0
ser.open()
g = gcode_encode('M106')
ser.write(b'g')
g = gcode_encode('M17')
ser.write(b'g')
g = gcode_encode('M18')
ser.write(b'g')
g = gcode_encode('G1 X0 Y120 Z120')
ser.write(b'g')
g = gcode_encode('G1 X50 Y120 Z60')
ser.write(b'g')
ser.close()
print("hi")
You are writing only the character 'g' to the port. If you want to write bytes of a variable g, you need to use bytes(g). The same is with f'<{len(gcode)}s', the characters in single or double quotes are not a command here, but just a string. Also you don't need packing of the string, just encoding.
Also add some pauses between commands using time.sleep().
I am trying to read from a file and display the data in rows 6, 11, 111 and 127 in Matlab. I could not figure out how to do it. I have been searching Matlab forums and this platform for an answer. I used fscanf, textscan and other functions but they did not work as intended. I also used a for loop but again the output was not what I wanted. I can now only read one row and display it. Simply I want to display all of them(data in rows given above) at the same time. How can I do that?
matlab code
n = [0 :1: 127];
%% Problem 1
figure
x1 = cos(0.17*pi*n)
%it creates file and writes content of x1 to the file
fileID = fopen('file.txt','w');
fprintf(fileID,'%d \n',x1);
fclose(fileID);
%line number can be changed in order to obtain wanted values.
fileID = fopen('file.txt');
line = 6;
C = textscan(fileID,'%s',1,'delimiter','\n', 'headerlines',line-1);
celldisp(C)
fclose(fileID);
and this is the file
1
8.607420e-01
4.817537e-01
-3.141076e-02
-5.358268e-01
-8.910065e-01
-9.980267e-01
-8.270806e-01
-4.257793e-01
9.410831e-02
5.877853e-01
9.177546e-01
9.921147e-01
7.901550e-01
3.681246e-01
-1.564345e-01
-6.374240e-01
-9.408808e-01
-9.822873e-01
-7.501111e-01
-3.090170e-01
2.181432e-01
6.845471e-01
9.602937e-01
9.685832e-01
7.071068e-01
2.486899e-01
-2.789911e-01
-7.289686e-01
-9.759168e-01
-9.510565e-01
-6.613119e-01
-1.873813e-01
3.387379e-01
7.705132e-01
9.876883e-01
9.297765e-01
6.129071e-01
1.253332e-01
-3.971479e-01
-8.090170e-01
-9.955620e-01
-9.048271e-01
-5.620834e-01
-6.279052e-02
4.539905e-01
8.443279e-01
9.995066e-01
8.763067e-01
5.090414e-01
-4.288121e-15
-5.090414e-01
-8.763067e-01
-9.995066e-01
-8.443279e-01
-4.539905e-01
6.279052e-02
5.620834e-01
9.048271e-01
9.955620e-01
8.090170e-01
3.971479e-01
-1.253332e-01
-6.129071e-01
-9.297765e-01
-9.876883e-01
-7.705132e-01
-3.387379e-01
1.873813e-01
6.613119e-01
9.510565e-01
9.759168e-01
7.289686e-01
2.789911e-01
-2.486899e-01
-7.071068e-01
-9.685832e-01
-9.602937e-01
-6.845471e-01
-2.181432e-01
3.090170e-01
7.501111e-01
9.822873e-01
9.408808e-01
6.374240e-01
1.564345e-01
-3.681246e-01
-7.901550e-01
-9.921147e-01
-9.177546e-01
-5.877853e-01
-9.410831e-02
4.257793e-01
8.270806e-01
9.980267e-01
8.910065e-01
5.358268e-01
3.141076e-02
-4.817537e-01
-8.607420e-01
-1
-8.607420e-01
-4.817537e-01
3.141076e-02
5.358268e-01
8.910065e-01
9.980267e-01
8.270806e-01
4.257793e-01
-9.410831e-02
-5.877853e-01
-9.177546e-01
-9.921147e-01
-7.901550e-01
-3.681246e-01
1.564345e-01
6.374240e-01
9.408808e-01
9.822873e-01
7.501111e-01
3.090170e-01
-2.181432e-01
-6.845471e-01
-9.602937e-01
-9.685832e-01
-7.071068e-01
-2.486899e-01
2.789911e-01
Assuming the file is not exceedingly large, the simplest way would probably be read the entire file & index the output to your desired lines.
line = [6 11 111 127];
fileID = fopen('file.txt');
C = textscan(fileID,'%s','delimiter','\n');
fclose(fileID);
disp(C{1}(line))
After importing this data file from Matlab with scipy.io.loadmat, things appeared to work fine until we tried to calculate the conditioning number of one of the matrixes within.
Here's the minimum amount of code that reproduces for us:
import scipy
import numpy
stuff = scipy.io.loadmat("dati-esercizio1.mat")
numpy.linalg.cond(stuff["A"])
Here's the extended stacktrace courtesy of iPython:
In [3]: numpy.linalg.cond(A)
---------------------------------------------------------------------------
LapackError Traceback (most recent call last)
/snip/<ipython-input-3-15d9ef00a605> in <module>()
----> 1 numpy.linalg.cond(A)
/snip/python2.7/site-packages/numpy/linalg/linalg.py in cond(x, p)
1409 x = asarray(x) # in case we have a matrix
1410 if p is None:
-> 1411 s = svd(x,compute_uv=False)
1412 return s[0]/s[-1]
1413 else:
/snip/python2.7/site-packages/numpy/linalg/linalg.py in svd(a, full_matrices, compute_uv)
1313 work = zeros((lwork,), t)
1314 results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt,
-> 1315 work, -1, iwork, 0)
1316 lwork = int(work[0])
1317 work = zeros((lwork,), t)
LapackError: Parameter a has non-native byte order in lapack_lite.dgesdd
All obvious ideas (like flattening and reshaping the matrix or recreating the matrix from scratch reassigning it element by element) failed. How can I want to massage the data, then, in order to make it more agreeable with numpy?
It's a bug, fixed some time ago: https://github.com/numpy/numpy/pull/235
Workaround:
np.linalg.cond(stuff['A'].newbyteorder('='))
This works for me:
In [33]: stuff = loadmat('dati-esercizio1.mat')
In [34]: a = stuff['A']
In [35]: try: np.linalg.cond(a)
....: except: print "Fail!"
Fail!
In [36]: b = np.array(a, dtype='>d')
In [37]: np.linalg.cond(b)
Out[37]: 62493201976.673141
In [38]: np.all(a == b) # Verify they hold the same data.
Out[38]: True
Apparently it's something wrong with the byte order (endianness?) of each number in the resulting ndarray and not just with the ndarray object itself.
Something like this but more elegant should do the trick:
n, m = A.shape()
B = numpy.empty_like(A)
for i in xrange(n):
for j in xrange(m):
B[i,j] = float(A[i,j])
del A
B = A
print numpy.linalg.cond(A) # 62493210091.354507
(For some reason an in-place replacement still gives that error - so there's something wrong with the byte order of the whole object, too.)