Last rune of golang unicode/norm iterator not being read - unicode

I'm using the golang.org/x/text/unicode/norm package to iterate over runes in a []byte. I've chosen this approach as I need to inspect each rune and maintain information about the sequence of runes. The last call to iter.Next() does not read the last rune. It gives 0 bytes read on the last rune.
Here is the code:
package main
import (
"fmt"
"unicode/utf8"
"golang.org/x/text/unicode/norm"
)
func main() {
var (
n int
r rune
it norm.Iter
out []byte
)
in := []byte(`test`)
fmt.Printf("%s\n", in)
fmt.Println(in)
it.Init(norm.NFD, in)
for !it.Done() {
ruf := it.Next()
r, n = utf8.DecodeRune(ruf)
fmt.Printf("bytes read: %d. val: %q\n", n, r)
buf := make([]byte, utf8.RuneLen(r))
utf8.EncodeRune(buf, r)
out = norm.NFC.Append(out, buf...)
}
fmt.Printf("%s\n", out)
fmt.Println(out)
}
This produces the following output:
test
[116 101 115 116]
bytes read: 1. val: 't'
bytes read: 1. val: 'e'
bytes read: 1. val: 's'
bytes read: 0. val: '�'
tes�
[116 101 115 239 191 189]

It is possible this is a bug in golang.org/x/text/unicode/norm and its Init() function.
In the package's test and example that I see all use InitString. So as a workaround, if you change:
it.Init(norm.NFD, in)
to:
it.InitString(norm.NFD, `test`)
things will work as expected.
I would suggest opening up a bug report, but beware that since this is in the "/x" directory that package is considered experimental by go developers.
(BTW, I used my the go debugger to help me track down what's going on, but I should say its use was far the kind of debugger I'd like to see.)

Related

Solving project Euler 25 in a functional manner using Scala

I am currently using Project Euler to learn Scala.
I am stuck on problem 25 with a java.lang.OutOfMemoryError exception.
Here is the question:
What is the index of the first term in the Fibonacci sequence to contain 1000 digits?
What I have come up with:
def fibonacciIndex(numOfDigits: Int): Option[Int] = {
lazy val fibs: LazyList[Int] = 0 #:: fibs.scanLeft(1)(_ + _)
fibs.find(_.toString.length == numOfDigits)
}
I am trying to do this in a purely functional way.
Has anyone got some suggestions to improve the memory usage?
Thanks in advance!
EDIT: I was overflowing the Int type. Solved like so:
def fibonacciIndex(numOfDigits: Int): Int = {
lazy val bigFibs: LazyList[BigInt] = BigInt(0) #:: bigFibs.scanLeft(BigInt(1))(_ + _)
bigFibs.indexWhere(_.toString.length == numOfDigits)
}
Let's run your program in parts.
lazy val fibs: LazyList[Int] = 0 #:: fibs.scanLeft(1)(_ + _)
fibs.take(100).foreach(println)
It prints:
0
1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
6765
10946
17711
28657
46368
75025
121393
196418
317811
514229
832040
1346269
2178309
3524578
5702887
9227465
14930352
24157817
39088169
63245986
102334155
165580141
267914296
433494437
701408733
1134903170
1836311903
-1323752223
512559680
-811192543
-298632863
-1109825406
-1408458269
1776683621
368225352
2144908973
-1781832971
363076002
-1418756969
-1055680967
1820529360
764848393
-1709589543
-944741150
1640636603
695895453
-1958435240
-1262539787
1073992269
-188547518
885444751
696897233
1582341984
-2015728079
-433386095
1845853122
1412467027
-1036647147
375819880
-660827267
-285007387
-945834654
-1230842041
2118290601
887448560
-1289228135
-401779575
-1691007710
-2092787285
511172301
-1581614984
-1070442683
1642909629
572466946
-2079590721
-1507123775
708252800
-798870975
-90618175
-889489150
As you can see integer overflows, and so it never outputs more than certain number of digits (11 I think including -). So it keep in computing next value.
Additionally LazyList remembers each value that it computed. It is lazy but memoizes. So each new number that is computed here
fibs.find(_.toString.length == numOfDigits)
is also remembered. It continues until you run out of memory.
To fix that you should replace LazyList with e.g. Iterator or Iterable and store numbers as BigInts.

How is risc-v neg instruction imeplemented?

How is the neg pseudo instruction implemented with only one sub?
I don't understand, as neg is R[rd] = -R[rs1]. But if I have sub, it is R[rs1] - something.
The "something" in this case is the zero register. but you're not subtracting that from the register, you're subtracting the register from that.
The:
neg rd, rs
pseudo-instruction is meant to put the negation of rs into rd. The
sub rd, zero, rs
instruction subtracts rs from zero, placing the result into rd.
rd := -rs ; example: -(42) -> -42
rd := 0 - rs ; 0 - 42 -> -42
Since -x is the same as 0 - x, they are equivalent.
If you want a more comprehensive list of pseudo instructions and what they map to, here an image which details some, including the specific one you asked about:

TCP connection crashes after reading only a few strings

I'm working with the ESP8266 wifi module programming it in LUA using NodeMCU. I was able to connect from an iPhone app to the TCP server created by the wifi module to send 3 floats. I parse the huge string into 3 strings and send them using uart to and Arduino. It works just fine but after about 10 inputs it crashes/freezes. I need the constant flow of data to keep coming but consistently and I'm unable to do it reliably.
I've printed after each line to keep track of what is actually happening and even after that i'm still not sure what is going on.
Code
`
print("11\n")
wifi.setmode(wifi.STATION)
print("22\n")
wifi.sta.config("WDTS03","Walker14!")
print("33\n")
elWiFi =(wifi.sta.getip())
if elWiFi ~= nil then
print(wifi.sta.getip())
end
print("44\n")
if srv~=nil then
print("444\n")
srv:close()
print("555 \n")
end
print("Create server \n")
srv=net.createServer(net.TCP)
print("1\n")
if srv ~= nil then
print("srv !=nil \n")
srv:listen(6969,function(conn)
print("listening \n")
if conn ~= nil then
print("incoming\n")
conn:on("receive",function(conn,numbers)
print("2\n")
print(#numbers)
print("chekcing for nil \n")
if numbers ~= nil then
print("3\n")
p = string.find(numbers, "x=")
print("4\n")
q = string.find(numbers, "&y")
print("5\n")
if p ~= nill then
print("6\n")
if q ~= nil then
print("7\n")
x = (string.sub(numbers,p+2, q-1))
print("x=" .. x)
end
end --p ~= nill
print("8\n")
p = string.find(numbers, "y=")
print("9\n")
q = string.find(numbers, "&z")
print("10\n")
if p ~= nill then
print("11\n")
if q ~= nil then
print("12\n")
y = (string.sub(numbers,p+2, q-1))
print("y=" .. y)
end
end --p ~= nill
print("13\n")
p = string.find(numbers, "z=")
print("14\n")
q = string.find(numbers, " H")
print("15\n")
if p ~= nill then
print("16\n")
if q ~= nil then
print("17\n")
z = (string.sub(numbers,p+2, q-1))
print("z=" .. z)
end
end-- p ~= nill
print("18\n")
end --numbers ~= nil
print("54\n")
--conn:send("test\n")
end)
print("55 \n")
end
print("66 \n")
end)
print("77\n")
end
print("666\n")`
and I get the following output
11
22
33
44
Create server
1
srv !=nil
77
666
> listening
incoming
55
66
listening
incoming
55
66
listening
incoming
55
66
listening
incoming
55
66
2
338
chekcing for nil
3
4
5
6
7
x=0.1722259521484375
8
9
10
11
12
y=-0.7733306884765625
13
14
15
16
17
z=-0.5716094970703125
18
54
2
337
chekcing for nil
3
4
5
6
7
.
.--repeats a few times
.
y=-0.005340576171875
13
14
15
16
17
z=-0.9838409423828125
18
54
PANIC: unprotected error in call to Lua API (attempt to call a nil value)
�l� �=+���T2n���
NodeMCU 0.9.6 build 20150704 powered by Lua 5.1.4
11
22`
It fails on the "conn:on" argument for "srv:listen" after a few times
Thanks for the help and sorry if formatting is messed up. First time
Too long for a comment, sorry.
PANIC: unprotected error in call to Lua API (attempt to call a nil value)
Hard to tell which value was nil/null. Create a Minimal, Complete, and Verifiable Example (MCVE) to reduce the amount of code one needs to analyze.
The following snippet will eventually lead to out-of-memory due to closed upvalues. Each of your callback functions should use its own copy of the passed socket instance rather referencing the one of a wrapping callback function. The third line should NOT re-use the conn variable but define one with a new name.
See https://stackoverflow.com/a/37379426/131929 for details.
srv:listen(6969,function(conn)
if conn ~= nil then
conn:on("receive",function(conn,numbers)
NodeMCU 0.9.6 build 20150704
Don't use those old 0.9.x binaries, they're outdated and contain lots of bugs. http://nodemcu.readthedocs.io/en/latest/en/#getting-started helps you get started with a new firmware.

"LapackError: Parameter a has non-native byte order in lapack_lite.dgesdd" when importing from Matlab files

After importing this data file from Matlab with scipy.io.loadmat, things appeared to work fine until we tried to calculate the conditioning number of one of the matrixes within.
Here's the minimum amount of code that reproduces for us:
import scipy
import numpy
stuff = scipy.io.loadmat("dati-esercizio1.mat")
numpy.linalg.cond(stuff["A"])
Here's the extended stacktrace courtesy of iPython:
In [3]: numpy.linalg.cond(A)
---------------------------------------------------------------------------
LapackError Traceback (most recent call last)
/snip/<ipython-input-3-15d9ef00a605> in <module>()
----> 1 numpy.linalg.cond(A)
/snip/python2.7/site-packages/numpy/linalg/linalg.py in cond(x, p)
1409 x = asarray(x) # in case we have a matrix
1410 if p is None:
-> 1411 s = svd(x,compute_uv=False)
1412 return s[0]/s[-1]
1413 else:
/snip/python2.7/site-packages/numpy/linalg/linalg.py in svd(a, full_matrices, compute_uv)
1313 work = zeros((lwork,), t)
1314 results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt,
-> 1315 work, -1, iwork, 0)
1316 lwork = int(work[0])
1317 work = zeros((lwork,), t)
LapackError: Parameter a has non-native byte order in lapack_lite.dgesdd
All obvious ideas (like flattening and reshaping the matrix or recreating the matrix from scratch reassigning it element by element) failed. How can I want to massage the data, then, in order to make it more agreeable with numpy?
It's a bug, fixed some time ago: https://github.com/numpy/numpy/pull/235
Workaround:
np.linalg.cond(stuff['A'].newbyteorder('='))
This works for me:
In [33]: stuff = loadmat('dati-esercizio1.mat')
In [34]: a = stuff['A']
In [35]: try: np.linalg.cond(a)
....: except: print "Fail!"
Fail!
In [36]: b = np.array(a, dtype='>d')
In [37]: np.linalg.cond(b)
Out[37]: 62493201976.673141
In [38]: np.all(a == b) # Verify they hold the same data.
Out[38]: True
Apparently it's something wrong with the byte order (endianness?) of each number in the resulting ndarray and not just with the ndarray object itself.
Something like this but more elegant should do the trick:
n, m = A.shape()
B = numpy.empty_like(A)
for i in xrange(n):
for j in xrange(m):
B[i,j] = float(A[i,j])
del A
B = A
print numpy.linalg.cond(A) # 62493210091.354507
(For some reason an in-place replacement still gives that error - so there's something wrong with the byte order of the whole object, too.)

Need help identifying and computing a number representation

I need help identifying the following number format.
For example, the following number format in MIB:
0x94 0x78 = 2680
0x94 0x78 in binary: [1001 0100] [0111 1000]
It seems that if the MSB is 1, it means another character follows it. And if it is 0, it is the end of the number.
So the value 2680 is [001 0100] [111 1000], formatted properly is [0000 1010] [0111 1000]
What is this number format called and what's a good way for computing this besides bit manipulation and shifting to a larger unsigned integer?
I have seen this called either 7bhm (7-bit has-more) or VLQ (variable length quantity); see http://en.wikipedia.org/wiki/Variable-length_quantity
This is stored big-endian (most significant byte first), as opposed to the C# BinaryReader.Read7BitEncodedInt method described at Encoding an integer in 7-bit format of C# BinaryReader.ReadString
I am not aware of any method of decoding other than bit manipulation.
Sample PHP code can be found at
http://php.net/manual/en/function.intval.php#62613
or in Python I would do something like
def encode_7bhm(i):
o = [ chr(i & 0x7f) ]
i /= 128
while i > 0:
o.insert(0, chr(0x80 | (i & 0x7f)))
i /= 128
return ''.join(o)
def decode_7bhm(s):
o = 0
for i in range(len(s)):
v = ord(s[i])
o = 128*o + (v & 0x7f)
if v & 0x80 == 0:
# found end of encoded value
break
else:
# out of string, and end not found - error!
raise TypeError
return o