Below is an example of window join:
.up.t:([]sym:3#`abc;time:10:01:01 10:01:04 10:01:08;price:100 101 105);
.up.a:101 103 103 104 104 107 108 107 108;
.up.b:98 99 102 103 103 104 106 106 107;
.up.q:([]sym:`abc; time:10:01:01+til 9; ask:.up.a; bid:.up.b);
.up.w:-2 1+\:.up.t.time;
wj[.up.w;`sym`time;.up.t;(.up.q;(min;`ask);(max;`bid))]
Output:
sym time price ask bid
abc 10:01:01 100 101 99
abc 10:01:04 101 103 103
abc 10:01:08 105 107 107
I'm unable to figure out, how .up.w is impacting/affecting the result of the window join?
It's easier to see if you return the lists un-aggregated and then try changing the windows
q)wj[-2 1+\:.up.t.time;`sym`time;.up.t;(.up.q;(::;`ask);(::;`bid))]
sym time price ask bid
--------------------------------------------------
abc 10:01:01 100 101 103 98 99
abc 10:01:04 101 103 103 104 104 99 102 103 103
abc 10:01:08 105 107 108 107 108 104 106 106 107
q)wj[-2 2+\:.up.t.time;`sym`time;.up.t;(.up.q;(::;`ask);(::;`bid))]
sym time price ask bid
---------------------------------------------------------
abc 10:01:01 100 101 103 103 98 99 102
abc 10:01:04 101 103 103 104 104 107 99 102 103 103 104
abc 10:01:08 105 107 108 107 108 104 106 106 107
The window determines how many (and which) quotes are included in the aggregation based on the plus/minus range around the trade times
Related
I am new to MongoDB. I am trying to save a query result to a json file however keep getting this error:
query '[102 105 110 100 40 123 32 36 97 110 100 58 91 32 123 100 105 115 112 108 97 121 78 97 109 101 58 47 94 65 47 105 125 44 32 123 100 105 115 112 108 97 121 78 97 109 101 58 47 101 115 36 47 125 32 44 123 102 114 105 101 110 100 115 67 111 117 110 116 58 123 36 108 116 58 50 53 125 125 93 32 125 44 123 100 105 115 112 108 97 121 78 97 109 101 58 49 44 102 111 108 108 111 119 101 114 115 67 111 117 110 116 58 49 44 102 114 105 101 110 100 115 67 111 117 110 116 58 49 125 41 46 115 111 114 116 40 123 100 105 115 112 108 97 121 78 97 109 101 58 45 49 125 41]' is not valid JSON: invalid character 'i' in literal false (expecting 'a') try 'mongoexport --help' for more information
The code I am using is
mongoexport --db cw2 --collection cl --query="find({ $and:[ {displayName:/^A/i}, {displayName:/es$/} ,{friendsCount:{$lt:25}}] },{"displayName":1,"followersCount":1,"friendsCount":1}).sort({"displayName":-1})" --out=data/cw2q2.json
Can anyone see what the issue is? I am not sure i understand what the actual message is saying.
My interest is in exploring how K8s resource objects are created from specs submitted by client-API; and how they are communicated across kube master services and pods.
I have started exploring the K8s source code and reached certain depth.
For this exploration I am using kubectl create -f <path to yaml> and tracking how this command is handled in source code.
here is brief summary
https://github.com/kubernetes/kubernetes/blob/v1.14.0/pkg/kubectl/cmd/create/create.go#L96 I started with this function and followed on function trail (thanks to this blog); and gradually reached to
https://github.com/kubernetes/kubernetes/blob/v1.14.0/staging/src/k8s.io/client-go/rest/request.go#L708 - http request is formed here
https://github.com/kubernetes/kubernetes/blob/v1.14.0/staging/src/k8s.io/client-go/rest/request.go#L395 - the body of the request is formed here.
https://github.com/kubernetes/kubernetes/blob/v1.14.0/staging/src/k8s.io/client-go/rest/request.go#L732 - this functions send http request and gets response back for processing, here I have addedklog.Infof("custom log sizeofdata:%T$$$$,%d$$$$,%s$$$$,%#v",*req,unsafe.Sizeof(*req),*req,*req) to find size of the request being sent.
My hypothesis is; this request size in bytes should change as the change in specs submitted by the user (yaml file passed with kubectl create -f)
Now I tried to feed two different resource spec(with kubectl create -f ) and see how the HTTP request and request body changes (its size).
Yaml 1 - nginx sample
https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/application/deployment.yaml
Yaml 2 - centos sample
apiVersion: v1
kind: Pod
metadata:
name: cent-os-test
spec:
containers:
- name: centos-test-container
image: centos
command: ["sleep"]
args: ["100"]
restartPolicy: Never
Surprisingly both times output of the logging line above was same Byte size (248 Bytes)
for log requested by klog.Infof("custom log sizeofdata:%T$$$$,%d$$$$,%s$$$$,%#v",*req,unsafe.Sizeof(*req),*req,*req)
I have used $$$$ as field separator while printing.
Following are log entries in case of both of yamls
for Yaml 2 centos ---
I0405 22:22:37.285666 360723 request.go:745] custom_log size of data : http.Request $$$$, 248 $$$$, {POST https://10.xxx.xxx.xxx:6443/api/v1/namespaces/default/pods HTTP/1.1 %!s(int=1) %!s(int=1) map[Accept:[application/json] Content-Type:[application/json]] {%!s(*bytes.Reader=&{[123 34 97 112 105 86 101 114 115 105 111 110 34 58 34 118 49 34 44 34 107 105 110 100 34 58 34 80 111 100 34 44 34 109 101 116 97 100 97 116 97 34 58 123 34 110 97 109 101 34 58 34 99 101 110 116 45 111 115 45 116 101 115 116 34 44 34 110 97 109 101 115 112 97 99 101 34 58 34 100 101 102 97 117 108 116 34 125 44 34 115 112 101 99 34 58 123 34 99 111 110 116 97 105 110 101 114 115 34 58 91 123 34 97 114 103 115 34 58 91 34 49 48 48 34 93 44 34 99 111 109 109 97 110 100 34 58 91 34 115 108 101 101 112 34 93 44 34 105 109 97 103 101 34 58 34 99 101 110 116 111 115 34 44 34 110 97 109 101 34 58 34 99 101 110 116 111 115 45 116 101 115 116 45 99 111 110 116 97 105 110 101 114 34 125 93 44 34 114 101 115 116 97 114 116 80 111 108 105 99 121 34 58 34 78 101 118 101 114 34 125 125 10] 0 -1})} %!s(func() (io.ReadCloser, error)=0x8087b0) %!s(int64=223) [] %!s(bool=false) 10.xxx.xxx.xxx:6443 map[] map[] %!s(*multipart.Form=<nil>) map[] %!s(*tls.ConnectionState=<nil>) %!s(<-chan struct {}=<nil>) %!s(*http.Response=<nil>) <nil>} $$$$, http.Request{Method:"POST", URL:(*url.URL)(0xc001dc6880), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json"}, "Content-Type":[]string{"application/json"}}, Body:ioutil.nopCloser{Reader:(*bytes.Reader)(0xc000886f60)}, GetBody:(func() (io.ReadCloser, error))(0x8087b0), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"10.xxx.xxx.xxx:6443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:context.Context(nil)}
for Yaml 1 nginx ---
I0406 00:07:19.873854 410863 request.go:745] custom_log size of data : http.Request $$$$, 248 $$$$, {POST https://10.xxx.xxx.xxx:6443/apis/apps/v1/namespaces/default/deployments HTTP/1.1 %!s(int=1) %!s(int=1) map[Accept:[application/json] Content-Type:[application/json]] {%!s(*bytes.Reader=&{[123 34 97 112 105 86 101 114 115 105 111 110 34 58 34 97 112 112 115 47 118 49 34 44 34 107 105 110 100 34 58 34 68 101 112 108 111 121 109 101 110 116 34 44 34 109 101 116 97 100 97 116 97 34 58 123 34 110 97 109 101 34 58 34 110 103 105 110 120 45 100 101 112 108 111 121 109 101 110 116 34 44 34 110 97 109 101 115 112 97 99 101 34 58 34 100 101 102 97 117 108 116 34 125 44 34 115 112 101 99 34 58 123 34 114 101 112 108 105 99 97 115 34 58 50 44 34 115 101 108 101 99 116 111 114 34 58 123 34 109 97 116 99 104 76 97 98 101 108 115 34 58 123 34 97 112 112 34 58 34 110 103 105 110 120 34 125 125 44 34 116 101 109 112 108 97 116 101 34 58 123 34 109 101 116 97 100 97 116 97 34 58 123 34 108 97 98 101 108 115 34 58 123 34 97 112 112 34 58 34 110 103 105 110 120 34 125 125 44 34 115 112 101 99 34 58 123 34 99 111 110 116 97 105 110 101 114 115 34 58 91 123 34 105 109 97 103 101 34 58 34 110 103 105 110 120 58 49 46 55 46 57 34 44 34 110 97 109 101 34 58 34 110 103 105 110 120 34 44 34 112 111 114 116 115 34 58 91 123 34 99 111 110 116 97 105 110 101 114 80 111 114 116 34 58 56 48 125 93 125 93 125 125 125 125 10] 0 -1})} %!s(func() (io.ReadCloser, error)=0x8087b0) %!s(int64=316) [] %!s(bool=false) 10.xxx.xxx.xxx:6443 map[] map[] %!s(*multipart.Form=<nil>) map[] %!s(*tls.ConnectionState=<nil>) %!s(<-chan struct {}=<nil>) %!s(*http.Response=<nil>) <nil>} $$$$, http.Request{Method:"POST", URL:(*url.URL)(0xc000a96380), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json"}, "Content-Type":[]string{"application/json"}}, Body:ioutil.nopCloser{Reader:(*bytes.Reader)(0xc0008abb90)}, GetBody:(func() (io.ReadCloser, error))(0x8087b0), ContentLength:316, TransferEncoding:[]string(nil), Close:false, Host:"10.xxx.xxx.xxx:6443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:context.Context(nil)}
Why both of times the HTTP request size is 248 bytes? Shouldn’t it increase in case of the nginx yaml since it passes more lines of spec information?
Would you refer to the length of the *req variable rather then it's size
klog.Infof("custom log sizeofdata:%T$$$$,%d$$$$,%s$$$$,%#v",*req,len(*req),*req,*req)
Similar to the below snippet illustrating that size of a variable of type string is always 8 but the length of a string that a variable references can vary:
package main
import "fmt"
import "unsafe"
func main() {
s1 := "foo"
s2 := "foobar"
fmt.Printf("s1 size: %T, %d\n", s1, unsafe.Sizeof(s1))
fmt.Printf("s2 size: %T, %d\n", s2, unsafe.Sizeof(s2))
fmt.Printf("s1 len: %T, %d\n", s1, len(s1))
fmt.Printf("s2 len: %T, %d\n", s2, len(s2))
}
Result:
s1 size: string, 8
s2 size: string, 8
s1 len: string, 3
s2 len: string, 6
I have a matrix in Matlab as below:
a =
1 169 158 147
1 104 165 66
728 105 276 43
950 113 971 40
1 107 810 23
227 133 48 15
618 131 107 20
class(a)
ans =
'double'
I want to add a column to this matrix. When I try this command, i get a wrong result:
A={1;2;3;4;5;6;7}
vertcat(A,a)
Answer is like below:
What I wanted was:
1 1 169 158 147
1 1 104 165 66
1 728 105 276 43
1 950 113 971 40
1 1 107 810 23
1 227 133 48 15
1 618 131 107 20
What mistake I am making and how to I fix it?
Thanks
P.S: I am new to Matlab
I have 96 x 96 images, I want to convert each image to pixels and save it to a text file. This is an example of code :
frame = imread('c.jpg');
allpixels = reshape(frame, 96*96, 3)
The Output of the command window is :
a
allpixels =
211 194 176
200 183 163
186 169 143
170 150 123
154 133 104
149 128 99
156 135 106
157 136 109
155 135 110
148 126 102
147 123 97
143 118 88
.....
I want to save only the content of the variable allpixels to a text file. It means the text file should be exactly like this :
211 194 176
200 183 163
186 169 143
170 150 123
154 133 104
149 128 99
156 135 106
157 136 109
155 135 110
148 126 102
147 123 97
143 118 88
I have used :
diary('out.txt');
diary on
But out.txt will contain all the command window output, like this :
a
allpixels =
9216×3 uint8 matrix
211 194 176
200 183 163
186 169 143
170 150 123
154 133 104
149 128 99
156 135 106
157 136 109
155 135 110
148 126 102
147 123 97
143 118 88
How can I do that ?
diary is for activity log of your MATLAB session. To store the specific matrix you can use the following code:
dlmwrite('out.txt', a, 'delimiter', ' ');
See the details here.
I have sample code emulating my actual code. Where I have cell arrays outside the parfor loop. I have to perform computations on strings and numerical outputs will be stored in arrays which I can write to a csv file after each parfor loop. So I made dummy code. But I couldn't get it to execute. The error message is: "subscription mismatch at line 6".
ftemp=fopen('temp.csv','w');
march=cell(1,20);tc=0;
march={'ab' 'cd' 'ef' 'gh' 'ij' 'kl' 'mn' 'op' 'qr' 'st' 'uv' 'AB' 'CD' 'EF' 'GH' 'IJ' 'KL' 'MN' 'OP' 'QR'};
for i=1:10
matlabpool open 4;
parfor j=1:1:20
a(j,1)=randi(200,1,1);
b(j,2)=j+tc;
c(j,3)=march{1,j};
d(j,4)=(randi(200,1,1)/200);
end
fprintf(ftemp,'%d\t%d\t%s\t%f',a,b,c,d);
matlabpool close
clear a b c d;
tc=tc+20;
end
fclose(ftemp);
quit
The cause of the Error is that you are trying to assign a cell into an array in line 9.
I made some changes into your code, they are described in comments.
ftemp=fopen('temp.csv','w');
march=cell(1,20);tc=0;
march={'ab' 'cd' 'ef' 'gh' 'ij' 'kl' 'mn' 'op' 'qr' 'st' 'uv' 'AB' 'CD' 'EF' 'GH' 'IJ' 'KL' 'MN' 'OP' 'QR'};
for i=1:10
parpool local; % here in my version of Matlab 2015 there are no more "matlabpool" if it doesen't work change it back
parfor j=1:20
%i changed variable a(j,1) into a(j) and b(j,2) into b(j): may
%contain empty arrays , idem: for c and d
a(j)=randi(200,1,1);
b(j)=j+tc;
c{j}=march{1,j}; % changed c(j)= march{1,j}; : cause of error
d(j)=randi(200,1,1)/200;
end
fprintf(ftemp,'%d\t%d\t%c\t%d',a,b,char(c),d); % char(c) in order to convert cell array to array of strings
delete(gcp) % there isn't such thing "matlabpool close" , the right expression is "delete(gcp)"
clear a b c d;
tc=tc+20;
end
fclose(ftemp);
%quit : i remove because it closes matlab , please put it back if u really
%want to close matlab after operation
Note: if parpool doesent work for your version of matlab , please change it back to the expression you were using matlabpool open 4;
Well for the unexpected outputs and number of rows in the output , it's because the wrong use of fprintf . We need to print element by element , which means the fprintf has to be inside the parfor loop, so your code should look like this:
ftemp=fopen('temp.csv','w');
march=cell(1,20);tc=0;
march={'ab';'cd';'ef';'gh';'ij';'klm';'mn';'op';'qr';'st';'uv';'ABls';'CD';'E3F';'GH';'IJ';'dynaKL';'MN';'OP';'QR'};
matlabpool open 4;
for k=1:10
%parpool local; % here in my version of Matlab 2015 there are no more "matlabpool" if it doesen't work change it back
parfor j=1:20
%i changed variable a(j,1) into a(j) and b(j,2) into b(j): may
%contain empty arrays , idem: for c and d
a(j)=randi(200,1,1);
b(j)=j+tc;% indexing purposes to identify the order when parallel processing going on
if length(march{j})>2 % this kind of conditions and computations are there in my actual code
c{j}='skip';
else
c{j}=march{j};
end
%d(j)=randi(200,1,1)/200;
fprintf(ftemp,'%d\t%d\t%s\t\n',a(j),b(j),c{j}); % char(c) in order to convert cell array to array of strings
end
clear a b c;
tc=tc+20;
end
fclose(ftemp);
matlabpool close;
the output is:
9 1 ab
171 2 cd
7 3 ef
98 4 gh
102 5 ij
20 6 skip
53 7 mn
174 8 op
36 9 qr
30 10 st
130 11 uv
127 12 skip
133 13 CD
65 14 skip
118 15 GH
130 16 IJ
139 17 skip
168 18 MN
57 19 OP
25 20 QR
22 21 ab
39 22 cd
83 23 ef
22 24 gh
159 25 ij
40 26 skip
164 27 mn
78 28 op
194 29 qr
88 30 st
125 31 uv
1 32 skip
2 33 CD
112 34 skip
161 35 GH
170 36 IJ
55 37 skip
59 38 MN
18 39 OP
134 40 QR
80 41 ab
118 42 cd
108 43 ef
174 44 gh
97 45 ij
157 46 skip
85 47 mn
98 48 op
40 49 qr
11 50 st
171 51 uv
139 52 skip
90 53 CD
70 54 skip
173 55 GH
150 56 IJ
186 57 skip
155 58 MN
136 59 OP
96 60 QR
158 61 ab
118 62 cd
124 63 ef
127 64 gh
26 65 ij
124 66 skip
91 67 mn
186 68 op
63 69 qr
137 70 st
170 71 uv
98 72 skip
132 73 CD
80 74 skip
160 75 GH
20 76 IJ
156 77 skip
142 78 MN
110 79 OP
51 80 QR
18 81 ab
29 82 cd
40 83 ef
49 84 gh
102 85 ij
113 86 skip
96 87 mn
44 88 op
166 89 qr
90 90 st
21 91 uv
60 92 skip
44 93 CD
166 94 skip
103 95 GH
123 96 IJ
77 97 skip
163 98 MN
138 99 OP
111 100 QR
94 101 ab
133 102 cd
158 103 ef
13 104 gh
26 105 ij
117 106 skip
90 107 mn
58 108 op
156 109 qr
79 110 st
196 111 uv
168 112 skip
192 113 CD
160 114 skip
56 115 GH
129 116 IJ
191 117 skip
157 118 MN
170 119 OP
30 120 QR
137 121 ab
128 122 cd
12 123 ef
38 124 gh
38 125 ij
122 126 skip
97 127 mn
129 128 op
142 129 qr
154 130 st
99 131 uv
85 132 skip
129 133 CD
111 134 skip
108 135 GH
78 136 IJ
102 137 skip
48 138 MN
100 139 OP
89 140 QR
114 141 ab
12 142 cd
184 143 ef
145 144 gh
5 145 ij
48 146 skip
92 147 mn
64 148 op
87 149 qr
32 150 st
136 151 uv
103 152 skip
90 153 CD
73 154 skip
28 155 GH
191 156 IJ
63 157 skip
120 158 MN
104 159 OP
178 160 QR
148 161 ab
36 162 cd
129 163 ef
11 164 gh
172 165 ij
186 166 skip
145 167 mn
142 168 op
150 169 qr
185 170 st
14 171 uv
141 172 skip
22 173 CD
163 174 skip
48 175 GH
164 176 IJ
117 177 skip
25 178 MN
110 179 OP
111 180 QR
175 181 ab
60 182 cd
195 183 ef
44 184 gh
163 185 ij
4 186 skip
103 187 mn
95 188 op
127 189 qr
10 190 st
11 191 uv
182 192 skip
162 193 CD
179 194 skip
76 195 GH
104 196 IJ
153 197 skip
103 198 MN
4 199 OP
154 200 QR
Also the output has 200 rows.
ftemp=fopen('temp.csv','w');
march=cell(1,20);tc=0;
march={'ab';'cd';'ef';'gh';'ij';'klm';'mn';'op';'qr';'st';'uv';'ABls';'CD';'E3F';'GH';'IJ';'dynaKL';'MN';'OP';'QR'};
matlabpool open 4;
for k=1:10
%parpool local; % here in my version of Matlab 2015 there are no more "matlabpool" if it doesen't work change it back
parfor j=1:20
%i changed variable a(j,1) into a(j) and b(j,2) into b(j): may
%contain empty arrays , idem: for c and d
a(j)=randi(200,1,1);
b(j)=j+tc;% indexing purposes to identify the order when parallel processing going on
if length(march{j})>2 % this kind of conditions and computations are there in my actual code
c{j}='skip';
else
c{j}=march{j};
end
%d(j)=randi(200,1,1)/200;
end
fprintf(ftemp,'%d\t%d\t%s\n',a,b,c{:}); % char(c) in order to convert cell array to array of strings
clear a b c;
tc=tc+20;
end
fclose(ftemp);
matlabpool close;
but the output of the program i coudlnt get it
130 127 Abf5ȱ3³]À#¾
1 2
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 168 57 ®$ «E�,vp|}¤
21 22 !"#$%&'(
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 88 125 S([g 0'0Â"
41 42 +,-./0123456789:;<
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 161 170 7Na�U;6´q¶nl®v¤
61 62 ?#ABCDEFGHIJKLMNOP
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 171 139 ZFºPµ~R( `b[
81 82 STUVWXYZ[\]^_`abcd
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 132 80 ||£)Á1[ªavb-
101 102 ghijklmnopqrstuvwx
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 186 63 ª`neepw»fq°r1
121 122 {|}~�
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 51 160 M(1´uOAF¦{¬©
141 142 ���
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 90 21 <,uZgh�lu¨B,¦
0
161 162 £¤¥¦§¨©ª«¬®¯°±²³´
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82 192 160 o£¿^hyµÄ¨�Ä�O
181 182 ·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈ
97 98 cd
101 102 gh
105 106 skip
109 110 op
113 114 st
117 118 skip
67 68 skip
71 72 IJ
115 107 ip
77 78 OP
81 82
i mean to say why only 124 iterations took place instead of 200. and why these arbitrary outputs are there in the 3rd column