I have an application in which I use apache Kafka and I try to do some tests.
My application consists of:
a) a service which receive http requests and sends the messages in Kafka. b) a mongoDB which receives the messages from kafka and store them.
My Kafka consists of 1 Kafka Broker, 1 topic and 1 partition. Also I use keys for messages in Kafka.
For my tests I use Apache Bench.
I created a script test.sh for my first test for run concurrency many requests. I send ever the same json object.
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test1.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test2.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test3.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test4.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test5.txt &
First test:
1)1 partition . I make 5.000.000.000 http requests and the end-to-end throughput is 53.22 req/sec.
Also, I created a script test2.sh for my second test for run concurrency many requests. I send different json objects with different key and I checked that every object stored at different partition.
ab -s 180 -p request_post/post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test1.txt &
ab -s 180 -p request_post/post2.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test2.txt &
ab -s 180 -p request_post/post3.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test3.txt &
ab -s 180 -p request_post/post4.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test4.txt &
ab -s 180 -p request_post/post5.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test5.txt &
Second test:
2) 5 partitions . I make 5.000.000.000 http requests and the end-to-end throughput is 50.13 req/sec.
I expected higher throughput in second test.
So the question is : why I don't get better throughput with more partitions?
(if you need more information ask me)
Related
I am new to Kubernetes. I have written a deployment for my python console application. this app is subscribed to NATS (message queue).
I have written command in liveness prob to check the connection_id(my app connection id) in the nats server is present. otherwise, restart the pod as the application is not running properly, in that case.
I have tried different commands. for example
livenessProbe:
exec:
command:
- sh
- -c
#- curl -s nats:8222/connz?cid=$(cat /tmp/cid) | python3 -c "import sys, json; print(json.load(sys.stdin)['connections'][0]['cid'])" | echo
#- curl -s http://nats:8222/connz?cid=$(cat /tmp/cid) | grep "$(cat /tmp/cid)"
- curl -s http://nats:8222/connz?cid=$(cat /tmp/cid) | grep "cid"
initialDelaySeconds: 10
periodSeconds: 10
and another few curl commands. then when I remove /tmp/cid file. it should fail, right? But it does not.
If I run this command
curl -s http://nats:8222/connz?cid=$(cat /tmp/cid) | grep -c "\"cid\": $(cat /tmp/cid)"
I get io.k8s.api.core.v1.ExecAction.command: got "map", expected "string" this issue.
Any suggestion?
curl -s http://localhost:8222/connz?cid=$(cat /tmp/cid) | grep -c "cid.* $(cat /tmp/cid),"
This finally worked for me.
Stuck with the issue for 2 days :(
I have successfully sealed a file to the tpm after having taking ownership of it using sudo tpm_sealdata -i inputfile -o encryptedfile -p 0 -p 1 -p 2 -p 3 -p 4 -p 5 -p 6 -z, however upon attempting to run sudo tpm_unsealdata -i encryptedfile -o newfile -z I simply recieve an error of "Unable to write to output file" however, the new file has been created and is simply empty. what could be causing tpm_unsealdata to successfully create the file, but then fail to write to it?
I am trying to run a home security camera using Rasberry Pi Model B
I want to save the stream to a file locally (USB if possible) and also stream so I can pick this up on my network
The command I have is not working for both - any suggestions?
raspivid-o security.h264 -t 0 -n -w 600 -h 400 -fps 12 | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/}' :demux=h264
Try this command:
raspivid -o - -t 0 -n -w 600 -h 400 -fps 12 | tee security.h264 | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/}' :demux=h264
The tee command writes the output to the standard output and to the specified files.
I'm wondering if there is a way to dump an HTTP stream no matter what happens on the server side.
If I use curl --retry 999 or wget --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 -t 0, the connection and download are resumed in case of network errors, but if the session is terminated by the server there is no retry. The connection is being ended and that's it. I need a perpetual retry even on FIN.
Do wget or curl have some special parameters to archive this?
Is there a tool that is not wget or curl that can archive this? A single command would be appreciated since the output is being piped.
To avoid local failure or so, you can put it into while loop, bash script
while [ 1 ]; do
wget -t 0 --timeout=15 --waitretry=1 --read-timeout=20 --retry-connrefused --continue
if [ $? = 0 ]; then break; fi; # check return value, break if successful
sleep 1s;
done;
You may try also another solution
FILENAME=$1
DOWNURL=$2
wget -O "`echo $FILENAME`" "`echo $DOWNURL`"
FILESIZE=$(stat -c%s "$FILENAME")
while [ $FILESIZE \< 1000 ]; do
sleep 3
wget -O "`echo $FILENAME`" "`echo $DOWNURL`"
FILESIZE=$(stat -c%s "$FILENAME")
done
You can play with the limit 1000. If the file is smaller then the while loop will try again.
I can train a model using Bagging from the command line like this --
java -Xmx512m -cp $CLASSPATH weka.classifiers.meta.Bagging -P 100 -S 1 -num-slots 1 -I 10 \
-split-percentage 66 \
-t $traindata \
-d $model \
-W weka.classifiers.trees.REPTree -- -M 2 -V 0.001 -N 3 -S 1 -L -1 -I 0.0 \
> $out
But I can't reuse the same model to do prediction from the command line. I guess the command should be something like --
java -Xmx512m -cp $CLASSPATH weka.classifiers.meta.Bagging \
-l $model \
-T $testdata \
-W weka.classifiers.trees.REPTree \
-p 0 \
> $wkresult
But it does not work, any idea?
EDIT: However, when I am doing with a single classifier (i.e. no bagging), it works. The commands were like this --
java -Xmx512m -cp $CLASSPATH weka.classifiers.bayes.NaiveBayesMultinomial \
-split-percentage 66 \
-t $traindata \
-d $model \
> $out
java -Xmx512m -cp $CLASSPATH weka.classifiers.bayes.NaiveBayesMultinomial \
-T $testdata \
-l $model \
-p 0 \
> $wkresult
You need to call a different class to evaluate the model. The command line should be something like
java -cp $CLASSPATH weka.classifiers.Evaluation weka.classifiers.meta.Bagging \
-T $testdata -l $model
You may need to specify some of the additional options you gave when training the classifier. Also have a look at the commandline options for the evaluation class. More information here.