Scenario: Read data from a CSV file with unknown number of records. Use the data to create Soap XML MSg and Post Method. Continue to do this until all the records have been read.
Problem: I used ReadyAPI to perform these actions and was able to achieve the intended TPS at server with whatever i have provided in VU's option. Tried with 150 vu's and observed constant load of 150 requests at the server. But when i try to do the same in JMeter, i was not able to achieve more than 70 TPS and the load isn't evenly distributed as well no matter how many threads i use. I am using a Thread Group, CSV DataSet Config, UserDefined Parameters to create unique request ID and JSR223 PreProcessor with Groovy Script as a child of HTTPRequest to remove empty xml tags.
Read some posts where it was mentioned that JMeter throughput will be stagnant based on servers response capability. But it's not in my case since i can generate 150TPS with ReadyAPI. Annual Licensing costs and Renewal costs associated with ReadyAPI is the Reason that i am looking for solution with JMeter.
Not only the server need to be able to respond fast enough, JMeter must be able to send requests fast enough as well.
Default JMeter configuration is suitable for tests development and debugging and creating some load (rather limited though), you need to properly tune your JMeter instance in order to fully utilize your machine resources so make sure to follow:
JMeter Best Practices
9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
If your single machine is not powerful enough to conduct the required load it's possible to run JMeter in distributed mode using as many load generators as needed in order to create the necessary number of virtual users/requests per second
Related
Assuming there is a Flask web server that has two routes, deployed as a CloudRun service over GKE.
#app.route('/cpu_intensive', methods=['POST'], endpoint='cpu_intensive')
def cpu_intensive():
#TODO: some actions, cpu intensive
#app.route('/batch_request', methods=['POST'], endpoint='batch_request')
def batch_request():
#TODO: invoke cpu_intensive
A "batch_request" is a batch of many same structured requests - each one is highly CPU intensive and handled by the function "cpu_intensive". No reasonable machine can handle a large batch and thus it needs to be paralleled across multiple replicas.
The deployment is configured that every instance can handle only 1 request at a time, so when multiple requests arrive CloudRun will replicate the instance.
I would like to have a service with these two endpoints, one to accept "batch_requests" and only break them down to smaller requests and another endpoint to actually handle a single "cpu_intensive" request. What is the best way for "batch_request" break down the batch to smaller requests and invoke "cpu_intensive" so that CloudRun will scale the number of instances?
make http request to localhost - doesn't work since the load balancer is not aware of these calls.
keep the deployment URL in a conf file and make a network call to it?
Other suggestions?
With more detail, it's now clearer!!
You have 2 responsibilities
One to split -> Many request can be handle in parallel, no compute intensive
One to process -> Each request must be processed on a dedicated instance because of compute intensive process.
If your split performs internal calls (with localhost for example) you will be only on the same instance, and you will parallelize nothing (just multi thread the same request on the same instance)
So, for this, you need 2 services:
one to split, and it can accept several concurrent request
The second to process, and this time you need to set the concurrency param to 1 to be sure to accept only one request in the same time.
To improve your design, and if the batch processing can be asynchronous (I mean, the split process don't need to know when the batch process is over), you can add PubSub or Cloud Task in the middle to decouple the 2 parts.
And if the processing requires more than 4 CPUs 4Gb of memory, or takes more than 1 hour, use Cloud Run on GKE and not Cloud Run managed.
Last word: Now, if you don't use PubSub, the best way is to set the Batch Process URL in Env Var of your Split Service to know it.
I believe for this use case it's much better to use GKE rather than Cloud Run. You can create two kubernetes deployements one for the batch_request app and one for the cpu_intensive app. the second one will be used as worker for the batch_request app and will scale on demand when there are more requests to the batch_request app. I believe this is called master-worker architecture in which you separate your app front from intensive work or batch jobs.
I have a question regarding **writing test for HTTP DELETE method in JMeter using Concurrency Thread Group**. I want to measure **how many DELETEs** can it perform in certain amount of time for certain amount of Users (i.e. Threads) who are sending Concurrent HTTP (DELETE) Requests.
Concurrency Thread Group parameters are:
Target Concurrency: 50 (Threads)
RampUp Time: 10 secs
RampUp Steps Count: 5 secs
Hold Target Rate Time (sec): 5 secs
Threads Iterations Limit: infinite
The thing is that HTTP DELETE is idempotent operation i.e. if inovked on same resource (i.e. Record in database) it kind of doesn't make much sense. How can I achieve deletion of multiple EXISTING records in database by passing Entity's ID in URL? E.g.:
http://localhost:8080/api/authors/{id}
...where ID is being incremented for each User (i.e. Thread)?
My question is how can I automate deletion of multiple EXISTING rows in database (Postgres 11.8)...should I write some sort of script or is there other easier way to achieve that?
But again I guess it will probably perform multiple times same thing on same resources ID (e.g. HTTP DELETE will be invoked more than once on http://localhost:8080/api/authors/5).
Any help/advice is greatly appreciated.
P.S. I'm doing this to performance test my SpringBoot, Vert.X and Dropwizard RESTful Web service apps.
UPDATE1:
Sorry, I've didn't fully specify reason for writing these Test Use Case for my Web Service apps which communicate with Postgres DB. MAIN reason why I'm actually doing this testing is to test PERFORMANCES of blocking and NON-blocking WEB Server implementations for mentioned frameworks (SpringBoot, Dropwizard and Vert.X). Web servers are:
Blocking impelementations:
1.1. Apache Tomcat (SpringBoot)
1.2. Jetty (Dropwizard)
Non-blocking: Vert.X (uses own implementation based on Netty)
If I am using JMeter's JDBC Request in my Test Plan won't that actually slow down Test execution?
The easiest way is using either Counter config element or __counter() function in order to generate an incrementing number on each API hit:
More information: How to Use a Counter in a JMeter Test
Also the list of IDs can be obtained from the Postgres database via JDBC Request sampler and iterated using ForEach Controller
For testing purpose, I need to simulate client for generating 100,000 messages per second and send them to kafka topic. Is there any tool or way that can help me generate these random messages?
There's a built-in tool for generating dummy load, located in bin/kafka-producer-perf-test.sh (https://github.com/apache/kafka/blob/trunk/bin/kafka-producer-perf-test.sh). You may refer to https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java#L106 to figure out how to use it.
One usage example would be like that:
bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000 --throughput 100000
The key here is the --throughput 100000 flag which indicated "throttle maximum message amount to approx. 100000 messages per second"
The existing answers (e.g., kafka-producer-perf-test.sh) are useful for performance testing, but much less so when you need to generate more than just "a single stream of raw bytes". If you need, for example, to simulate more realistic data with nested structures, or generate data in multiple topics that have some relationship to each other, they are not sufficient. So if you need more than generating a bunch of raw bytes, I'd look at the alternatives below.
Update Dec 2020: As of today, I recommend the use of https://github.com/MichaelDrogalis/voluble. Some background info: The author is the product manager at Confluent for Kafka Streams and ksqlDB, and the author/developer of http://www.onyxplatform.org/.
From the Voluble README:
Creating realistic data by integrating with Java Faker.
Cross-topic relationships
Populating both keys and values of records
Making both primitive and complex/nested values
Bounded or unbounded streams of data
Tombstoning
Voluble ships as a Kafka connector to make it easy to scale and change serialization formats. You can use Kafka Connect through its REST API or integrated with ksqlDB. In this guide, I demonstrate using the latter, but the configuration is the same for both. I leave out Connect specific configuration like serializers and tasks that need to be configured for any connector.
Old answer (2016): I'd suggest to take a look at https://github.com/josephadler/eventsim, which will produce more "realistic" synthetic data (yeah, I am aware of the irony of what I just said :-P):
Eventsim is a program that generates event data for testing and demos.
It's written in Scala, because we are big data hipsters (at least
sometimes). It's designed to replicate page requests for a fake music
web site (picture something like Spotify); the results look like real
use data, but are totally fake. You can configure the program to
create as much data as you want: data for just a few users for a few
hours, or data for a huge number of users of users over many years.
You can write the data to files, or pipe it out to Apache Kafka.
You can use the fake data for product development, correctness
testing, demos, performance testing, training, or in any other place
where a stream of real looking data is useful. You probably shouldn't
use this data to research machine learning algorithms, and definitely
shouldn't use it to understand how real people behave.
You can make use of Kafka Connect to generate random test data. Check out this custom source Connector https://github.com/xushiyan/kafka-connect-datagen
It allows you to define some settings like message template and randomizable fields to generate test data. Also check out this post for detailed demonstration.
I'm submitting multiple POST submits on a REST API using same input Json. That means multi users (ex: 10000) are submitting the same POST with same Json to measure the performance of POST request, but I need to capture the result of completion on each submission using a GET method and still measure the performance of GET as well. This is a asynchronous process as follows.
POST submit
generates an ID1
wait for processing
in next step another ID2 will be generated
wait for processing
in next step another ID3 will be generated
wait for processing
final step is completion.
So I need to create a jmeter test plan that can process this Asynchronous POST submits by multi users and wait for them to be processed and finally capture the completion on each submission. I need to generate a graph and table format report that can show me latency and throughput. Sorry for my lengthy question. Thanks, Santana.
Based on your clarification in the comment, looks to me like you have a fairly straight forward script, which could be expressed like this:
Thread Group
HTTP Sampler 1 (POST)
Post-processor: save ID1 as a variable ${ID1}
Timer: wait for next step to be available
HTTP Sampler 2 (GET, uses ${ID1})
Post-processor: save ID2 as a variable ${ID2}
Timer: wait for next step to be available
HTTP Sampler 3 (GET, uses ${ID1} and ${ID2})
Post-Processor: extract completion status
(Optional) Assertion: check completion status
I cannot speak about which Timer specifically to use, or which Post-processor, they depend on specific requests you have.
You don't need to worry about multiple users from JMeter perspective (the variables are always independent for the users), but of course you need to make sure that multiple initial POSTs do not conflict with each other from application perspective (i.e. each post should process independent data)
Latency is a part of the standard interface used to save results in the file. But as JMeter's own doc states, latency measurement is a bit limited in JMeter:
JMeter measures the latency from just before sending the request to just after the first response has been received. Thus the time includes all the processing needed to assemble the request as well as assembling the first part of the response, which in general will be longer than one byte. Protocol analysers (such as Wireshark) measure the time when bytes are actually sent/received over the interface. The JMeter time should be closer to that which is experienced by a browser or other application client.
Throughput is available in some UI listeners, but can also be calculated in the same way as JMeter calculates it:
Throughput = (number of requests) / (total time)
using raw data in the file.
If you are planning to run 100-200 users (or for debug purposes), use UI listeners; with the higher load, use non-UI mode of JMeter, and save results in CSV which you can later analyze. I say get your test to pass in UI mode first with 100 users, and then setup a more robust multi-machine 10K user test.
I need to limit the number of orchestration instances spawned while debatching a large message in a streaming disassemble receive pipeline. Let’s say that I have a large xml coming in that contains 100 000 separate "Order" message. The receive pipeline would then debatch it and create 100 000 "ProcessOrder" orchestrations. This is too much and I need to limit that.
Requirements
The debatching needs to be done in a streaming manner so that I only load one "Order" message in memory at a time before sending it to the messagebox;
The debatching needs to be throttled based on the number of current running "ProcessOrder" orchestration instances (say if I already have 100 running instances, the debatching would wait till one is over to send another "Order" message to the messagebox).
Where I'm at
I have the receive pipeline that does the debatching and functional modifications to my messages. It does what it should in a streaming manner and puts individual messages in VirtualStreams;
I have an orchestration and helper methods that can limit the number of “ProcessOrder” orchestration instances.
The problem
I know that I can run a receive pipeline inside an orchestration (and that would solve my problem since on every "getnext" call to the pipeline, I could just hold on if there are too many running orchestration instances) but, digging in biztalk dlls, I noticed that using Microsoft.XLANGs.Pipeline.XLANGPipelineManager still loads up all the messages in memory instead of enumerating them like Microsoft.BizTalk.PipelineOM.PipelineManager does. I know they are putting every messages in VirtualStream but this is still inadequate, memory wise, for such a large message number.
Question
My next step would be to run the receive pipeline directly in the receive port (so it would use Microsoft.BizTalk.PipelineOM.PipelineManager) without having the orchestration that limits the number of “ProcessOrder” instances, but to meet the requirements, I would need to add a delay logic in my pipeline. Is this a viable option? If not, why? and what other alternative do I have?
You should debatch all messages once from pipeline and store those individual messages in MSMQ before even they are processed by orchestration. Use standard pipeline to debatch messages as they are efficient to handle large files debatching. MSMQ is available for free through Turn On Windows Features. Using MSMQ is very easy and does not require any development. Sending to MSMQ will be very fast 100K messages is not issue at all.
Then have a receive location to read from MSMQ. Depending on your orchestration throughput, you can control message flow by using BizTalk receive host throttling or by receiving the messages from MSMQ in Order or using the combination of both. Make sure you have separate host instance for both receive MSMQ and send MSMQ and for your orchestration processing.
This will be done through all configurations without any extra code simplifing your design. Make sure you have orchestration with minimum number of persistent points.