how can i measure stress testing for the iPhone app ?
i need stress testing not performance testing, for example 100 users access the database of the app which is on the server at the same time.
any help?
thanks in advance
First, you need to decide if you need to test the client-side (iPhone) app, the server-side code, or both.
Testing ONLY the server-side, might make this much easier - especially if it is using HTTP to communicate with the server and exchanges data via a text-based format (XML, JSON, etc). There are many web load testing tools available which can handle this scenario. Using our Load Tester product, for example, you would configure the proxy settings on your iPhone to point to our software running on a local machine. Then start a recording and use the application. Load Tester will record the messages exchanged with the server. You can then replay the scenario, en masse, to simulate many users hitting your server simultaneously. The process, at a high level, is the same with most of the web load testing tools.
Of course, the requests to the server can't be replayed exactly as recorded - they'll need to be customized to accurately simulate multiple users. How much customization is needed will depend on the kind of data being exchanged, the complexity of the scenario and the ability of the tool to automatically configure dynamic fields (and this is one area where the abilities of the tools vary greatly).
Hope that helps!
A basic simulation would involve running your unit tests on OS X, using many simultaneous unit test processes (with unique simulated users, and other variables).
If you need more 'stress', add machines - you'll likely end up hitting io or network limits from one machine relatively early on.
Related
I want to test my Rest API with end-to-end tests. As I understand, the difference between integration tests is that we don't do in-memory system configuration, but use real test DB and network requests.
But I can't understand how to handle third-party API requests(like GitHub or Bitbucket API).
Is it a normal practice to create a fake Github account with fake data that would be fetched by my tests ?
And what to do with access tokens, not all services are public and even public services can fail with rate limit.
Is it a normal practice to create a fake Github account with fake data that would be fetched by my tests ?
Yes. The purpose of an E2E test (vs an integration test) is to verify that the full system works with all the real system components in place, both the ones you control and the ones you don't. This can be hard to setup and a pain to maintain; but many of those pain points will be exposing real potential issues in your production service. How your service responds to that instability is itself a feature to be tested: does your system crash and burn, or does it gracefully present an error message and support good retry handling?
This also nets you a type of coverage that mocks cannot provide: If the third party API you're using is naughty and introduces some sort of breaking change, your E2E tests will catch it. This is a decent reason to continually run your E2E suite; not just during deploys.
The next level of this sort of testing is chaos engineering where not only do you test your production systems, but you purposefully introduce faults (yes, into prod) in order to ensure that your service can really handle the pressure.
And what to do with access tokens, not all services are public and even public services can fail with rate limit.
Your staging environment should be configured with separate sandbox accounts for external services. I'm not sure what you mean by "not all services are public" but just strive to keep your staging environment (or test users on prod) as identical to a real prod user as possible. For services that don't support multiple access tokens, you can get creative and try to clearly delineate your test data within their system.
Rate limits can be annoying, but if you're getting close enough that your tests push you over the limit, then you should be pursuing a strategy to address that anyways (negotiating with the service, getting multiple accounts, ...).
Running your tests against 3rd party services can result in slow and flaky tests when the service is down or when network latency triggers certain testing timeouts. Not to mention you run the risk of triggering API rate limits depending on the 3rd party service you're hitting. Your tests should ideally be deterministic, not failing randomly, and not needing conditional logic to handle errors within a particular test case. If you expect to need to handle errors, then there should be a specific test case to cover those errors that runs in every build, not waiting for non-deterministic failures to come in from the 3rd party.
One argument people will make is that your tests should notify you if the 3rd party API breaks for one reason or another. Generally speaking, though, most major 3rd party APIs are extremely stable and are unlikely to make breaking changes. Even if it does happen, this is an awkward and confusing way to find out that the API is broken, and in all likelihood, your tests aren't going to be the first place you hear it from. More likely your customers and your production error tracker will notify you. If you want to track when these services change or go down, it makes sense to have a regular production check of some sort to verify it.
As for how to write tests around these situations, that's a little more tricky. There are tools such as VCR in Ruby which work well for stubbing out your language's internet connections and allowing you to stub out, record, and customize responses (there's a list of similar implementations in other languages further down in their readme). That doesn't work for when your browser connects to those resources in automated end-to-end tests, though. There are tools that proxy your browser's web connection such as Puffing Billy in Ruby, but it's a pretty involved process to set up, including managing security certificates. This seems pretty brittle and hard to debug when something isn't working quite right.
Your best bet for writing tests that are deterministic and maintainable may be to fake out the service in test mode. thoughtbot has a pretty decent video on this and here's a high-level article from CircleCI. Essentially, you swap in an adapter in test mode that stands in for your 3rd party service integration. Maybe what you can do on your local machine is make it possible to optionally use the real service or the adapter via an environment variable in order to verify that the tests run the same against both. You could also set up a daily build to run against the real thing so that it would verify that the tests still work alright without introducing a lot of flakiness to your more frequent builds. One issue I've run into, though, is that even if I set up a test account on that 3rd party service, the results will change over time as I add or modify information for the sake of testing new functionality, such as adding new repos, modifying issues, etc. It requires additional consideration for maintaining your test account as a set of fixtures for all of your tests.
One additional tool I've come across that may be helpful is the likes of ngrok-tunnel (Ruby again). This is only relevant in cases where you need the 3rd party service to contact your app, since they can't send requests across the web to localhost:3000. If you've configured some sort of webhooks, services like this can make testing a lot more straightforward.
Apologies if something similar has been asked in the future but my search didn't return, what I would consider, directly related.
I am trying to implement a service with its backend in AWS EC2/S3 and front-end in iPhone and the service is more or less like a todo-list. This is not a novel idea but will help me in a class I teach about IT infrastructure.
Unfortunately I have access to only my own iPhone and I cannot demonstrate scalability over AWS, etc.
Is there a way/software tool/framework to simulate virtual users for this app that can send requests to the AWS servers pretending to be from different accounts/apps?
The simulator should send requests just like my actual iphone app would send if I were to add an item to the list or delete or edit.
I understand stress testing is a well established topic but here I want to just simulate multiple users and demonstrate scalability instead of trying to push the Web service to its limits. Neither am I sure if this completely overlaps with traffic simulation.
Any help will be deeply appreciated.
You might be able to do it using Apache JMeter. That depends on what you have going on on the backend. But it supports the following server types:
Web - HTTP, HTTPS
SOAP
Database via JDBC
LDAP
JMS
Mail - SMTP(S), POP3(S) and IMAP(S)
Native commands or shell scripts
You should be able to wire something together with that.
http://jmeter.apache.org/
http://www.opensourcetesting.org/performance.php
I've used it at various points to simulate VERY heavy loads for my services running in AWS/EC2.
Apache Benchmark is a very convenient tool for doing HTTP load testing -- you can have it make concurrent requests to simulate multiple users. It's main advantage over other tools is that it's simple and easy to get started with. If your backend listens on HTTP, it might be worth trying ab before investing any time in something more complex.
I'm using blocking TCP sockets in C and I want to simulate a high load on the server when there are many simultaneous connections and then I want to measure the time necessary to access the server via a browser during this high load time (the server understands HTTP headers).
Also each client request ends fast (sends a HTTP header - gets text).
How do I do this (without crashing my local machine -> I tried using fork to make many clients; also, I have a virtual machine too).
If anyone has an idea or some general directions about how to do this, it would mean a lot.
Edit: I need to run this with my own client, which uses a modified version of the OpenSSL library to connect to my SSL/TLS server, so I can't use external test tools.
I want to know how to build the client and the server. I don't know too much about other sockets than the blocking ones, I'm just skimming through the UNIX Network Programming book of Richard Stevens, but I was wondering if anyone could point out the exact solution.
Thank you !
The easiest resolution to this would be to download an existing stress testing framework such as fwptt ( http://fwptt.sourceforge.net/ ).
If you want to implemennt your own stress testing framework, I'd suggest you lose the blocking nature of your code and go with a parallel design that will scale beautifully. The limiting factor is pretty much your CPU then.
Having two physical servers would be ideal, so that then your stress test isn't affecting the CPU (and therefore the response times) of the server. Also that VM of yours drains up precious CPU time.
I need to implement an in-application message-box system (like any forum/social network has).
Number of users is in the tens of thousands and up.
I am thinking to do it using an actual email server and write a client for it inside the application (or just pick an open source client and adjust it to appear like a private messages system).
Or
Should I just do it as an integral part of my application (DB, server side and all)?
Which approach is the correct one?
(I do not think this is a consideration, but the bulk of the application is written in PHP).
Use your application back-end.
Email (SMTP/IMAP) sounds simple but has way too much administration and programming overhead if you do it properly - or you get random problems.
If you need to send emails to your users (which you shouldn't need to - I quite like how Stackoverflow works for example in that respect) then just be a client to email (SMTP).
Another issue with 'email' is it's an old protocol which does not fit into todays needs. Hence we have new protocols and applications, like XMPP and Google Wave.
I've been talking with someone about the possibility of a iPhone development contract gig. All I really know at this point is that there is a company that wants to make an iPhone app that will hit their internal database. I'm not sure what the database type is( Oracle, MySQL, etc...).
I've wanted to know if the database type was Oracle or MySQL if there is a big learning curve for connecting to one of these across the internet?
If it's a real pain I may do more research before accepting the conract.
I would advise against directly accessing the database from the iPhone application.
Usually, you would create a web service which accesses the database, and then you consume that web service from the iPhone application.
Create a web service. This allows you to make the iphone app more of a thin client. Let the application push commands to the web service for processing and interaction with the database returning only the data needed by the app.
This option is better for the app, the database, and the customer's security.
You can easily perform the connection over the internet, the same way you would locally, but you are opening the database up to attacks if it will accept communication from any remote IP address. Typically you will just connect via a socket open to the server's remote IP address over the open port, MySQL's default port is 3306.
I would recommend against this sort of system in general unless there is some critical reason they want their internal database exposed to the world's hacker community.
What I am doing is creating a web service using Sinatra to access the online database.
Those answers from 2009 are mostly obsolete now.
http://ODBCrouter.com/ipad (new) has XCode client-side ODBC libraries, header files and multi-threaded Objective C objects that let your apps send SQL to server-side ODBC drivers and get back binary results! This reduces the need to stop and separately maintain SOAP/REST servers that can get pretty frightening anyway after a while maintaining it.
The XML schemes were okay for transferring static configurations to mobile devices "every once in a while", but XML was meant for infrequent inter-company type transfers in a "server environment" (with power cords, wired networks and air conditioning) and is definitely not efficient for frequent database queries coming in from n-copies of a mobile app. There are third-party JSON libraries that help things, but even with JSON, everything has to be encoded (and decoded) from the binary representation in the database to text representation on the server (only fine if it's going to be shown to the user in a web browser anyway, but not fine if the mobile app is going to translate it right back into binary so that it can perform calculations "behind the scenes" to what is going on with the user). Aside from the higher network overhead and battery power the mobile CPU will draw with XML and JSON, it will also make you buy more RAM and CPU power on the back-end server faster than just using an ODBC connection to the database.