I am testing pub sub region failure.
So I only need one region topic up. And if I ping the same topic with another endpoint , it should fail. I am testing with emulator. But in emulator there is no way to add regional endpoints. How can I test this then ?
I am testing a scenario where I can check if one region is down , I can send the message to the other region.
This is not a scenario that the emulator is designed to test. Communicating with the emulator requires one to change the address one contacts to be the address of the emulator. Some client libraries do this internally when the emulator environment variable is set.
Even if it did allow you to specify regional endpoints, it wouldn't have a way to selectively reject requests that come in on one address and not the other.
I'm not sure this is the kind of scenario that requires the emulator as those tests are really designed to ensure proper behavior given the way the Pub/Sub service itself works. In this case, you need only test that the reaction to RPC failures is as expected, so you could fake error responses using libraries designed for such things in the language you are using and ensure that you then send requests to a different region when those initial failures are returned.
Related
I want to test my Rest API with end-to-end tests. As I understand, the difference between integration tests is that we don't do in-memory system configuration, but use real test DB and network requests.
But I can't understand how to handle third-party API requests(like GitHub or Bitbucket API).
Is it a normal practice to create a fake Github account with fake data that would be fetched by my tests ?
And what to do with access tokens, not all services are public and even public services can fail with rate limit.
Is it a normal practice to create a fake Github account with fake data that would be fetched by my tests ?
Yes. The purpose of an E2E test (vs an integration test) is to verify that the full system works with all the real system components in place, both the ones you control and the ones you don't. This can be hard to setup and a pain to maintain; but many of those pain points will be exposing real potential issues in your production service. How your service responds to that instability is itself a feature to be tested: does your system crash and burn, or does it gracefully present an error message and support good retry handling?
This also nets you a type of coverage that mocks cannot provide: If the third party API you're using is naughty and introduces some sort of breaking change, your E2E tests will catch it. This is a decent reason to continually run your E2E suite; not just during deploys.
The next level of this sort of testing is chaos engineering where not only do you test your production systems, but you purposefully introduce faults (yes, into prod) in order to ensure that your service can really handle the pressure.
And what to do with access tokens, not all services are public and even public services can fail with rate limit.
Your staging environment should be configured with separate sandbox accounts for external services. I'm not sure what you mean by "not all services are public" but just strive to keep your staging environment (or test users on prod) as identical to a real prod user as possible. For services that don't support multiple access tokens, you can get creative and try to clearly delineate your test data within their system.
Rate limits can be annoying, but if you're getting close enough that your tests push you over the limit, then you should be pursuing a strategy to address that anyways (negotiating with the service, getting multiple accounts, ...).
Running your tests against 3rd party services can result in slow and flaky tests when the service is down or when network latency triggers certain testing timeouts. Not to mention you run the risk of triggering API rate limits depending on the 3rd party service you're hitting. Your tests should ideally be deterministic, not failing randomly, and not needing conditional logic to handle errors within a particular test case. If you expect to need to handle errors, then there should be a specific test case to cover those errors that runs in every build, not waiting for non-deterministic failures to come in from the 3rd party.
One argument people will make is that your tests should notify you if the 3rd party API breaks for one reason or another. Generally speaking, though, most major 3rd party APIs are extremely stable and are unlikely to make breaking changes. Even if it does happen, this is an awkward and confusing way to find out that the API is broken, and in all likelihood, your tests aren't going to be the first place you hear it from. More likely your customers and your production error tracker will notify you. If you want to track when these services change or go down, it makes sense to have a regular production check of some sort to verify it.
As for how to write tests around these situations, that's a little more tricky. There are tools such as VCR in Ruby which work well for stubbing out your language's internet connections and allowing you to stub out, record, and customize responses (there's a list of similar implementations in other languages further down in their readme). That doesn't work for when your browser connects to those resources in automated end-to-end tests, though. There are tools that proxy your browser's web connection such as Puffing Billy in Ruby, but it's a pretty involved process to set up, including managing security certificates. This seems pretty brittle and hard to debug when something isn't working quite right.
Your best bet for writing tests that are deterministic and maintainable may be to fake out the service in test mode. thoughtbot has a pretty decent video on this and here's a high-level article from CircleCI. Essentially, you swap in an adapter in test mode that stands in for your 3rd party service integration. Maybe what you can do on your local machine is make it possible to optionally use the real service or the adapter via an environment variable in order to verify that the tests run the same against both. You could also set up a daily build to run against the real thing so that it would verify that the tests still work alright without introducing a lot of flakiness to your more frequent builds. One issue I've run into, though, is that even if I set up a test account on that 3rd party service, the results will change over time as I add or modify information for the sake of testing new functionality, such as adding new repos, modifying issues, etc. It requires additional consideration for maintaining your test account as a set of fixtures for all of your tests.
One additional tool I've come across that may be helpful is the likes of ngrok-tunnel (Ruby again). This is only relevant in cases where you need the 3rd party service to contact your app, since they can't send requests across the web to localhost:3000. If you've configured some sort of webhooks, services like this can make testing a lot more straightforward.
We have a large number of small devices deployed in the wild that communicate with a central server via a REST API. It will send a status update on a regular basis with some sensor data and will also ask the server if there are any software updates that need to be applied. If there are, it will download it etc.
I'd like some way to prevent my server API from being used by unauthorised clients. Here are the solutions that I have and the issues which I anticipate. This question is somewhat open ended but it is mostly a question of best practices for, what I think is, a common situation.
Keep a single username/password on the server and then use basic auth over https. This is simple but if I change it, I'd have to somehow change the credentials on all of my devices (which number in the low thousands). This is hard to do reliably. There's also a problem with how the devices are going to get the update without first changing their creds.
The second is to create a username/password for every device. Then I can update individual ones and manage them but there's a lot of state on the server and I'm worried about having this much data on the server.
Some kind of automatic key based system whereby the devices can ask for an expirable key and then use that for all transactions. If the key expires or is invalidated, that device can't connect anymore. This would necessitate some kind of id for the device which I'm not sure how to handle.
So, how would I do this?
Update
In our setup, we have assigned hostnames for each of the devices we're going to deploy created beforehand. Once the devices switch on, they will contact the server with their hostnames and thereby register themselves. The MAC address and other information is passed as part of the initial registration handshake.
So, the "unique identifer" which was mentioned in the answer (and comment) by Noksi is the hostname. This can be easily spoofed (since the hostnames have a pattern). However, when the devices first come online, we can get the mac and, after that, only refresh the token if the request comes from the same mac. This opens up the possibility of a fake registration happening the first time though.
If there is some way to uniquely identify each device (similar to a MAC address) then that would be the key. Keep a registry of the devices and each device would register with the server. the server would provide the device with a token and associate it (the token) with the registered device. the device uses that token to make requests. tokens can be made to expire and renewed as needed or can be black listed if needed. This seems to be similar to option 3 in the proposed options.
I setup an Active/Passive cluster with Pacemaker/Corosync/DRBD. I wanted to make an Asterisk server HA. The solution works perfectly but when the service fails on one server and starts on another all registered SIP clients with the active server will be lost. And the passive server show nothing in the output of:
sip show peers
Until clients make a call or register again. One solution is to set the Registration rate on clients to 1 Min or so. Are there other options? For example integrating Asterisk with a DBMS helps to save this kind of state in a DB??
First of all doing clusters by non-expert is bad idea.
You can use realtime sip architecture, it save state in database. Complexity - average. Note, "sip show peers" for realtime also show nothing.
You can use memory duplicating cluster(some solution for xen exists) which will copy memory state from one server to other. Complexity - very complex.
how can i measure stress testing for the iPhone app ?
i need stress testing not performance testing, for example 100 users access the database of the app which is on the server at the same time.
any help?
thanks in advance
First, you need to decide if you need to test the client-side (iPhone) app, the server-side code, or both.
Testing ONLY the server-side, might make this much easier - especially if it is using HTTP to communicate with the server and exchanges data via a text-based format (XML, JSON, etc). There are many web load testing tools available which can handle this scenario. Using our Load Tester product, for example, you would configure the proxy settings on your iPhone to point to our software running on a local machine. Then start a recording and use the application. Load Tester will record the messages exchanged with the server. You can then replay the scenario, en masse, to simulate many users hitting your server simultaneously. The process, at a high level, is the same with most of the web load testing tools.
Of course, the requests to the server can't be replayed exactly as recorded - they'll need to be customized to accurately simulate multiple users. How much customization is needed will depend on the kind of data being exchanged, the complexity of the scenario and the ability of the tool to automatically configure dynamic fields (and this is one area where the abilities of the tools vary greatly).
Hope that helps!
A basic simulation would involve running your unit tests on OS X, using many simultaneous unit test processes (with unique simulated users, and other variables).
If you need more 'stress', add machines - you'll likely end up hitting io or network limits from one machine relatively early on.
I am developing an iPhone application that allows to basically click through a series of actions. These series are predefined and synced with a common configuration server.
That app might be running on multiple devices at the same time. All devices are assumed to have the same series of actions defined on them. All devices are considered equal, there is not a server and multiple clients or something like that.
(Only) one of these devices is used by a person at any given time, it is however possible that the person switches to a different device at any given time. All "passive" devices need to be synchronized with the active one, so that they display the same action.
The whole thing should happen as automatically as possible. No selection of devices, configuration, all devices in the same network take part in the same series of actions.
One additional requirement is that a device could join during a presentation (a series of actions) and needs to jump to the currently active action.
Right now, I see two options to implement the networking/communication part of that:
Bonjour. I have implemented a working prototype that can automatically connect with one (1) other device in the network and communicate with that. I am not sure at this point how much additional work the "multiple devices" requirement is. Would I have to open a set of connections for every device and manually send the sync events to all of them? Is there a better way or does bonjour provide anything to help me with that? What does Bonjour provide given that I want to communicate with every device in the network anyway?
Multicast with AsyncUdpSocket. Simply define a port and send multicast sync events out to that port. I guess the main issue compared to using bonjour with tcp would be that the connection is not safe and packets could be lost. This is however in a private, protected wlan network with low traffic if that would really be an issue. Are there other disadvantages that I'm not seeing? Because that sounds like a relatively easy option at this point...
Which one would you suggest? Or is there another, better alternative that I'm not thinking of?
You should check out GameKit (built in to iOS)--they have a lot of the machinery you need in a convenient package. You can easily discover peers on the network and easily send data back for forth between clients (broadcast or peer to peer)
In my experience Bonjour is perfect for what you want. There's an excellent tutorial with associated source code: Chatty that can be easily modified to suit your purposes.
I hobbled together a distributed message bus for the iphone (no centralized server) that would work great for this. It should be noted that the UI guy made a mess of the code, so thar' be dragons there: https://code.google.com/p/iphonebusmiddleware/
The basic idea is to use bonjour to form a network with leader election. The leader becomes the hub through which all the slaves subscribe to topics of interest. Then any message sent to a given topic is delivered to every node subscribed to said topic. A master disconnection simple means restarting the leader election process.