Alex and I recently gave a talk at Pipeline Conference about our approach of testing in production.
With our limited time we focused on things we check in production. Running our acceptance/integration tests, performance tests, and data fuzzing against our production systems. We also prefer doing user acceptance testing and exploratory testing in production.
In a Continuous-Deployment environment with several releases a day there’s little need or time for staging/testing environments. They just delay the rate at which we can make changes to production. We can always hide incomplete/untested features from users with Feature Toggles
“How do you cope with junk data in production?”
The best question we were asked about the approach of both checking and testing in production, was “How do you cope with the junk test data that it produces?”. Whether it is an automated system injecting bad data into our application, or a human looking for new ways to break the system, we don’t want to see this test data polluting real users’ views or reports. How do we handle this?
Stateless or Read Only Applications
Sometimes we cheat beacause it’s possible to make a separately releasable part of a system entirely stateless. The application is effectively a pure function. Data goes in, data comes out with some deterministic transformation.
These are straightforward to both check and test in production, as no action we take can result in unexpected side-effects. It’s also very hard to alter the behaviour of a stateless system, but not impossible – for example if you overload the system, its performance will be altered.
Similarly, we can test and check read-only applications to our heart’s content without worrying about data we generate. If we can keep things that read and write data separate, we don’t have to worry about any testing of the read-only parts.
Side-Effect Toggles
When we do have side-effects, if we make the side-effects controllable we can avoid triggering them except when explicitly checking that they exist.
For example, an ad unit on a web page is generally read-only in that no action you can perform with it can change it. However, it does trigger side effects in that the advertising company can track that you are viewing or clicking on the ad.
If we had a way of loading the ad, but could disable its ability to send out tracking events, then we can check any other behaviour of the ad without worrying about the side effects. This technique is useful for running selenium webdriver tests against production systems to check the user interactions, without triggering side effects.
In a more complex application we could have the ability to only grant certain users read-only access. That way we can be sure that bots or humans using those accounts can’t generate invalid data.
Data Toggles
Ultimately, if we are going to check or test that our production systems are behaving as we expect, we do need to be able to write data. One way to deal with this is data-toggles. A simple boolean flag against a record which indicates the data is test data. You can then use a similar flag at a user or query level to hide/show the test data.
This might sound like a bit of a pain, but often this functionality is needed in any case to fulfil business requirements –
Reporting systems often need a way to filter out data which is invalid, anomalous, or outdated. Test data is just one type of data that is invalid in normal reports.
Many systems need a security system to control which users have access to what data. This is exactly what we want to achieve – hiding data generated by test users from real users.
We can often re-use the permissions and filtering systems that we we needed to build anyway, to hide our test data.