In one of the companies where I collaborated, I talked part in a discussion about the scope of testing of a service that connects to a well-known financial institution in Mexico. If you have worked with this institution you will know that they have a sanbox environment, which allows us as quality engineers to perform tests simulating the behavior of a real user.
So… during the meeting I was calm because I had enough data to do a lot of invoices, I just had to worry about making a complete test plan/strategy. Suddenly someone mentions that there is no time to connect the sandox env with the staging env, so we will do all the testing in production, the argument to do that was “Everything can be tested in prod with real inputs and no customer can be affected because you will test with your own account.”
My first thought was “it’s a joke” and I waited for a microphone to open and laughter to be heard, but it wasn’t…. Everyone at the meeting was confident and supportive of the idea. I’m sure this scene sounds familiar, I want to share with you some reasons why you should not use the production environment for your testing.
It is dangerous.
Testing in production or with real data can lead us to unnecessarily expose a fundamental part of our system and/or company, whether at an economic, information or even security level. This could mean a loss of revenue, customers or even legal problems.
In addition, production environments are usually very secure and sensitive to external access. Testing in a production environment can introduce additional security risks (open VPNs, unsecured ports, decryptable databases, etc.).
It is too expensive.
This type of testing also involves a large investment of time and effort that sometimes involves people outside the quality area (an accountant, a developer, etc.) who must take steps to undo or redo the flows/inputs/outputs/documents/etc. that were affected during the test.
In some cases, bugs and issues can be difficult to reproduce in a consistent and coherent way, making it more difficult for quality teams to reproduce and report on that issue and for developers to fix them leading to longer resolution times and may affect the availability of the product or service to end users.
Lack of control.
You should be aware that when testing in production it is difficult to control all variables that could affect the testing process, which makes testing unpredictable. In some cases, access to the production environment is limited to authorized personnel only, which can limit the number of people who can test.
A good test suite requires a variety of test scenarios to ensure that the software meets the desired quality standards. Testing in a production environment may limit the ability to test different scenarios, which can result in incomplete testing and undetected bugs.
So, should I never test in production?
My answer is yes, you can test in production but not end-to-end and/or functional tests, there are three tests that I consider suitable for a production environment:
Synthetic Tests
Synthetic tests are a type of testing technique that involves generating simulated data or activities to monitoring real-world usage scenarios. The goal of synthetic tests is to identify potential issues that may arise in the system under different real scenarios.
For example, a common synthetic test involve monitoring hundreds of simultaneous users accessing to the application to see how the system performs under heavy load.
Whether your team developing a web application, mobile app, or enterprise software solution, synthetic tests are a valuable addition to any testing strategy.
A/B Testing
A/B testing is a testing technique to test the effectiveness of different design elements, features, etc. It involves creating two or more versions of the application, making small changes and measuring the impact on user behavior.
For example, a website may have two different versions of a home page. Users are randomly assigned to one of the two versions, and their behavior is tracked using metrics such as click-through or time on page. The data is then analyzed to determine which version performs better based on the specific KPI being measured.
Because of its purpose A/B testing is not usually 100% the responsibility of the QA team, it is a collaborative work with the product team (UX, UI, marketing, PO, etc).
Blue/Green deployments
Blue/green deployment involves creating two identical production environments, with one environment (blue) serving live traffic while the other environment (green) is updated with new code or configurations.
Once the green environment has been updated and thoroughly tested, traffic is redirected to the green environment, and the blue environment is updated.
By creating two identical production environments, blue/green deployment allows updates to be thoroughly tested before going live, reducing the risk of issues affecting live traffic
It is important to note that while testing in production may be necessary or beneficial in certain situations, it should always be done with caution and proper planning to minimize the risk of negatively impacting users or the live environment.
Proper monitoring, logging, and rollback procedures should be in place to address any issues that may arise during testing. Additionally, it is important to communicate with other teams (like customer support, sales, etc) about any testing or maintenance activities that may affect the user experience.
In summary…
Testing in a production environment can be a tempting solution for software developers and QA teams who are looking for a realistic testing environment or are facing time or cost constraints. However, testing in production should be approached with caution and careful planning to avoid negatively impacting users or the live environment. Adequate backup and rollback procedures should be in place to ensure that any issues or failures can be addressed quickly and effectively.
It is important to note that testing in production should not be used as a replacement for testing in separate staging or testing environments. These environments allow for more controlled and predictable testing scenarios, which can help catch issues and bugs before they reach production. Overall, a well-rounded testing strategy that includes both testing in separate environments and testing in production can help ensure that software is thoroughly tested and reliable for end-users.
By the way, at the end the company decided to continue testing in a production environment using real accounting data, they didn't count on that day being payday in many companies which implies that the product had much more traffic than usual, so this quickly got out of control because the application didn't even work as it did before this, tickets from angry customers kept coming in and they had to spend more than a day to return the changes involved.
QA Rocks 
Leave a Reply