Time and cost overruns: they're every engineering manager's nightmare. One of the simplest strategies for keeping your project on time and within budget is effective smoke testing.
What is smoke testing?
Smoke testing is a technique used in many industries. In vehicle maintenance, for example, vacuum seals can be tested by pumping smoke through the system to detect leaks.
The term originated in the plumbing trade. Blowing smoke through a pipe can instantly detect two types of faults: leaks and blockages. It can't, however, test how well the system will work under pressure and whether clean water will be delivered at the other end.
This is a good analogy for using smoke testing in software development. A smoke test won't guarantee the system is perfect, but it can quickly tell you whether it is functioning as expected. A good smoke test does basic checks to see that the most critical test cases can be completed. It doesn't check that the results are perfect; it simply verifies that the system is stable enough for meaningful testing.
Smoke tests should be fast rather than thorough. Identify essential processes across the entire system and create an end-to-end smoke test for each. Do not test for accuracy or performance – all you want to know is whether the process can be completed.
Ideally, a smoke test should not take more than half an hour to complete, even on a large system. On small systems, it should take only a few minutes.
Smoke testing is also known as build verification testing and confidence testing. It's often been likened to switching on an electrical circuit: if you see smoke, you don't need to test any further because you know it's broken.
Why is smoke testing important?
Imagine this scenario. You have a large project, and it's ready for QA testing. You've planned the test in detail, and you've booked, among other things:
A team of testers
Hardware facilities
A venue
A testing support team
Lots of coffee
You've set aside ten days for the testing and hope to meet your go-live target.
The testers log in. You're all good to go. Then, the web server starts returning various error messages because the back-end software has the wrong version of an obscure dependency. While the developers attempt to resolve this, the hardware sits idle, the testers play games, and the only thing that's really being utilized is the coffee. By the time the problem is resolved, you may have lost days of testing time, and your budget is in tatters.
It could be worse, though. The testers could work for three days before discovering the system has unrecoverable errors. The software then goes back to the developers, who make extensive changes, and all the tests must be done again.
Running a simple smoke test before releasing the build for testing can avoid all this. The objective of smoke testing is to answer the question: Is the system capable of running meaningful tests?
When should you run smoke tests?
Smoke tests are integral to any testing strategy and should be the last stage in the build process. Verifying that the system is stable before any deeper tests are carried out will save both time and money. Smoke tests can be run:
As the first stage in system testing
As the first stage in QA testing
As the first stage in regression testing
Before going live in production
Periodically as part of system health checks.
In the context of the traditional four stages of testing, the diagram below shows where smoke testing fits into the process.
When changes have been made to the system, smoke testing should be the first step in the regression test immediately before sanity testing. This is illustrated below.
In both of these cases, if the smoke test returns any errors, the build should fail and be returned to the developers for resolution. No further testing should take place.
The diagram below illustrates this testing process.
Smoke Testing vs. Sanity Testing
Smoke testing is often confused with sanity testing. They are both quick checks on the system before more detailed testing occurs. Do you really need both? Yes, you do. There are essential differences:
Purpose: Smoke testing verifies that the system as a whole is functional. Sanity testing checks to see that changes are working as they should and have not caused problems with other related functionality. Smoke testing answers the question: Can this system be tested? Sanity testing answers the questions: Do the changes work, and have they broken anything else?
Focus: Smoke testing is a broad but superficial test that checks that every essential aspect of the system is stable enough for further testing. Sanity testing is a narrower but deeper test. It's a more thorough test of the changes and the areas of the system that could be affected by them.
Repeatability: Smoke tests are standardized and repeated every time the system is built. Therefore, they can be pre-planned and scripted. On the other hand, each sanity test will be different since it tests a unique set of changes. Consequently, it will often be carried out manually without a predefined script.
Testing schedule: Smoke testing should be part of all types of testing, except for the very early stages, such as unit testing. Sanity testing is specifically a part of regression testing.
How should smoke tests be carried Out?
Since smoke tests will be run repeatedly, they should be well-planned and documented so they can be run quickly and with minimum effort. With any test plan, you must decide whether the test should be manual, automated, or hybrid (a mixture of manual and automated processes.)
Except for tiny and simple systems, smoke tests should be as automated as possible. This not only speeds up the testing process but eliminates human errors that could affect the test outcome. Since smoke tests are testing straightforward common paths, they're unlikely to need the type of decision-making that needs human intuition.
If you're using a CI /CD (Continuous Integration / Continuous Deployment) process, smoke tests should be built into that pipeline. A data clean-up should run after a smoke test because subsequent test phases will need clean data.
Planning and Building a Smoke Test: An Example from Banking
Smoke tests must be properly planned since they'll be run on every build. An incomplete plan can cost you money since it may verify the system as stable, only to have subsequent tests crash and have to be redone. On the other hand, a poorly planned smoke test can also add an unacceptable amount of time to every build. This is likely due to it either including inessential features or it's testing too deeply.
Let's take an example from banking and look at how you might plan and build your smoke test. In this example, the banking app is accessed via a web browser. It allows clients to manage their accounts online. The app looks like this:
Include All Stakeholders
Decisions regarding which features will and will not be included in the smoke test should be made with consideration from the team. You'll need to include relevant stakeholders. Stakeholders will help inform the needs of the business and your ICP (Ideal Customer Profile). Stakeholders can include:
Product owner: This will be someone who has a sound knowledge of the business processes and understands the needs of your ICP.
Business analyst: This person will be primarily responsible for creating the original functional specifications of the system.
Testing team: They will have experience on what can make or break a test.
Developers: They know the system intimately and can anticipate cause and effect.
Identifying Essential Processes
To do this, you'd need to go through the list of the system's use cases. For each one, you'd need to decide whether failing during testing would prevent further meaningful tests from taking place. If the answer is yes, it should be included in the smoke test. Otherwise, it should be excluded.
For the online banking application, your list could include:
In this example, your smoke test would only check the critical functions, i.e., whether a customer can log in, transfer funds, browse transactions, view transaction details, and request a statement. It would not check searching transactions, logging out, or other non-critical features.
You would typically include only valid transactions in the smoke test. Validation will be checked in a later test phase. You would include only minimal checking of the results of each test. For example, you wouldn't check that the statement was entirely accurate or aesthetically pleasing; you only want to know if a statement can be produced.
Unusual conditions and exceptions shouldn't be included in smoke tests
Prepare Test Cases
These could likely be selected and copied from the QA test plan. However, they should be formally documented since this test will be run frequently. For the banking application, a subset of the test cases may look like this:
Prepare Test Scripts
Now you have your planned test cases; you can go ahead and create test scripts. As mentioned previously, smoke tests should almost always be automated.
You probably decided on an automation tool for your testing when you planned the QA test. Since it’s a web-based system, you’d be likely to use something like Selenium. Your test tool’s user manual will guide you through the process of setting up test scripts.
It’s important to do a careful test of your smoke test since you’ll be relying on its accuracy in the future. Run your scripts against both a stable and an unstable build, and manually verify that they identify all issues correctly.
Review and Refine the Process
Your smoke tests shouldn’t be static. You need to monitor them throughout the lifespan of the system. Did they let through an error that broke the system and prevented a later test from completing successfully? Are they identifying non-existent errors? Are they taking too long to run? Have new features been added to the system that should be included in the smoke tests? You should have a process in place that ensures these reviews are carried out and acted on.
Limitations of Smoke Testing
A good smoke test should only do minimal checks to ensure that essential test cases can be completed end to end. Since the objective is to save time and costs, the test should be able to run quickly and not add large overheads to the process.
A smoke test should not:
Test extensive data validation
Check for complete accuracy of calculations
Carry out exhaustive checks that data has been stored correctly
Test performance
Test security procedures
Check that forms, pages, and documents are lined up correctly and look nice
Test every possible permutation.
Check the quality of error reporting
Test that the system works with obscure web browsers.
All these things will be checked during the main testing processes.
Benefits of Smoke Testing
Finally, let’s summarize the benefits of including effective smoke tests in your software development cycle.
Critical Issues are Identified and Fixed Early
If a smoke test is run as part of the build process, the system is being regularly checked for stability. It’s then easy for developers to know which changes ‘broke’ the system, and to find and fix the problem. This means it’s unlikely that the testing schedule will be delayed while the developers are debugging an error, and it also means developers waste less time in finding the source of the error.
Overall Testing Time is Reduced
A large percentage of errors will be caught by the smoke test, instead of by the testing team. This means there will be fewer delays, fewer bug reports, and less time wasted waiting for fixes before the tests can proceed.
Redundant Effort is Kept to a Minimum
Since everything must be retested after any bug fixes, testers can spend a lot of time running the same tests repeatedly. This problem is greatly reduced if major critical issues are fixed before testing even begins.
Job Satisfaction is Increased
Testers will be happier and less stressed if they can move forward with their work rather than being continually delayed by major system errors. Developers will also be less stressed if bugs can be fixed in time without pressure to meet fast-approaching deadlines. Happy employees generally do their jobs better, so this also impacts the efficiency of the project.
The Risk of Under Testing Due to Time Constraints is Reduced
If the testing schedule falls behind and the team is pressured to meet go-live targets, it’s tempting to skimp on the testing process. This, of course, results in customers being presented with a system that still contains bugs, which can cause huge problems. A good smoke testing strategy ensures that the testers have plenty of time to complete their job conscientiously.
The Bottom Line
Overall, the time spent in planning, implementing and running an effective smoke test is well worth it. The benefits in time, cost savings and personal satisfaction are enormous. Smoke testing should be a high-priority consideration in every development project.
Taking Control of Testing
While smoke testing ensures your system's readiness, flaky tests can still undermine your confidence in the build's stability. That’s why Trunk is building a tool to conquer flaky tests once and for all. You’ll get all of the features of the big guy's internal systems without the headache of managing it. You’ll be able to:
Autodetect the flaky tests in your build system
See them in a dashboard across all your repos
Quarantine tests with one click or automatically
Get detailed stats to target the root cause of the problem
Get reports weekly, nightly, or instantly sent right to email and Slack
Intelligently file tickets to the right engineer
If you’re interested in getting beta access, sign up here.
