How did QA miss this?

A lot happens between writing code and spending hours testing it, and sometimes the most frustrating bugs might have crept in elsewhere. This blog post offers a bird’s eye view of the different things that typically go wrong, and a few useful insights which improve QA processes for better outcomes. When faced with the question “How did QA miss this?”, shift the focus to collective responsibility: “How did we miss this?” and investigate the timing and context of the issue. Whether it’s related to browser versions, hosting services, server OS, software dependencies, outdated content management systems, or database errors… it’s a multifaceted problem and not just a QA engineer oversight.

The Collective Responsibility in Software Development

If a bug slips through, it’s usually not the fault of a single person or a team. Software development is a shared endeavor with multiple business stakeholders: developers writing code, project managers managing timelines and schedules, testers testing. Everyone has a part to play in determining the quality of the final outcome.

Breakdowns in communication are at the root of many project failures – usually because the dependencies and expectations among team members haven’t been clearly thought through.

Timing and Context of Issues

Bug detection depends on timing and context: what are the conditions when a bug appears that are different from when the software was tested? For example, a feature might work perfectly in the test environment, and be deployed at scale, but break due to real-world variables. By providing good test documentation a QA engineer shows how to reproduce the conditions that caused the bug to appear.

Browser Versions and Compatibility

Websites are intended to be viewed in browsers, and browsers have their own quirks and what works well in Chrome, may break in Firefox or Safari.

Cross-browser testing is crucial for minimizing this risk and tools like BrowserStack and Sauce Labs simulate a bunch of different browser environments. Testing based on website analytics data of your top browsers and platforms helps ensure the best user experience on the browsers and platforms that matter most. Even with this level of proactive testing, keep in mind that browsers like Chrome update 3 or more times a month and these updates risk breaking your website since it was last tested.

Hosting Services and Server OS

Quirks in timing and availability of server resources, or network conditions, typically lead to bugs that weren’t present earlier in development and testing. It’s important that the production environment is replicated in testing. Staging servers with the exact production setup help to identify issues related to the hosting environment, and monitoring server performance and logs typically show signs of trouble early on, before more time is spent building features.

Software Dependencies and Versioning

Just about any software package you depend on today consists of other software packages you didn’t write – libraries and frameworks whose maintenance and development you have no control over. While these dependencies help speed up development time, they also constitute new potential points of failure: a subtle code change in a fourth-party library or an accidental bug in a fifth-party dependency may cause your production application to break.

To mitigate this risk, track all dependencies and their versions in a spreadsheet. There are a few automated tools, such as GitHub’s Dependabot, that checks dependencies from time to time for updates. Checking and updating dependencies helps avoid many of these version-related problems.

Outdated Content Management Systems

Many web applications are built around Content Management Systems (CMS) like WordPress, Drupal, Sitecore, or Umbraco. If you’re using an older version of your CMS, you leave yourself open to security vulnerabilities and differences in compatibility. It might not handle features or events the way you want to, because those features or events were deprecated in older versions of the CMS.

Keeping the CMS and its plugins up to date is crucial to maintain compatibility with other systems. Schedule regular maintenance and update intervals to make sure that the system is running on the most current version, and stable for QA engineer testing. Perform testing in a staging environment before rolling out the updates to the production system.

Just last week a plugin on this website was updated, only to cause a fatal error with the version of PHP on our hosting server, so I updated to a later version of PHP. The latest version of PHP caused another conflict with the CMS theme, and had to be rolled back to the previous PHP version… for further debugging, and testing.

Database Errors and Migrations

Databases are another common culprit of bugs, especially where migrations or updates are concerned. Changing the schema of a database, corrupt data or indexing issues risks surprising the application with an error it’s not really prepared to handle.

Automated migration scripts (installed and run on schedule) are better than most manual updates, from one schema or version to another. Scheduled database backups and site monitoring typically head-off most problems – and even solves them quickly in most cases by rolling back to the backup if issues are found by the QA engineer.

Effective Issue Tracking and Reporting

Every modern software project needs a powerful issue tracking system for QA – ideally something like DoneDone, Jira, Bugzilla, or Trello – to make sure issues get captured, prioritized, and tracked all the way from review until all stakeholders are satisfied.

A high-quality issue report should have steps to reproduce along with screenshots or videos and anything else to help navigate the issue (eg, logs, error messages). Good communication from the QA engineer and proper documentation go a long way to reduce the resolution time.

The Role of User Feedback

Users also contribute immensely: software that’s live shows us things that have worked (or haven’t) for the end-user. Questionnaires, forums and user feedback gives a heads-up on usability issues that wouldn’t have occurred during development and testing.

Engage with the users both proactively and reactively, and always be ready to listen. We benefit more by understanding the user’s expectations. This is essential for building a good product with satisfied users.

Tools that capture how people use websites and generate heatmaps, session recordings, and insights are important. Clarity is a free tool that consolidates user behavior and uncovers trends, to make websites better and is ideal for retail, content publishers, small business owners, and everywhere in between.

Continuous Learning and Improvement

Software development is an ever changing landscape, with constant evolution of tools, techniques and “best practices”, regular education, training and the sharing of knowledge within the project team leads to constant learning and betterment. Experimentation, and innovation, are encouraged.

Building a Strong Team Culture

At the team level one of the keys to performance is strong culture. A sense of shared vision and values, and an environment where people feel that they are listened to and properly acknowledged for their contribution, are important sources of a team’s strength.

Ensuring that people open up and communicate, celebrating successes and learning from failures motivates teams to remain focused, keeping them engaged. A good team culture creates a sense of responsibility commitment to ensure that the software is of the highest quality.

The Importance of Documentation

Detailed documentation is key to preserving the integrity and reliability of the software because it effectively and efficiently communicates information about code, processes and decisions to team members and others. This enables new team members to quickly get up to speed and ensures continuity in case of turnover.

Regularly updating documentation that reflects changes in the software and processes helps to prevent most problems. Giving clear and accessible documentations also leads team members work in a better way by understanding each other’s work.

Conclusion

Sometimes I feel that developers and QA engineers make themselves an easy target to critique. Hopefully someone out there will read this and remember that we’re all working together for the best of it.

If you’re looking to take the software development process up a notch, try these strategies to see the difference they make. Let’s learn to create software together, and get it working in a way that benefits users.