Automated software testing is becoming more and more important for many software projects in order to automatically verify key functionality, test for regressions and help teams run a large number of tests in a short period of time. Many teams (especially larger projects) still require a significant amount of manual functional testing in addition to automated testing, either because of the lack of sufficient resources or skills to automate all tests.
There are various tools that help software teams build and execute automated tests. Many teams are actively using unit tests as part of their development efforts to verify critical parts of their projects such as libraries, models and methods. Historically, testing user interfaces of desktop-based applications via automated tests have been more challenging, and currently available tools for this are usually commercial and quite expensive.
With the growing number of web-based applications this is changing, however, as verifying and testing web-based interfaces is easier and there are various tools that help with this, including free open source projects. Please see below for a list of popular and useful tools, projects, books and resources to get started with automated software testing.
Automatically testing your web application is a good way to ensure that new versions of your application don't introduce bugs and regressions. Automation of your web application testing also allows your development team to make changes and refactor code with more confident, as they can quickly verify the functionality of the application after every change.
However, actually building automated tests for web applications can be challenging because the user interface of your application might change regularly, because of incompatibilities between browsers and because you usually need to support various server or client platforms. The following tools make it easier to build and execute automated tests for your web application.
Selenium is a popular automated web testing tool and helps you automate web browsers across different platforms. Selenium has the support of some of the largest browser vendors who have taken steps to make Selenium a native part of their browser.
Watir is a set of Ruby libraries for automating web browsers and allows you to write tests that are easy to read and maintain. Watir drives browsers the same way people do (it clicks links, fills in forms, presses buttons etc.) and also checks results such as whether expected text appears on the page.
Windmill is a web testing tool designed to help testers automate and debug web applications. It comes with a cross-browser test recorder, JavaScript integration and an interactive shell to automate web browsers.
Ranorex allows you to automate your web application testing (among other things) and both record user interactions and play them back to execute your tests. Ranorex is one of the more popular commercial tools to build and run automated web and GUI tests.
SoapUI is a cross-platform functional testing tool. It has been specifically designed to help automatically test APIs such as SOAP and REST interfaces to ensure the interoperability of different applications.
Sahi is a tool for automation of web application testing. Sahi is available as a free open source edition as well as a commercial Pro edition.
Tellurium is a web automation tool that allows you to design and write your automated tests using plain English without any scripting or programming experience.
Building robust automated GUI tests for desktop applications (e.g. on Windows or Mac systems) is quite difficult, as small changes to the user interface can often result in broken tests. The following tools help you build and execute robust GUI tests for various platforms and operating systems.
Squish is a GUI testing tool for various platforms, including QT, native Windows and Mac applications. Squish allows testers and developers to build automated tests using familiar scripting languages such as JavaScript, Perl, Python and Ruby.
Ranorex allows you to automate your desktop applications (among other things) and both record user interactions and play them back to execute your tests. Ranorex is one of the more popular commercial tools to build and run automated GUI and web tests.
TestComplete is an automated testing tool for the Windows platform. It allows you to record, script and run GUI tests for applications built using different frameworks and languages, such as .NET or C++.
Test Studio is an automated functional and load testing tool that helps you test applications on various platforms built using different frameworks and tools.
eggPlant is a GUI test automation tool for professional software applications and enterprise teams. It can be used to automate different application types, such as .NET, Java and Flash applications.
See below for a list of popular unit testing frameworks and tools for major platforms and programming languages. These frameworks can be used by programmers to test specific functionality in libraries and applications. Unit tests can then be used to automatically test new versions and builds as part of an automated build system or deployment process.
NUnit is a unit-testing framework for all .Net languages. It was initially ported from JUnut to .NET and has been redesigned to take advantage of many .NET language features.
xUnit.net is a community-focused unit testing tool for the .NET Framework written by the original inventor of NUnit. xUnit.net is the latest technology for unit testing C#, F#, VB.NET and other .NET languages.
The Python unit testing framework, sometimes referred to as "PyUnit", is a Python language version of JUnit. It's part of the Python framework and supports test automation, sharing of setup and shutdown code for tests and various other features.
JUnit is a simple unit testing framework to write repeatable tests in Java. JUnit has been important in the development of test-driven development and is one of the standard testing frameworks for Java developers.
TestNG is a Java testing framework inspired by JUnit and NUnit and introduces some new functionalities that make it more powerful and easier to use. TestNG is designed to cover multiple categories of tests, including unit, functional, end-to-end and integration tests.
PHPUnit is a popular framework for unit testing in PHP projects. It provides both a framework that makes it easier to write tests as well as the functionality to easily run, execute and analyze tests and results.
Lime is a unit and functional testing framework built for the popular Symfony PHP web application framework. The framework is designed to have readable output from tests, including color formatting, by following the Test Anything Protocol.
Ruby comes with its own standard unit testing framework as part of the Test::Unit namespace and can be used to define basic pass/fail tests and group tests. The framework also comes with tools to run single or whole groups of tests.
RSpec is a testing tool for the Ruby programming language. Born under the banner of behaviour-driven development, it is designed to make test-driven development more productive and enjoyable.
Cloud testing services allow you to run your automated web, mobile and unit tests in different environments and on multiple machines without having to build your own testing infrastructure. The below list of services provides a good overview of popular online Selenium and continuous integration services that are easy and affordable to use.
| Service | Type | Description | Pricing |
|---|---|---|---|
| Sauce Labs | Selenium & Mobile | Cross-browser Selenium & mobile testing | Starting @ $149/month |
| TestingBot | Selenium | Cross-browser Selenium testing | Starting @ $40/month |
| Gridlastic | Selenium | Cross-browser Selenium testing | Starting @ Free |
| CircleCI | Continuous Integration | Continuous integration service & unit testing | Starting @ $19/month |
| Tddium | Continuous Integration | Continuous integration service & unit testing | Starting @ $15/month |
| CloudBees | Continuous Integration | Continuous integration service & unit testing | Starting @ Free |
| Mailosaur | Email Testing | Service to integrate emails in automated tests | Starting @ Free |
The following books provide a good starting point to learn more about automated software testing in general, test-driven and behavior-driven development as well as integrating automated tests into your overall testing strategy.
Quite simply, test-driven development is meant to eliminate fear in application development. While some fear is healthy (often viewed as a conscience that tells programmers to "be careful!"), the author believes that byproducts of fear include tentative, grumpy, and uncommunicative programmers who are unable to absorb constructive criticism.
Selenium Testing Tools Cookbook is an incremental guide that will help you learn and use advanced features of Selenium WebDriver API in various situations for building reliable test automation. You will learn how to effectively use features of Selenium using simple and detailed examples. This book will also teach you best practices, design patterns, and how to extend Selenium.
This book describes how to build and implement an automated testing regime for software development. It presents a detailed account of the principles of automated testing, practical techniques for designing a good automated testing regime, and advice on choosing and applying off-the-shelf testing tools to specific needs. This sound and practical introduction to automated testing comes from two authors well known for their seminars, consultancy and training in the field.
Just Enough Test Automation shows test developers and users how to design, implement, and manage software test automation. Learn from authors Dan Mosley and Bruce Posey how to implement a powerful data-driven testing framework; automate unit testing, integrate testing and system/regression testing; and facilitate manual testing with automated tools.
Authors Dorothy Graham and Mark Fewster wrote the field's seminal text, Software Test Automation, which has guided many organizations toward success. Now, in Experiences of Test Automation, they reveal test automation at work in a wide spectrum of organizations and projects, from complex government systems to medical devices, SAP business process development to Android mobile apps and cloud migrations.
With Acceptance Test-Driven Development (ATDD), business customers, testers, and developers can collaborate to produce testable requirements that help them build higher quality software more rapidly. However, ATDD is still widely misunderstood by many practitioners. ATDD by Example is the first practical, entry-level, hands-on guide to implementing and successfully applying it.
In addition to the above mentioned books, there's a huge number of online resources to help you get started with test automation and test-driven development. We researched the best articles and online resources and please see the list of links below.
Software Testing has found its place in the software industry, with more and more organizations understanding the crucial role that it plays in quality software production. This article explains test automation strategies and approaches for software teams.
This presentation explains how eBay's quality engineering team uses automated testing in different projects and within different development methodologies (such as agile, waterfall, distributed etc.) and the lessons they learned.
This is a very useful discussion about the (possible) drawbacks and disadvantages of automated testing and covers areas such as programmer productivity, tools cost, skill level, organizational challenges and more.
This is an in-depth article about test-driven development (TDD), explains the pros and cons of TDD and contains additional links to related tools and resources. The article also tries to explain and clarify typical misconceptions of TDD and explains why and when teams should adapt test-driven development for their projects.
This article tries to explain when (and when not) to automate your software tests and provides a useful list of criteria. The article also discusses the need to spend testing efforts wisely to find the most important software defects.
In software testing, test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes to predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or add additional testing that would be difficult to perform manually.