I remember when I first came across test automation tools; it was 1997 and I was lucky enough to work in a telecommunications company that had money to spend on one such product. I didn’t know quite how to use it or how to make the generated scripts run in a reliable manner without intervention– but it was COOL and on the occasions when I did have the right display settings on and the product was opened exactly the right way and the stars were all aligned, the tool worked and I sat back happily drinking a cup of coffee watching buttons being pressed automatically and feeling like I’d achieved something wonderful!
The thing was however that the scripts I wrote weren’t reliable and I had to make sure I was around when they were running as the slightest change would cause a failure. I guess I wasn’t all that good at writing robust scripts.
By today’s standards, the UI automation tools of 1997 weren’t all that smart but 20 years later we’re still putting in timeouts for our automation scripts (yes we are) and we’re pulling our hair out trying to get screen objects recognized by the tools. There’s a whole heap of value in UI-based automation but it is not and it never has been a silver bullet. There’s a huge global community helping us to do things better; we’ve now got testing design patterns and anti-patterns (I’d have been lost without Alister Scott’s blog posts) but UI-based automation is still a pain to maintain.
Personally, I think a big thing that UI-based automation tools have going for them is that they use common testing terminology so that as a tester I am able to drive them (eventually). A negative that I see (other than script brittleness) is that bad things may be happening underneath the UI; wrong values may being written to a database table; data may be getting corrupted but my script may still say that all is well because it’s not checking for anything other than a message on the UI.
Underneath the UI there may be layers of complexity and for me to say that I’ve tested an application, I need to understand its structure and its complexity from top to bottom. In terms of tool maturity and pricing, I don’t yet feel that the automated tooling available for Testers wanting to test underneath the UI is up to scratch. The tools that I’ve come across have been very Developer-centric and I’ve found it difficult to use those tools to build the kinds of tests that are important to me as a Tester and to the business as a whole.
I have had to become comfortable using an IDE and rolling up my sleeves to understand database transactions, file structures and communication standards as well as looking at code (something I’d avoided for a long time). This was not a nice-to-have for me; it was essential. Even though I had a lot of business knowledge, I had to cover a lot of ground before being able to show value testing underneath the UI.
As I’ve been testing more and more APIs, I’ve seen that the steps to adding business value are basically the same steps but they take a lot of time. So, if you look at testing as being made up of three stages Crawl, Walk and Run then the Crawl and Walk stages take the longest time but they have to be completed before we can get to the valuable bits that the business is interested in – Run.
So, we’ve taken on the challenge to build a product that simplifies API testing by automating Crawl and Walk enabling Testers to configure and Run. Watch this space.
In the meantime, let us help with some static analysis of your API definition file. If you pass a poor Swagger file to a test team then you’re wasting time; reduce the feedback loop and get this right first. You can use our Swagger / OAS validator for free, just click here:
You can help! Please share this tool by clicking here:
Within seconds you will have a list of errors, warnings and useful suggestions for your dev team to work on to ensure your definition file is not just compliant but solid. This is the first step to RESTful API happiness. Please use the tool regularly and please share this useful tool with your network and let us know what your thoughts are.