0%

If you have read the Swiss Cheese Model section and are familiar with agile, you might have noticed something missing. In fact, if you read many of my posts or have worked with me, you will probably notice I rarely refer to the pyramid despite its ubiquity within the agile world.

The pyramid model is just that, a model. Models are useful for predicting and acting as a guide for decisions, but all models have limitations and understanding their limitations helps you make better decisions and understand when you need a new model.

I personally feel the pyramid is at its best in particular conversations – specifically, the cost vs. effect conversation.

If you look at the standard model:

Standard Test Pyramid

The test pyramid has a clear concern – certain test types cost more than others due to a combination of the length of the feedback cycle, stability, maintenance, brittleness, etc – so you should have more of the inexpensive tests and less of the expensive ones.

This focus makes the pyramid an effective tool for asking the question of where a test should be (the place which gives it the best cost v. effect ratio). This is a good thing to keep in mind and use within your own strategy building, but there is more to a strategy than its cost and feedback cycle.

The pyramid itself is a good rule of thumb for what was the standard project when it was created (websites with backends and maybe an outside dependency or two). However, the world of IT has moved in very different directions since its inception. Many projects exist in worlds far beyond the single website world (eg, microservices). The pyramid is an effective tool for some conversations, but the greater strategy for these projects needs a new model.

Similarly, many project similarly utilize tools which are the great tools for their use case, but those tools do not support unit tests which fit the pyramid (eg, data transformations in Apache Spark). While ideally we pick tools which support unit testing, this is not always possible and we should be able to adapt to the situation. These projects also need a model beyond the test pyramid.

The testing realm of IT needs a flexible model which can handle the fact projects are not simple and tooling is not always ideal. We can keep the principles of the test pyramid (cost effectiveness and quick feedback) close while understanding there is more to a quality strategy.

What makes a good error message?

A. Informs the user what went wrong
B. Gives the user technical information
C. Assures the user things will be looked into
D. Tells the user what to do?

Think about it before looking at the answer. Who it is for? What they are meant to do with the information? Why they might encounter these messages?

Think about your own experiences with things you use – what do you with error messages? What do you wish you had been told in the error message?

If all of this led to you guessing D, congratulations!

None of the other error messages are helpful to the user. A and B can be harmful to the application as they can lead to security risks due to the possibility of exposing information which could be exploited. C can be misleading. Only D gives the user actions for them to do.

Sometimes the only action you can tell them is to try again or contact customer service and that is okay! The point behind these messages is to empower the user. If the error is something they can fix, they should know that. If the error is something they cannot fix, they should know that too!

Whenever you encounter an error message, you should always ask yourself – “What is this telling me?” and “Is this enough information for an average user to make a reasonable action?”

Error messages are an important part of the user experience, whether you are working with a GUI, API, or some other form of customer interaction.

USER JOURNEY FLOWS

User flows give the team a big picture idea of what they are working towards – how the API will fit into a client program. Even APIs which will be consumed by multiple clients can benefit from understanding these flows.

These are models of how we would expect the user to use the system, screen by screen. The screens don’t need to be high fidelity (eg, a black box labeled “Login Screen” can easily suffice for a login), just a visual representation of where a particular screen would be. Connecting the screens would be transitions where any called services would be pointed out. You do not have to map all the user flows, but having your critical flows mapped can make following conversations easier.

For example, a flow involving logging in would only call out the happy path login. We would have sad paths in a separate documentation unless the sad path is complicated and important enough to warrant drawing a flow around it.

These can be used to aid priority sessions (eg, which flows are highest priority of enablement), determine what is and isn’t out of scope for that particular discussion, and help with sharing context to new team members. This section will be referenced in later sections and I personally believe this is the most important suggestion.

DOCUMENTATION

In an API project, your primary customers are the teams consuming the API. Ideally, you would treat your documentation as your User Interface and as such, it should get just as much attention and testing as everything else.

SWAGGER

The swagger documentation tool enjoys a high popularity. It results in pretty documentation which does come from the code, but has similar problems which comments do as far as getting out of date. Simply generating documentation from code just gives objects and the docs still need to be extended by hand. Additionally the docs can still be inaccurate.

IDEAL CASE

The ideal for documentation is a way to auto generate similar to swagger, but be connected to tests. The documentation which results will still need some editing by hand to truly be useful, but coming from tests increases confidence in the docs.

HANDLING DEPENDENCIES

DOWNSTREAM TEAM

Most API projects will be paired with a team building some sort of client, typically mobile. Sometimes this team only exists in theory, other times they will be already working, and other times they will be a project which starts up at the same time.

Best case scenario, you will be able to work closely together to define things like the contract, can test against each other easily, and have a good working relationship.

This part of the document would be completely unnecessary if the best case scenario ever happened. If you are on a unicorn project, skip this section and bravo to you.

Chances are pretty high an upstream team will lag behind, is either not co-located or you cannot work closely with them for some other reason, and has a slightly different understanding of various parts from your team. This is understandable and gives the API team both an edge in influencing the upstream team and a concern about how things will look once the upstream team finishes.

The top recommendation for working with an upstream team would be to develop some full stack user flows and integration tests which can run on every check in should partially alleviate concerns about interoperability. Getting a set of pact tests going where the two teams agree on the contract and set up independent tests to make sure both teams are obeying the contract at the same time will help keep down integration problems.

Ultimately, understand the upstream team is your first and primary customer. They need to be able to use the API in an efficient manner which fits the flows they are enabling. When deciding on story priority order, it helps to understand (preferably using user flows) what features they are enabling next and what they will need from the API team.

UPSTREAM DEPENDENCIES

Most API projects will also deal with one, if not multiple dependencies. These dependencies could have volatile test environments, have unexpected ways of responding, or depending on the project, these dependencies could even have work going on them while you are trying to develop against them!

In these instances, many of the above suggestions will still be helpful, but there are additional things you can do depending on which issues you are facing.

DEPENDENCIES UNDER DEVELOPMENT

These dependencies will have a team actively working on them for various reasons. This team is hopefully an ally of some kind and will respond to requests, but even if the team isn’t actively helping yours, there are steps you can take.

Contract Tests are the big deal here. These tests simply check that the contract of the call you are using downstream has not changed. Ideally these tests would run every time the other team changes something, but if you do not have insight into that (which is normal), you can figure out approximately how often the other team is expected to push changes and code based on that. Past projects have used every 2 hours during working hours, just once a day, and just once a week after the designated weekly “push time” .

UNRELIABLE DEPENDENCIES

The generic good practice is to setup some sort of health check monitoring which is capable of notifying the appropriate people when the dependency is down. Hopefully your team and whoever is supporting the dependency are able to work out a way to make it more reliable in the future.

PERFORMANCE

APIs are unique in that the lack of GUI does make it easier to run reliable performance tests against them. Most good practices from general software development still apply here (eg, have a separate environment, use the same data set for repeat runs for reliable results) with the addendum that if you can procure an environment just for testing, performance testing can absolutely be part of the pipeline.

Even if you cannot procure such an environment, you can still use throughput tests against the same environment you use for other automated tests. These throughput tests simply notify if the time it takes to make a call has significantly changed in some way.

Paired with monitoring metrics in production, you can build a reasonable, layered approach to watching the performance of your project.

EXTERNAL TESTING TEAMS

The biggest thing to remember with other teams working on the same project is everyone is on the same side. We all succeed if the project succeeds. It is really easy to get caught in the trap of acting like other teams are an enemy of some kind and develop a toxic relationship.

TRIAGE MEETINGS

They suck. If you are capable of replacing them with something which works better, I want you to teach me your ways.

These serve a purpose where most API projects wind up dealing with an external testing team who find defects and are not sure where the defects should go. The goal for this meeting, as in all meetings, is to be done with as accurate results as possible.

On previous projects, I have heavily encouraged the testing teams to contact the API teams about defects which could potentially be from our side so we can investigate them prior to the meeting – if we are lucky, we can narrow down how many defects need to be discussed this way and limit the meeting to defects which are tricky, have business repercussions (eg, solving it would result in a change or the bug itself is the result of an unexpected scenario which should be handled) or are otherwise not easily dealt with.

USEFUL TOOLS OF NOTE

POSTMAN

Useful for manual testing, sharing collections across the team, and the team behind it is adding many features.

Can attach a theoretical postman script of what should work to a story under development or to a defect

REST ASSURED

Useful for automated tests in Java due to its DSL.

On the surface, testing an API is similar to testing of other products – user flows, security, edge case behavior, etc, but different in that the first consumer of the API is a developer rather than a customer using a GUI of an app.

For this reason, part of the focus is on the dev experience – can a developer debug the problem with their code’s interaction with the API using the API error messages? Can they easily program their code to interpret errors? Can they use the documentation to figure out how to use the API for their purpose?

A lot of this is coverable by writing user flows for the API. By going through the experience of using it yourself, you can understand what the users will need to go through in order to accomplish their goals. Likewise, when it comes to debugging your user flows, always be looking for where the API is falling short on helping you. For example, when debugging the user flows, do the errors tell you when you are missing fields? Are you getting consistent error messages? Can you program your user flows against a consistent message format?

APIs are unique in that, unless it is asynchronous, they are easily covered by full stack automated tests. It is much easier to put a suite of maintainable tests around them than it is to put a similar suite around the end product (especially if that product is a mobile app). As a result, creating a large, robust suite of user flow tests (tests which attempt to accomplish goals a user of the end application would have) is a very good thing! You still want to make sure your tests give quick feedback and are reliable, but most APIs can have larger user flow test suites than UIs – whose user flow test suites can become flaky and slow.

Do you need to know how to code to test an API?

Not completely, but it helps. Similar to how knowing what happens during air travel can inform testing an app intended for a traveler, understanding what a dev goes through helps with testing a tool intended for developers. I would highly recommend a QA write the user flows themselves, pairing with developers as desired, because the debugging process for the user flows IS part of testing the API.

Writing and maintaining the suite exposes the team to the same pain as consumers of their API will feel, specifically in the user flow portion of the test suite.

To begin your journey through quality, you need to first setup your machine. This post assumes you are looking at a full stack website.

OS

To start with, you need to have the OS itself ready to go. Ideally your base for testing supports all the tooling you need. If you need to test things in a specific OS, VMs are generally good enough. These days, companies like Microsoft even offer free VMs for this purpose.

BROWSERS

Your primary browser ought to be one which matches your target users and all browsers your team plans to support should be thoroughly tested. However, even if you are not supporting them, other browsers ought not reveal or allow anything untoward.

For example, you will want to use the Lynx browser to see what happens in an all text environment.

You will also want to ensure one of your normal browsers is easy to turn off Javascript, Flash, etc. Likewise, moving security settings from nothing to most secure.

BROWSER FEATURES

Most of the major browsers come with useful features for testing. Ensure your primary browser has the following features either built in or accessible through add ons and be comfortable accessing them, later posts will go in depth with ways to use them

View page source
Manipulate page source
Javascript console
Network calls with response headers & bodies, request headers & bodies, and timing
Local storage & cookies viewing and manipulation
Mobile emulation
Ability to change user agent

BROWSER ADD ONS

There a number of useful addons out there which will make testing far easier. Here’s a list of ones I commonly use:

Link Redirect Trace is a useful extension showing you all the hops through a redirect loop
Wave Evaluation Tool is a quick way to analyze the accessibility of your website

OTHER USEFUL TOOLS

Additionally, tools outside the browser can greatly expand your capabilities

Burp Suite has a number of useful tools, allowing you to do things like crawl a website or use its proxy abilities to modify incoming and outgoing requests
Postman is very useful as you can build up a collection of requests and use its variable and environment capabilities to easily work directly with APIs
VM Box for testing functionality in other OSs.
Any document for tracking what you are doing. Specifically, if there is something repetitive you cannot automate, writing it down is a good thing to do.

WHAT IS THE SWISS CHEESE MODEL?

Swiss Cheese Model

A multi-layered approach to ensuring the quality of a project, inspired by the Swiss cheese model of accident prevention used in aviation safety, engineering, and many other industries.

Just as swiss cheese has holes, so to will any layer you introduce into a software development strategy. For example, a unit test suite can be very effective at testing the functionality of individual methods, but if you integrate with outside systems, unit tests alone fall short of providing full coverage.

Furthermore, using a multi-layered approach enables you to ask if it is reasonable for a layer to handle a particular event. Thinking about UI tests, they can be capable of testing many features, however, the slow and more costly nature of UI tests means it is not effective to have a large number of them.

The final important note is a quality strategy is about more than just your automated tests. It is how you approach a problem, the analysis you do, the metrics you put in place to ensure what you are doing accomplishes the goal.

HOW DO I USE IT?

In order to exemplify how to use this model, we are going to go through the process for a generic airline API. In this example, our API will have three consumers (the website, the iOS app, and the android app) and two dependencies (a service which provides flight information and a database of customer information)

Understand Who Cares

First, in order to know what your layers should be, you need to know what your concerns will be. Start with identifying key players – aka who cares about the project – and identify the concerns they have.

Here is an example for a generic airline app API:
Airline API Mindmap

For this exercise, it is important to note everyone and everything the app effects, regardless of their importance to the project and whether they are an intended actor (eg, end customers), negative actor (eg, malicious users), or indirect actor (business). Prioritization can come later, but it is important this exercise is an unfiltered list.

As a result, we not only identify the end customers and business, but we also call out the team who has to consume it, the consuming apps, our own team who needs to support it, and malicious users who will want to exploit it.

Understanding the various people who care about what we are doing enables us to make a better strategy, but also enables us to more effectively look at these when weighing priorities.

For example, even if malicious users are not usually considered in discussions around business needs, we still need to evaluate the risks associated with them – does our app handle sensitive information? Does our app give malicious users another door to get into the system with? Does our app make it easier for our users to fall prey to social hacking? If the answer to any of these is yes, the security the API must put into place needs to be weighed more heavily.

We need to keep our immediate consumers in mind, not just the end users, as the immediate consumers are the ones interacting with our API. Knowing what they would use the API for and what the needs of the programs they are writing can guide our ability to write tests and evaluate the responses appropriately.

Understanding what the business seeks to gain from the API helps us understand whether we are even building the right thing. For example, if our API is difficult to extend even though it is otherwise wonderful to use and incredibly reliable, it would fail the business need (in this example) of responding to the market quickly.

The teams building and maintaining the API should be called out as their interactions with the API matter. If this team isn’t able to effectively work with their own project, even if it is wonderful to the outside world, there is a large risk due to maintenance concerns as well as future security risks.

Finally, we want to make certain to call out any usage inside the business outside the normal feature -> customer relationship such as data analytics teams. Understanding these teams needs can help shape what sort of logs we are making. Integrating thinking about them from the very beginning makes for a more effective strategy.

Understand Architecture

The next step is to take a look at your intended architecture. Even if the specifics might change, this step is for highlighting things such as integration points and dependencies. Additionally, as you learn more about the architecture, you can use the information to update your strategy based on the changes from the initial picture.
Airline API Architecture

The proposed architecture this fictional team has at this point in the project (the very beginning), involves a DB for any information the API needs to keep track of (eg, authentication information) which is independent of its dependencies.

Once you have this drawing, you should think about the lines between them.

For example, the two apps will likely usually be contacting the API on unsecured and potentially unreliable networks. The two servers in our scenario are both in house in the same system as the API itself. The website is more likely to be used from home and hotel computers – still high potential for insecure connections. These are all factors which should be considered for the strategy.

Match Concerns to Layers

Once you have gathered concerns from 2 areas – the people/things who care and the architecture – it is time to create layers and decide what layer is intended to handle which concerns.

Sometimes a concern might need multiple layers – if you can break down the concern in a way which shows why, this helps with better understanding the concern. Eg, one concern noted above is around the malicious users who will want to exploit the system. We can break this down into at least two smaller concerns: wanting confidence hacking the system will be difficult (via penetration testing) and wanting to ensure developers do not introduce common code vulnerabilities (via static code analysis).

Similarly, a layer can also handle multiple concerns. It is a good idea to note when this is the case for if in the future a concern is moved to a different layer, but the other concerns are not able to move.

Noting what layers cover which concerns also makes it easier to know when to retire a layer. If you introduce a new layer which takes on concerns from others and accomplishes them in a better way, the old ones should be retired. The intent for a strategy is to be the most cost (time & resources) effective while giving the business and team confidence in the product they are building.

Some example concern-to-layer mappings:

Customer experience -> product analysis
Delivery team confidence around changing code -> Unit tests
Ease of understanding old work when on new features -> Unit tests
Difficult to hack system -> Penetration testing
Stop introducing vulnerabilities to the code -> Static code analysis

THINGS TO BE AWARE OF

Layers cost money. Everything has a price. In the above example, penetration testing can be an expensive timely affair and depending on the working relationship with the penetration testers, will likely involve a lot of work on the dev team’s part to ensure their work is done within certain parameters and time to make the best use of the penetration testers’ time.

Ultimately a layer needs to always be worth its cost. If you are introducing a layer which only protects you from something minor whose risk, cost to fix, and impact on business are all low, this might not be a layer worth introducing. Every layer should be preventing something impactful enough to justify the cost of introducing and maintaining it.

HOW DO YOU KNOW IT IS WORKING?

To answer this question, there is a small modification we need to make to the image of the swiss cheese model introduced above.

Swiss Cheese Model with bowl at bottom to catch the issues

There is no such thing as perfection. There can be flaws in every step of the process or simply a shift in the market leading to a product not performing to perfection. As a result, we cannot measure our strategy’s success simply by bugs or some similar metric. Instead, our metric should focus on the fallout when there is a problem.

Many companies have some sort of Customer Success team, who are the first line when customers need help or have feedback. This team, or whatever your group uses as its first line, represents the bowl at the bottom – they are who deals with what falls through, whether it be a confusing customer experience, a bug interfering with customers, or market fall out due to an exposed security vulnerability.

Your strategy succeeds so long as that team is succeeding. If they are falling behind or otherwise overwhelmed (eg, catastrophic bug happens), then you are not succeeding and should examine your process with what is happening to see what can be improved with the information from Customer Success.