Functional vs non-functional testing

The functional, non-functional and automatic sandwich maker test

A few days ago my children asked me (again) what exactly I was doing in my job. I talked to them about defects in applications, how to prevent them from appearing, types of tests...

...and while I was talking to two children about test typology while preparing some sandwiches, I came up with an analogy to differentiate functional tests from non-functional ones. Let's go there.

Imagine you're building an automatic sandwich maker to use at a birthday party with dozens of kids. And you want to make sure it's going to work properly because otherwise the next day the school's whatsapp group is going to put you on the line!

What does the machine do (i.e. Functional Tests)?

To ensure that your machine actually makes sandwiches and not hamburgers or pizzas, you need to do functional testing.

You should perform, for example:

• Bread and filling test: testing that the machine places the bread correctly and adds the ingredients in the correct order. It would be a disaster if the cheese ends up on the outside of the sandwich, especially if you end the party by having to clean the sandwich maker.

• Button test: making sure the "Make Sandwich" and "Stop" buttons work as they should. Imagine the scene if the "Stop" button throws the sandwich in your face instead of turning the machine off.

And how does it do it (i.e. non-functional testing)?

In this case, non-functional testing focuses on aspects not related to how the sandwich itself looks like but to the experience of use of the sandwich maker. Let's look at a few examples:

• Speed test: can the machine make a sandwich in 30 seconds or does it take longer than a Dragon Ball fight? If it is too slow, it may be preferable to make the sandwiches by hand.

• Can the machine make 1000 sandwiches one after another (load test)? Can it serve 100 children at a time (DDos attack!)? Overheating or crashing is not an option when you have dozens of hungry little critters staring you in the eye.

• Aesthetic test: if your machine is extremely ugly or smelly, even the bravest kids will run away looking for the potato chip one.

I am sure that as you have been reading this little story, you have taken this simile to the software development field and probably you have seen yourself in some similar situation ?

What are functional tests?

Functional tests are those that test the functionality of an application. They are mainly black box tests, so they do not go into the source code.

The functionality is reflected in the requirements, specifications, test cases, acceptance criteria, or in a backlog definition meeting. These can be defined by the customer, the product team, the CTO, etc.

From there it will probably end up becoming one or more tasks. These will be converted into a ticket, post-it or similar on a board with columns.

Once the functionality is implemented and deployed in an environment, this is when the test validates that the actual result matches the expected one.

Generally, these tests are performed when the software has already reached an advanced state of development and its components are incorporated with the rest, which allows detecting possible integration problems.

That would be an idyllic scenario.

But what problems do we face in the real world?

Vague or non-existent specifications or requirements.

No acceptance criteria

The test environment doesn't even remotely resemble the production environment.

But that's the subject of another post...

Non-functional testing

Regarding non-functional tests, things change a lot.

They are more, let's say, "bohemian" tests, as they are somewhat removed from the standards. They deal with aspects such as the end user's usage expectations, testing the how rather than the what.

But beware, this does not mean that they are less important than functional testing. They are complementary tests. And very critical.

Let's look at a clear example:

A functional test can be to validate that, after entering a correct username and password on an authentication page, you access to the home page. If this happens, the functional test is a success. And if it happens after 30 seconds of loading, it is still a functional success.

However, a non-functional test that checks that the authentication is done in less than 2 seconds would not have been satisfactory.

The problem arises when it comes to defining what criteria we will use to consider that the test passes successfully or is a failure.

It is easy to measure the loading time or how many concurrent users a system has, but it is not so easy to define the expected value. Why 2 seconds and not 1.8 seconds? or 0.3? Should the system support 20 concurrent users? why not 40? Is it easy to find the login button or access online support? How easy? How secure or vulnerable is my site?

These are concepts that are difficult to assess quantitatively but of vital importance to the user experience of an application.

Comparison table

With these definitions on the table, we can get an idea of the main differences between functional and non-functional testing.

Now let's look at some examples of each:

I hope this has been an enjoyable and interesting read, giving you a clearer idea of the differences between functional and non-functional testing. It's time to eat that sandwich now!