I've fuzzed a small part of the GitHub API. Here are my findings.

Disclaimer

!!! WARNING !!! If you choose to run the steps in this article, please note that you will end up with a significant number of dummy GitHub repos under your username. Over 1k in my case. There is a script at the end of the article that you can use to delete them. Be careful not to delete your real repos though!

Repos

TLDR

This is a concrete example on how you can use CATS and a personal view on how I would improve the GitHub API. Don’t expect critical bugs/issues. The findings are small things that will make the GitHub API more usable, predictable and consistent.

The Beginning

Building good APIs is hard. There are plenty of resources out there with plenty of good advice on how to achieve this. While some things are a must, like following the OWASP REST Security Practices, some others might be debatable, based on preference, “like using snake_case or camelCase for naming JSON objects”. As the number of APIs usually grows significantly, even when dealing with simple systems, it’s very important to have quick ways to make sure the APIs are meeting good practices consistently.

I’ve shown in a previous article how easy it is to use a tool like CATS to quickly verify OpenAPI endpoints while covering a significant number of tests cases. But that was a purely didactic showcase using the OpenAPI demo petstore app. Which was obviously not built as a production ready service. Today I’ll pick a real-life API, specifically the GitHub API, which recently published their OpenAPI specs.

I’ve downloaded the 2.22 version and saved the file locally as github.yml. Before we start, we need to create an access token in order to be able to call the APIs. Make sure it has proper scopes for repo creation (and deletion when using the script at the end of the article). Also, as the API is quite rich (the file has 59k lines), I’ve only selected the /user/repos path for this showcase. You’ll see that there are plenty of findings only using this endpoint.

You can run CATS as a blackbox testing tool and incrementally add minimal bits of context until you end up with consistent issues or a green suite.

As shown in the previous article, running CATS is quite simple:

cats --contract=github.yml --server=https://api.github.com --paths="/user/repos" --headers=headers_github.yml

With the headers_github.yml having the following content:

all:
  Authorization: token XXXXXXXXXXXXX

Let’s see what we get on a first run:

First Run

We have 42 warnings and 156 errors. Let’s go through the errors first. Looking at the result of Test 118 we see that a request failed due to the name of the repository not being unique. Indeed, CATS, for each Fuzzer, preserves an initial payload that will be used to fuzz each of the request fields. This means that we need a way to force CATS to send unique names for the name field. Noted!

Test 118

Test 426 says that If you specify visibility or affiliation, you cannot specify type.. Let’s note this down also.

Test 426

Considering the above 2 problems are reported consistently, let’s give it another go with a Reference Data File. This is the refData_github.yml file that will be used:

/user/repos:
  name: "T(org.apache.commons.lang3.RandomStringUtils).random(5,true,true)"
  type: "cats_remove_field"

CATS supports dynamic values in properties values via the Spring Expression Language. Using the above refData file, CATS will now generate a new random name everytime it will execute a request to the GitHub API. Also, using the cats_remove_field value, CATS will remove this field from all requests before sending them to the endpoint. More details on this feature here.

Running CATS again:

cats --contract=github.yml --server=https://api.github.com --paths="/user/repos" --headers=headers_github.yml --refData=refData_github.yml

We now get 17 warnings and 90 errors. Again, looking through the tests failures/warnings there are some tests which are failing due to the fact the since and before are not sent in ISO 8601 timestamp format (more on this inconsistency in the Findings section).

Second Run

We’ll now update the refData file to look as follows:

/user/repos:
  before: "T(java.time.Instant).now().toString()"
  since: "T(java.time.Instant).now().minusSeconds(86400).toString()"
  name: "T(org.apache.commons.lang3.RandomStringUtils).random(5,true,false)"
  type: "cats_remove_field"

and run CATS again:

cats --contract=github.yml --server=https://api.github.com --paths="/user/repos" --headers=headers_github.yml --refData=refData_github.yml

Third Run

We now get 5 warnings and 83 errors. Looking through the errors and warnings, there is a significant amount which I consider legit issues, while some are debatable points, depending on preference/standards being followed. Let’s go through the findings.

Findings

Invalid values for boolean fields implicitly converted to false

One of the Fuzzers that CATS has is the RandomStringsInBooleanFieldsFuzzer. This Fuzzer works on the assumption that if you send an invalid value into a boolean field, the service should return a validation error. Obviously, the GitHub API does not do this, but is rather silently converting the value to false. It’s true this is a consistent behaviour, applying for all boolean fields like auto_init, allow_merge_commits, etc., but I would personally choose to return a validation error in these cases.

This is in contradiction with the OWASP recommendation around strong input validation and data type enforcing.

Invalid values for enumerated fields implicitly converted to the default value (?)

The InvalidValuesInEnumsFieldsFuzzer will send invalid values in enum fields. It expects a validation error in return. The GitHub API does not seem to reject invalid values, but rather convert them to a default value and respond successfully to the request.

This is in contradiction with the OWASP recommendation around strong input validation and data type enforcing.

Integer fields accepting decimal or large negative or positive values

The DecimalValuesInIntegerFieldsFuzzer expects an error when it sends a decimal value inside an integer field. The GitHub API seems to accept these invalid values in the team_id field without returning any error, but rather resulting in a successful processing of the request. Same applies for ExtremePositiveValueInIntegerFieldsFuzzer and ExtremeNegativeValueIntegerFieldsFuzzer which will send values such as 9223372036854775807 or -9223372036854775808 in the team_id field. Strings also seem to be accepted in the team_id field.

This is in contradiction with the OWASP recommendation around strong input validation and data type enforcing.

Accepts unsupported or dummy Content-Type headers

The GitHub API seems to successfully accept and process requests containing unsupported (according to the OpenAPI contract) Content-Type headers such as: image/gif, multipart/related, etc. or dummy ones such as application/cats.

This is in contradiction with the OWASP recommendation around validation around content types.

Accepts duplicate headers

The GitHub API does not reject requests that contain duplicate headers. The HTTP standard itself allows duplicate headers for specific cases, but allowing duplicate headers might lead to hidden bugs.

Spaces are not trimmed from values

If the request fields are prefixed or trailed with spaces, they are rejected as invalid values. For example sending a Haskell space-prefixed value in the gitignore_template will cause the service to return an error. Although this is inconsistent with the fact that if you trail or prefix the since and before fields with spaces, the values get trimmed successfully and converted to dates. As a common approach I think that services should consistently trim spaces by default (maybe with some business-driven special cases) for all request fields and perform the validation after.

Accepting new fields in the request

The GitHub API seems to allow injection of new json fields inside the requests. The NewFieldsFuzzer adds a new catsField inside a request, but GitHub API accepts it as valid. This is again in contradiction with the OWASP recommendation which suggests rejecting unexpected content.

Doesn’t make proper use of enumerated values

There are cases when it makes sense to use enums rather than free text. Some examples are the gitignore_template field or the license_template field, which are rejecting invalid values. They are obviously having a pre-defined list of values, but do not enforce this in any way in the contract. Having them listed as enums will also make it easier to understand what are all supported templates for example.

Fields are not making proper use of the format

There are 2 fields called since and before which seem to actually be a date-time, although in the OpenAPI contract they are only marked as string without any additional format information. In the description of the fields it states that this is actually an ISO date, but it will also be good to leverage the features of OpenAPI and mark them accordingly.

Date-Time Error

Mismatch of validation between front-end driven calls and direct API calls

The homepage field seems to accept any values when doing a direct API call, but when you try to set this from the UI, you will get an error saying that you need to enter a valid URL.

Validation Error from the UI

Having the same level of validation for backend and frontend is another good practice for making sure you don’t end up with inconsistent data.

Additional findings

There are some other failures which might seem debatable or not applicable:

  • description accepts very large strings (50k characters sent by CATS), although the GitHub API doesn’t actually have constraint information in the contract; again this is not necessarily a problem, but it’s important for the contract to enforce constraints
  • The CheckSecurityHeadersFuzzer expects a Cache-Control: no-store as per the OWASP recommendations, but the endpoint does not operate critical data, so allowing caching of the information is fine

Cleaning Up

Before proceeding, please be careful to not delete your real repos. This is the script I’ve used to delete the repos. Run it incrementally and please check the repos file before deleting.

# get the latest 100 repos (by creation date)
curl -s "https://api.github.com/users/ludovicianul/repos?sort=created&direction=desc&per_page=100" | jq -r '.[].name' >> repos

# Maybe check them a bit before deleting to make sure you won't delete real repos
for URL in `cat repos`; do echo $URL;  curl -X DELETE -H 'Authorization: token PERSONAL_TOKEN' https://api.github.com/repos/ludovicianul/$URL; done

# delete the file
rm repos

#start over until you delete everything

You need to run it several times as it deletes in batches of 100.

Final Conclusions

CATS can be very powerful and can save lots of time while testing APIs. Because it tests every single field within a request, it’s easy to spot inconsistencies or cases when the APIs are not as explicit or as restrictive as it should. Although I only tested a single endpoint from the GitHub API, I suspect most the findings will apply to all the other endpoints.

Why not give it a try for your API?

How to write self-healing functional API tests with no coding effort

Context

APIs are everywhere, and it’s critical to have ways to efficiently test that they work correctly. There are several tools out there that are used for API testing, and the vast majority will require you to write the tests from scratch, even though in essence you test similar scenarios and apply the same testing techniques as you did in other apps/contexts. This article will show you a better way to tackle API testing that requires less effort on the “similar scenarios” part and allows you to focus on the more complex part of the activities.

Testing APIs

When thinking about testing APIs, from a functional perspective, there are several dimensions which come to mind:

  • verifying the structure of the exchanged data i.e. structural validations (length, data type, patterns, etc). This is where you send a 2 character string when the API say the minimum is 3, you put a slightly invalid email address, an slightly invalid format for a date field and so on.

  • boundary testing and negative scenarios, similar a bit to the above, but focusing on “breaking” the application. This is where you go to the extreme: you send extremely large values in Strings, extremely positive or negative values in Integers and so on.

  • behaviour according to documentation i.e. APIs are responding as expected in terms of HTTP status codes or response payloads. This is where you check that the service responds as documented: with a proper HTTP response code (2XX for a success, 400 for a bad request, etc) and a proper response body.

  • functional scenarios i.e. APIS are working as expected in terms of expected business behaviour. This is where you check that the response body contains the right business information: when you perform an action (like creating an entity, altering its state) the service correctly responds that the action was performed and with relevant information.

  • linked functional scenarios i.e. you create an entity and you get its details after to check they are the same. This is where you get a bit more end-to-end and you perform an action (like creating an entity) and with the return identifier you go and check the existence (you do a GET based on the identifier)

Ideally all the above scenarios must be automated. And it’s achievable to do this for the last 2 categories, but when having complex APIs with lots and lots of fields within the request payload, it becomes very hard to make sure you create negative and boundary testing scenarios for all fields.

What are the options for API test automation

There are several frameworks and tools on the market which can help to automate all these. Just to name a few:

They are all great tools and frameworks. You start writing test cases for the above categories. The tools/frameworks will provide different sets of facilities which might reduce specific effort during implementation, but ultimately you end up writing the actual tests to be executed (i.e. code). Even if you’ve done this before, and you know exactly what to test, even if you create a mini-framework that provides facilities for the common elements, you still need to write a considerable amount of code in order to automate all your testing scenarios.

And what happens when the API changes a bit? Some fields might be more restrictive in terms of validation, some might change the type, some might get renamed and so on. And then the API changes a bit more? This is usually not very rewarding work. Software engineers are usually creative creatures, and they like challenges and solving problems, not doing boring stuff. There are cases when one might choose to leave the API as is in order to prevent changing too many test cases.

Is there a better way?

But what if there is an alternative to this, and the first 3 categories above can be fully automated, including the actual writing of the test case. And also make the next 2 categories extremely simple to write and maintain. This is the reason I wrote CATs.

CATs has 3 main goals in mind:

  • remove the boring activities when testing APIs by automating the entire testing lifecycle: write, execute and report of test cases
  • auto-heal when APIs change
  • make writing functional scenarios as simple as possible by entirely removing the need to write code, while still leveraging the first 2 points

CATs has built-in Fuzzers which are actually pre-defined test cases with expected results that will validate if the API response as expected for a bunch of scenarios:

  • send string values in numeric fields
  • send outside the boundary values where constraints exist
  • send values not matching pre-defined regex patterns
  • remove fields from reuqests
  • add new fields inside the requests
  • and so on…

You can find a list of all the available Fuzzers here: https://endava.github.io/cats/docs/fuzzers/.

There is one catch though. Your API must provide an OpenAPI contract/spec. But this shouldn’t be an issue in my opinion. It’s good practice having your API documented in a tools-friendly format. Many companies are building OpenAPI specs for their API for easier tools integration.

Using CATs

Let’s take an example to show exactly how CATs works. I’ve chosen the Pet-Store application from the OpenAPITools example section. The OpenAPI spec is available in the same repo: https://github.com/OpenAPITools/openapi-petstore/blob/master/src/main/resources/openapi.yaml.

Looking at the contract, how much time would you estimate would take to create an automation suite to properly test this API? 1 Day? 2 Days? Several days? Using CATs this will probably be a couple of hours.

Let’s start the pet-store application on the local box first (you need to have Docker installed):

docker pull openapitools/openapi-petstore
docker run -d -e OPENAPI_BASE_PATH=/v3 -p 80:8080 openapitools/openapi-petstore

This will make the app available at http://localhost for the Swagger UI, and the API will be available at http://localhost/v3.

Download the latest CATs version. Download also the openapi.yml from above.

Before running CATs, please read how it works in order to better interpret the results.

Running the built-in test cases

You can now run:

 cats --server=http://localhost/v3 --contract=openapi.yml

You will get something like this: first_run.png

Not bad! We just generated 437 test cases, out of which 78 were already successful, and we’ve potentially found 181 bugs. To view the entire report, just open test-report/index.html. You will see something like this:

test_report_first_run.png

Now let’s check if the reported errors are actually bugs or just configuration issues. We will deal with the warnings after. Checking the first failing test (just click on the table row) we can see that the reason for failing is actually due to the fact that we didn’t send any form of authentication along with the requests. Looking in the openapi.yml we can see that some endpoints require authentication while some others require an api_key.

auth_required.png

CATs supports passing any type of headers with the requests. Let’s create the following headers.yml file:

all:
   Authorization: Bearer 5cc40680-4b7d-4b81-87db-fd9e750b060b
   api_key: special-key

You can obtain the Bearer token by authenticating in the Swagger UI. The api_key has a fixed value as stated in the pet-store documentation.

Let’s do another run, including now the headers.yml file:

cats --contract=openapu.yml --server=http://localhost/v3 --headers=headers.yml

A bit better now. 93 successful and 140 potential bugs.

second_run.png

Again, let’s check if there are any configuration issues, or they are valid errors. Looking at test 1253 we can see that the target endpoint has a parameter inside the url.

test_1253.png

This is a limitation of CATs as it cannot both fuzz the URL and the payloads for POST, PUT and PATCH requests. And looking at the contract, indeed, there are several other endpoints that are in the same situation. For this particular cases CATs uses the urlParams argument in order to pass some fixed values for these parameters. The provided values must result in existing HTTP resources so that the fuzzing works as expected.

We run CATs again, considering also the urlParams:

cats --contract=openapi.yml --server=http://localhost/v3 --headers=headers.yml --urlParams="username:username,petId:10,orderId=1"

91 passed test cases and “just” 81 potential bugs.

third_run.png

Let’s check again if these are actual errors. This time, they all seem to be valid bugs:

  • Test 429: the server returns a 500 Internal Error instead of a normal 200 empty response
  • Test 520: CATs injects a new field inside the request, but the server still responds with 200 instead of returning a 400 Bad Request
  • Test 523: CATs sends a null value in a non-required field (according to the contract) and expects a 2xx response, while the server replies with 500 evend ID cannot be null
  • and so on…

We now truly have 81 bugs which must be fixed in order to have a stable service.

Now, going into the warnings zone, these are usually tests which fail “just a bit”. The server usually responds withing the expected HTTP response code family, but either the HTTP status code is not documented inside the contract, or the response body doesn’t match the one documented in the contract. Either way, these also must be fixed as it shows that the OpenAPI spec is incomplete or that the actual implementation deviated from it.

Looking at what we’ve got at the end, with a small effort investment we were able to “write”, run and report 430 test cases which otherwise would have taken significant more effort to implement and run. We have both validation that the service works well in certain areas, it must update the OpenAPI specs or tweak the service to response as expected and got a list of bugs for behaviour which is not working properly.

Running custom test cases

But we can do even more. As mentioned previously CATs also supports running custom tests with minimal effort. This is done using the FunctionalFuzzer. The FunctionalFuzzer will run tests configured within a FuntionalFuzzer file that has a straightforward syntax. The cool think about the custom tests is that you don’t need to fill in all the details within the request, just the fields that you care about. Below is an example of running 2 simple test cases that create a Pet and verify the details of a Pet. This is saved as functionalFuzzer-pet.yml. (Ideally we should correlate these 2, but it seems that the pet-store API is not returning an ID when you create a Pet).

/pet:
  test1:
    description: Create a new Pet
    name: "CATs"
    expectedResponseCode: 200
/pet/{petId}:
  test2:
    description: Get the details of a given Pet
    petId: 11
    expectedResponseCode: 200
    verify:
      id: 11
      category#id: 7

We now run CATs again (notice the cats run sub-command):

cats run --contract=openapi.yml --server=http://localhost/v3 --headers=header_pet.yml functionalFuzzer-pet.yml

And we get the following result:

custom

The 2 warnings are also due to the fact that the response codes are not properly documented in the OpenApi contract.

And you get the same benefits as when writing the test cases using actual code:

  • you can choose the elements to verify in the responses, using the verify section
  • you can check for expected HTTP status codes using expectedResponseCode
  • and you can even pass values from one test to another using output

As mentioned above, ideally, when getting the Pet details, you should pass an ID received when initially creating the Pet to make sure that is the same Pet. The functionalFuzzer-pet.yml will look like this:

 /pet:
   test1:
     description: Create a new Pet
     name: "CATs"
     expectedResponseCode: 200
     output:
        pet_d: id
 /pet/{petId}:
   test2:
     description: Get the details of a given Pet
     petId: ${pet_id}
     expectedResponseCode: 200
     verify:
       name: "CATs"

What about auto-healing

Auto-healing is built in. As CATs generates all its tests from the OpenAPI contract, when the API updates i.e the OpenAPI contract updates also, CATs will generate the tests accordingly.

Conclusions

CATs is not intended to replace your existing tooling (although it might do it to a considerable extent). It’s a simple way to remove some of the boring parts of API testing by proving built-in test cases which can be run out-of-the box. It also makes sure it covers all fields from request bodies and has a good tolerance for API changes.

In the same time, it might be appealing for people that want to automate API test cases, but might not have the necessary skills to use tools required significant coding.

Why not give it a try?

Visual Cultures Part 1 - XFDs

My team (when I say team, I actually mean 14 individual teams at the moment) is using big visible dashboards since they weren’t a thing. In fact, not only dashboards, but our entire culture is very visual. We like things to be transparent and easily accessible with a simple look. This applies to everything: sprint/iteration progress, code quality, system test environments health, release readiness (I know, but not all of us are doing fully-automated-kubernetees-driven-continuous-delivery-with-10-deploys-per-day), build stability and so forth. Even if everything has a digital form that displays similar information, we prefer to also have the physical/visual stuff. It’s instant and meaningful.

One of the first things we’ve used was an eXtreme Feedback Device (XFD). There are many geeks within the team. What better way to express your geeky skills than DIY techy projects? And an XFD seemed like a very good candidate. For those unfamiliar with what an XFD is a short definition would be an instant and visual way you can monitor a Continuous Integration build health.

And so, we’ve built one :)

XFD Version 1

XFD 1.0

The first version of the XFD was built using: • an Arduino board • an RGB LED • a Java agent that monitored the CI server’s builds and was sending bytes according to the health on a serial port to the Arduino board • some Arduino code that was interpreting the bytes from the Java agent and setting the LED lights accordingly

And it worked very well. Build failed! Bang!!! XFD was turning 🔴. Build in progress! Bang! XFD was pulsing 🔵. System test environment was unhealthy! Bang! XFD was turning yellow. You get the point. We now had instant visual feedback for things very important for the team.

The Java agent was configurable so you could accommodate multiple CI builds or different team events (like daily stand-ups) based on specific time frames.

Over time, as we become more teams, the need for diversity increased. This is why we’ve created different flavors of the XFD. They were working in the same way but had different aesthetics.

More minimalistic XFD

Christmas version

XFD 2.0

We’ve used the 1.0 solution for almost 6 years. During an office move last year, we’ve noticed that everything was fine as long as you leave the XFD in one place. Once you need to move it around, it was a bit difficult for the 2018 geeks to cope with having so many wires (power-in, power-out, USB) and bulky designs.

Welcome XFD 2.0! We didn’t actually build one. We bought Xiaomi Yeelight RGB light bulbs. Quite cheap, portable and API driven :) Very geek friendly. We’ve re-written the Java agent from scratch in order to communicate with the bulb. And everything was working like before, but in a cleaner way…

… until the need for diversity stroke again. After all what each team had was just a light bulb. So, we’ve launched a competition between teams. Who creates the best looking XFD (while still being powered by a light bulb)? I must say that the results are impressive. We now have 12 really cool concepts that in the same time capture the uniqueness of each team. All pictures below.

Simpsons Jedi Smurfs Fight Club Transformers Black Sheep Wolfpack X-Men Florence Monsters Incredibles Lucky 13

XFDs are so early 00’s - Do they really work?

I strongly believe they do. But they don’t “just work”. You must create a specific mindset. Cultures do not emerge because things are there - that’s it; people need to behave in a certain way. They emerge if they are part of a bigger picture. From a mindset that considers that visual items are important. And that visual ways to show stuff are key for transparency and instant feedback.

The code for XFD 2.0

You can also start your own XFD in 2 minutes (after you buy a Xiaomi Yeelight RGB LED). The code is on GitHub: https://github.com/ludovicianul/yeextreme

Be visual!