Tasty is a command-line tool that runs API tests defined and grouped in TOML files. An upcoming enhancements PR is in progress, and I want to take a break from code for just a bit to both share what’s all on my mind for Tasty’s future and to give interested folks (hello!) a window into the way I think about building command-line tools. Tasty solves a personal need,1 and as an open source project, maybe it can solve yours, too.
In this post, I’ll be going over some of the developer experience decisions that are driving the upcoming enhancements, how they influence the tool’s development, and what to look forward to as we edge closer to a stable v1.0 release in the near future. There are two features I want to ship in the upcoming release that will enable my own workflows to migrate entirely over to Tasty: the ability for subsequent tests to reference prior tests (and to leverage regex where needed), and granular control over expectations.
# Test response referencing and regex
The first feature I want to talk about is test response referencing with regex-based response validation. When a test passes, we should be able to reference the response values in subsequent test payloads. In the examples provided later, I’ll show you how I would pass tokens from a hypothetical authentication endpoint into other tests with the changes I’m working on–which isn’t possible today in Tasy, since it only tests the presence of response data. This of course is fine for simple APIs like the ones I’ve written for simple intranet services that are locked down at the HA Proxy layer, but the goal for Tasty is to also be able to validate response data in cases where APIs return dynamic values. Again, auth tokens are a good example case because they represent properties that must be present and meet certain criteria–but we won’t know the value of these before the request to retrieve them runs.
Let’s look at an example TOML file that illustrates a test using the expected values from a previous test’s response as properties in its payload:
[my_first_test]
method = "POST"
route = "auth/login"
payload.email = "[email protected]"
payload.password = "notreal123"
expect_http_status = 200
expect_response_includes.status = "ok"
expect_response_includes.access_token = "(.*)"
expect_response_includes.refresh_token = "(.*)"
[my_second_test]
method = "GET"
route = "some/protected"
payload.auth_token = { from = "my_first_test", property = "access_token" }
expect_http_status = 200
expect_response_includes.status = "ok"
expect_response_includes.profile = { email = "[email protected]" }
The syntax above is NOT the final syntax as we head toward v1.0. Please keep reading to the end–or just skip to the bottom of this article to see what the syntax will actually look like.
Here we have the test my_first_test expecting an access_token in the response. The regex pattern (.*) will match the entire contents of the access_token property from the data payload; it’s akin to saying “literally anything and everything that comprises this string response.” One thing that might stand out is the access_token expectation: right now, we assume that the property exists on the root of the data payload. This wont work in a library setting; what if you need data from a nested property; { some: [first: {}, second:, or_third: { nested: { property: "like this" }}]}?
One solution might be to nest the values using some clever Toml syntax. We could try to get a nested property like this:
payload.some_field = { from = "the_test_with_the_field", property.some_parent.some_child = "what we want" }
But this introduces some unnecessary complexity in using nested TOML as a reference artifact for an extracted property. Even though the payload expectation is written in nested TOML:
[some_test]
# ...
expect_response_includes.profile.feed_data.current_pinned = "abc123"
… we can’t really use the same nested TOML syntax when referencing this expectation’s resultant property. After all, what would that look like in the test file?
[some_other_test]
# ...
payload.current_pinned = { from = "some_test", response.profile.feed_data = "current_pinned" } # ???
That just wont work out very well because we’re creating an assignment with parts of the value that comprise the entire reference. All affordances introduce some form and quantity of cognitive overhead, but this introduces a specific kind of cognitive overhead that leads to cognitive friction for the test author. What we need is a way to reference the entire path of the response, rather than two parts of the reference broken up into the left and right portions of an assignment operation.
Put another way, we need a way to reference the key part of an extracted property’s key-value pair, which may be nested in a JSON path.
So, what if we just use the JSON path?
[some_test]
# ...
expect_response_includes.profile.feed_data.current_pinned = "abc123" # Don't do this; keep reading
[some_other_test]
# ...
payload.current_pinned = { from = "some_test", path = "profile.feed_data.current_payload" }
This feels like the right path (no pun intended). I don’t like that there are two ways to declare the actual property–one in the prior test’s expectations, and another in the subsequent test’s payload injection–but this kind of cognitive overhead is synonymous with other kinds of cognitive overhead already part of our ecosystem (i.e., the dot-notation for the path name being used in other systems). In this case, we’re introducing a variant of existing cognitive complexity that we deal with day-to-day, rather than a new kind of complexity to account for.
# Granular expectations control
The second feature I want to talk about is graular expectations control. If we were to add an additional kind of data to test expectations against, how would we express this in the test case syntax? For example, lets’s say we want to check that a certain endpoint returns a specific header. Right now, Tasty can’t do that–but the issue isn’t the lack of an affordance that can just be written and shipped, it’s that Tasty was not really written in a way that encourages extension of core functionality. This brings up some design notes I’ll be talking about later in this post which are all part of what’s getting closer and closer with every enhancement: a stable v1.0.0 release.
Let’s get back to expectations. The only affordance we have is the verbose expect_response_includes key. This has worked well so far for what I needed it for, and I gave myself permission to ship it without worrying too much about future hypothetical needs. To add support for additional types of expectations (like expectations for headers), we have two options:
Our first option is to introduce another explicit key, like expect_header_includes. This should lead us to an architectural question about whether future extensions will also require their own explicit key. Since we have already experienced wanting to introduce another expectation, we can be reasonably sure that there may be more expectations that we aren’t considering now (and in fact, I can think of a few off the top of my head that I don’t need to care about right now but that perhaps some other Tasty user in the future might want to submit a contribution to address). In that case, we ought to avoid another explicit key and instead explore a way to naturally organization our expectations in a structured way.
The way I am doing this is to create a top-level expect key on each test case, and then reorganize the project code to better enable extendability. The first “extension” in this new code structure will be support for payload responses.
This, of course, does introduce a breaking change for previous tests–unless we decide to keep support for expect_header_includes. But since “we” in this case is just Present Me and Future Me (hello if you’re reading this), the fact that this is such a young library with a userbase of one, I’m not too worried right now about making this change. After all, if there are going to be big changes like this, they should be toward stabilizing the developer experience in v1.0.0.
Here’s the expected syntax:
expect.response = { my_token = "(.*)" }
The above lets Tasty know that we expect the response to contain a my_token key with any value. Since we’re using regex for the contents of the property, we can also do something like this:
expect.response = { username = "UN(.*)" }
Let’s look at another example that brings this all together:
[my_first_test]
method = "POST"
route = "auth/login"
payload.email = "[email protected]"
payload.password = "notreal123"
expect.http_status = 200
expect.response.status = "ok"
expect.response.access_token = "(.*)"
expect.response.refresh_token = "(.*)"
[my_second_test]
method = "GET"
route = "some/protected"
payload.auth_token = { from = "my_first_test", property = "response.access_token" }
expect.http_status = 200
expect.response.status = "ok"
expect.response.profile.email = "[email protected]"
Notably, you use string-based dot-notation to reference the response values from previous test expectations. Neat!
This gives us the ability to provide additional support in the future for other kinds of expectations, like:
expect.headers.x-cdn-geo-id = "abc123"
expect.headers.user-agent = "Browser (version) something"
Importantly, I don’t have to worry about defining the future syntax for header support right now–but these changes will enable future support to be added while simultaneously improving the ergonomics of writing tests with Tasty.
There’s much to be done still code-wise, and even more to be done to the readme before this is ready, but I’m excited to get this shaped up and to a point where it’s more usable by more people. We’re not quite ready for the stable 1.0 release, but these changes get us one step closer.
Alright, time to get back to the IDE.
-Jesse