My four year old son is into withholding information.
“How was school?”
“What did you learn today?”
“Were the other kids mean to you?”
Like most kids, he finds it fun to keep me in the dark. Of course, what he doesn’t realize is I can help him with problems, but only if he tells me about them.
Software without an automated test suite behaves like my four year old. It keeps secrets and makes it really hard to proactively fix problems. In contrast, software with good test coverage calls out “Yo, I’m not doing what you expected!” It grabs your attention so that you can fix things before they become big problems. Automated tests let your code step up and communicate like an adult.
Unfortunately, it’s very tempting for developers to skip writing tests. The mantra “if it’s not tested, it’s broken” echoes around within the development community because writing tests can be a drag – it’s a time consuming process that reveals flaws which the dev might prefer to gloss over. If you want well tested code, which you should, then you need to give the tests an audience outside of just the dev team. The push for good tests needs to be from the whole company because most developers need prodding to maintain good test coverage. Developers, including me, feel safe and often subconsciously prefer to keep non-tech stakeholders in the dark, just like my four year old.
I led our company, Power of Two, into this trap last year as we developed the foundations of our service. At the time we were rapidly iterating through customer development, evolving the business model on a weekly basis. When I presented the challenge of keeping the code aligned with our ever-changing business model the dev team leader suggested that we avoid the overhead of writing tests. He rightly argued that writing tests would take time away from feature creation and that test suites easily become brittle and break on changes that aren’t actually problems. What I didn’t yet understand was that the business value of good test coverage far outweighs these costs.
Our dev team did great work in every way except test coverage. We quickly adapted our code base with every pivot. We built great feature fast. And we created software that held its tongue like my four year old. Now, a year on, almost every day I find myself wanting to slap the software upside the head and tell it to start talking. Without full test coverage (a situation we are working hard to fix) every change is frightening. We frequently hit with unexpected breakages because, even though the code is well-structured, there are inevitably dependencies lurking, forgotten, in the code. We also often find ourselves pondering “why does the code work this way?” Our developers are not capricious. all the code in our system has purpose. Still, without tests to clarify the expectation that necessitated a given special case, it is often a struggle to remember why we wrote what we wrote.
Poor test coverage is a problem for devs. it is a also a huge lost opportunity for the non devs because because the code is silent to them. Well-written tests, or even just well-written test titles, declare how the code is expected to work. These are the statements that bridge the business needs into the coded reality. Non-devs should understand and follow the expectations that are declared in the tests. The tests are their direct, non-technical window into the code base. Unless non-devs engage with the test suite they are entirely reliant on the devs to explain the code base. These explanations often get garbled. When your code base doesn’t talk, the company has to play a game of telephone (Chinese whispers here in the UK):
- non-dev to dev: How do we expect our messaging feature to work?
- dev to code base: Let me see. How do you work?
- code base to dev: I send a message and record the content and when it is deleted.
- dev to non-dev: It sends a message and records the content.
- non-dev to dev: Let’s let people delete messages.
- dev to non-dev: Oh yeah, it already does that.
- non-dev to dev: Let’s add a way to tell when it is read.
- dev to code base: I expect you to record when a message is opened (having assumed read == opened)
- code base to dev: I record when a message is opened
- dev to non-dev: The code does what you want.
- non-dev to dev: Great.
- … Several days later.
- non-dev to dev: How come messages are marked as read when they are only opened?
- … and so forth.
The point is that when the code can’t speak for itself the non-devs are stuck outside of the conversation. In contrast, here is a typical conversation at Power of Two:
- code base to all:
- When a message is sent I record the content.
- When a message is deleted I record the date.
- non-dev to dev: Lets add a way to tell when a message is read.
- dev to code base: I expect you to record when a message is openned (having assumed read == openned).
- code base to all: When a message is opened I record the date.
- non-dev to dev: Wait, I don’t want to know when it is opened, but rather when it was marked as read.
- dev – non-dev: Ahha, I misunderstood.
- dev to code base: When a message is marked as read you should record the date.
- code base to all: When a user marks a message as read I record the time.
- non-dev to self: Rockin!
As you can see there is still potential for confusion, but by letting the code base speak for itself the misunderstanding is resolved more quickly, and the non-developer does not have to rely on what the dev says the code does.
More importantly the non-devs now directly get a host of other information about things like when which features are completed, how complex they were, which other expectations were impacted, and so forth. All in all, the conversation becomes more rich, more direct and more valuable. When the code talks to everyone, the whole team will make better decisions, spend less time trying to stay coordinated, and stay out of the the destructive us versus them mindset that often erupts between devs and non-devs.
Also, when the devs know that everyone reads their tests they have strong incentive to keep the test suite up to date and well written. This prevents the developers from quietly falling away from good test coverage and bringing on all the future grief mentioned above.
- Make your code base into an adult that can speak for itself with an automated test suite.
- Create a conversation about expectations between your code base and your entire team, non-devs included.
- Iterate faster, reduce confusion and don’t dig your self into the whole caused by code without automated tests.
Here’s how we run this at Power of Two.
Each time we do a release, which is several times a day, we email a notice like the following to the full team. This notice summarizes the release, lists the expectations for the changed code and provides the commit messages for greater detail.
Here is the gist of our release notes generation script. It is written in python using Fabric.
=== RELEASE MESSAGES --- 7.1g0 Feature: Initial coaching stocks 7.1g1 Tweak 7.1g2 Tweak: Fix behavior test CHANGED EXPECTATIONS (not all tests have changed) --- winwin_project/apps/processes/tests/profile.py: - New user should have a consistency score - User with none consistency score should get a score when they login - User with none consistency score should not get a score at a new period - New period should add a non login period to the consistency score - New login for period should make the most recent consistency period into yes - Actual login should make most recent period a login for consist history - Needs coach flag should become true when a user logs in - Email users action should redirect to content creation page - Set coach message should do nothing - Apply assignment should do nothing winwin_project/apps/profiles/tests.py: - Needs coach flag for a new user should be true - Staff user link should return a url - Logged in function should alter member profiles - Logged in function should not alter staff profiles COMMIT MESSAGES --- * 27fde36 - Better admin for coaching using stocks adjusted to handle blank database during testing (23 hours ago by Jesse) * 7fb8b23 - Showed requesting help and reordered stock admin. (23 hours ago by Jesse) * 7f6cadf - Associated a coach with a signupkey to auto-assign. (23 hours ago by Jesse) * da61b51 - Removed script that set up south. (23 hours ago by Jesse) | * de200fc - release fix and db copy to dev server (16 hours ago by Jesse) | * c47a08a - Merge branch 'master' of po2-winwin (16 hours ago by Jesse) | * 68805b8 - Load copy of live db into a dev server. (13 hours ago by Jesse) |\| | |\ |/ / * | c84b7e3 - Merge branch 'master' of po2-winwin (13 hours ago by Jesse) | * 1cd398c - Merge remote branch 'origin\master' into coaching (22 hours ago by Jesse) | * 5ee41dd - Members stock and don't mark staff as needing coach. (13 hours ago by CRM) |/ * 9677d74 - Merge branch 'coaching' (13 hours ago by Jesse) (tag: 7.1g0) * 4e4ebe9 - Print nothing to do (13 hours ago by CRM) (tag: 7.1g1) * 1c5adf9 - Aligned behavior test with new user accordion. (13 hours ago by DB) (HEAD, tag: 7.1g2, staging\release, release) ===
I hope that you find these ideas helpful.