Who I Am and Where We Are

My name's Ben Mohrbacher. I'm a staff engineer here at Kin and currently in charge of the Quoting team. I've been in the industry for about 15 years with the majority of those operating in Ruby. Kin is largely a Ruby shop with a large Rails monolith. However, we're actively working to refine our architecture so we can scale. Part of this proactive work is defining the standards that will empower our next phase of growth. This post describes one of them.

Straight to the Point

I won't bury the lede: if you write Ruby code and use RSpec I encourage you to write all your tests using the following rules. Even if you don't write Ruby or use RSpec, there's still a good chance the lessons here are useful for your language and testing framework. I call this "writing tests procedurally."

• Put everything you need to pass each individual test into your it blocks.
• Don't use before, let, let!, subject, described_class, shared_examples, shared_context, etc. — anything that pulls test setup out of the it block.
• Don't use helper functions for your test file. That's often a signal that the underlying code is overly complex.
• Only use context as a way to group related tests, not as a way to group their setup.
• Know with deep and unflinching certainty when you want to break one of these rules. "Because it's slightly faster to get this PR out" is not a good reason. Believe it or not, "Because this test is long" is also not a good reason.

What People Frequently Do

This is the style I've encountered across the industry among those who write Ruby code and use RSpec. It's a standard approach that you'll see reflected in much of our existing codebase. I'll call this the DRY (Don't Repeat Yourself) style.

describe MyClass do
  subject { MyClass.new(arg1:, arg2:) }
  let(:arg1) { 5_000 }
  before do
    FooConfabulator.call
  end
  context "when it's a foo" do
    let(:arg2) { :foo }
    it "does a baz correctly" do
      expect(subject.baz).to be_correct
    end
    it "calls to the qux system" do
      expect(QuxSystem).to receive(:interact).with(:foo)
      subject.baz
    end
  end
end

These kinds of toy examples get used a lot when people are first learning to write tests in RSpec. "It's easier," it's explained, "because the setup code doesn't get duplicated. When you need to write a new test, you write a new it block and put your expectation in there."

This has not been our experience with how these test files grow. I imagine it's not a lot of people's experience. I have a spec file open on my screen right now that does not feature a single it within my vertical viewing area. I have to scroll back and forth or split my view to reference the test setup on one pane and the expectation on another. This is not my ideal viewing experience. I call this the visibility problem.

How we see these test files growing is that test setup and test expectations become lovers grown distant by an ever-expanding gulf of interposing lines. Their tumultuous relationship is torn further asunder as setup is overwritten by let definitions at various and multiple layers of nesting. The test setup for a single test can easily end up shredded across hundreds of lines, multiple levels of indentation, and — in the case of shared_examples — multiple files. In addition to testing the code, it tests my patience.

What I'm Suggesting You Do

Instead, please consider the rules I outlined at the beginning of this post. Write your tests procedurally — a simple recipe of "arrange," "act," and "assert" in a single location per test. The above example would instead look like the following:

describe MyClass do
  context "when it's a foo" do
    it "does a baz correctly" do
      FooConfabulator.call
      arg1 = 5_000
      arg2 = :foo
      my_class = MyClass.new(arg1:, arg2:)
      expect(my_class.baz).to be_correct
    end
    it "calls to the qux system" do
      FooConfabulator.call
      arg1 = 5_000
      arg2 = :foo
      my_class = MyClass.new(arg1:, arg2:)
      expect(QuxSystem).to receive(:interact).with(:foo)
      my_class.baz
    end
  end
end

You'll notice in this toy example that it's roughly the same number of lines as the DRY style. This won't always be the case, but in shorter spec files I've found it to be true. It's not necessarily much larger unless your setup requirement is larger.

The Pros of Writing Tests Procedurally

• Setup remains co-located with test code.
• The order of setup code is clear and definitive; code is run from top to bottom.
• Tests only depend on the Code Under Test and not each other.
• Large amounts of test setup is a useful signal about the architecture of your code and its friction points. A lot of test setup for a class shows where you can refactor to remove complexity.
• Since tests don't depend on each other, they can be reordered within a file or moved between files with little fanfare.
• Adding a new test does not require understanding the rest of the file. You do not need to scroll up and down to determine which setup is already supplied.
• Updating many tests is just a "find and replace" away most of the time. If you think you missed something, run the whole file and it'll tell you. This beats the hell out of trying to update a let and seeing it broke several tests before you realized you needed a whole other level of nesting for your expectation.
• There are fewer rules to learn or teach about how to organize the test code. E.g., "let/before should be visible from the test block." You simply put it all in the it block.
• Having to remember or convey fewer rules makes this style very culturally portable. It does not require a lot of experience or intuition to execute.
• You avoid running test setup that doesn't matter for your expectations. This can easily happen if you use before and results in wasted cycles.
• Your descriptions don't need to be chopped up along five layers of nesting. Nesting is exclusively an organizational choice and does not impact test setup.

The Cons of Writing Tests Procedurally

• Test setup is duplicated. You'll be rewriting roughly the same setup in every it block you write for the same Code Under Test.
• The result of the above is that your it blocks and spec files can be more lines than if you write them DRY.

Why Not DRY?

Right up top let me make it clear that let, before, subject, described_class, shared_examples, shared_context, etc. all suffer from the visibility problem. All are capable of making it difficult to tell at a glance what setup is being defined for a given test.

Why not let/let!?

lets don't run in the order they're defined, unless you use let! exclusively. This obscures dependencies between setup, especially if you have some of them defined in a nested context above or below where you're currently reading.

Overriding let definitions at other levels of nesting can be difficult to follow along with. It's fine if the override happens right next to the test. It's less fine if it's in a context off the top of your screen. Having to assemble context for all the setup in multiple locations sucks — this is the basis of the visibility problem.

Why not before?

before blocks don't care if your tests need to run that setup, they're going to do it anyway. Yes, you could be disciplined enough to only ever put appropriate setup into appropriately-nested blocks. Will everybody on the team know and be that disciplined? It is difficult to guarantee.

Why not subject?

While in many cases you could establish the instance you intend to test at the top, many times you'll need to pass arguments in. Then you've immediately got the problem of needing to use let to define them because otherwise the subject block doesn't have it available in scope. Choosing to use subject only when it's not a problem creates a friction point when you either need to remember to change it or tear it out.

Why not described_class?

Abstracting away the name of the class you're testing falls under a very loose version of the visibility problem. It's not a huge deal, but I personally prefer not having to scroll up or glance at my IDE's gutter to see which file I'm in. This is probably the least onerous piece of RSpec you could use.

Why not shared_examples/shared_contexts?

Oh you better believe that's a visibility problem.

I admit a lack of experience in writing custom matchers (a subset of "put it in another file") or utilizing a lot of shared_examples. I'm sure the people who write them mean well, but I don't want to have to open several files to understand what a test is actually testing.

Common Concerns

Ew, yuck, gross, etc.

I want to make it clear that I understand everybody has their preferences. However, we work in groups, and consensus and consistency is important when you work in groups. It's hard to maintain consistency when there's more rules to follow — you have to make sure everybody is taught every rule, and you have to enforce every rule. You can try to solve the latter with linters, but it's hard to overcome the former. I prefer to operate under a set of rules that don't change under context.

"Just remember that in X situation you should do Y instead of Z" allows ample room for error.

It's a change for some folks, especially if you're very used to trying to DRY up your test code. I've written a zillion tests DRY and a zillion tests procedurally. I really recommend you try doing things procedurally if you haven't. It's very nice to remain ignorant of other tests in a file, to not have to hunt up and down for the context of what you're working on, and to be able to sketch everything out in one spot.

I'm not recommending this out of philosophical purity; I'm recommending it because it's been extremely useful to me and to my team.

That isn't very DRY

I understand we spend all our time building abstractions in code and reducing duplication and making things smarter, not harder. Yet, here I am, telling you to not do that with tests. But tests aren't the same as application code. Tests are a feedback mechanism we use to ensure the correctness of our code. They're a different kind of thing, and we can treat them differently.

The tests are already dependent on the code they're testing. By adding dependencies between tests, we begin to tightly couple the implementation of those tests to each other. This makes the section of the test where the expectation lives much smaller, but it doesn't make writing or reading them easier.

Writing tests as DRY as possible takes discipline. It takes a variety of rules to remember how and when to slice up the tests and their setup. Plus, it doesn't scale. We want our tests and our test files to be able to grow arbitrarily. I don't want to have to fiddle with test setup 100 or more lines away from the test I'm trying to write just so I won't repeat a one-line factory call. The juice quickly becomes worth a fraction of the squeeze.

It's cleaner to separate tests and their setup

I value pragmatism, and I find it pragmatic to quickly slap a test together while ignoring the rest of the content in the test file. Personally, I think it is cleaner to keep related items together rather than apart.

My unit tests are much bigger now and that feels bad

You should listen to that feeling. If you require a ton of setup to get your code tested, that's a signal that the code has too many dependencies and needs to be fixed. In a world where you abstract the setup away, it's easy to ignore setup bloat. It's not easy to ignore it when it's every single test you write.

It's harder to tell what's set up differently between two tests this way

True, I won't argue with that. I will say that I rarely recall scenarios where differentiating between two tests more easily was super important. In general, I'm focused on a single test and its output — one at a time. When using tests for feedback, once the setup and assertions are ready, whether they are configured differently is typically irrelevant. If you focus on having one expectation (or a small cluster of related expectations) per test, then marking out the distinctive setup for that expectation is usually trivial. If it's more complex, see my previous statements about that being a useful signal about your architecture.

In Conclusion

Writing tests procedurally is the least clever way to write tests. You should spend your cleverness budget on things that really warrant it. Instead, simplify the way you write individual tests to co-locate all your setup and expectations together. Along the dimensions that we most frequently use tests, it is easier to read and write them, and it will scale with the growth of your codebase and your company.