New Tricks for an Old Dog

Much of my year has been spent helping new team members find their way around the expansive and complex codebase that is the TweetDeck front-end, trying to build a happy and productive group of people around a substantial codebase with many layers of legacy.

I’ve loved doing this. Everything from writing new documentation, drawing diagrams, and holding technical architecture sessions teaches you something you didn’t know or exposes an area of uncertainty that you can go work on.

In this article, I hope to share some experiences and techniques that will prove useful in your own situation and that you can impress your friends in some new and exciting ways!

How do you do, fellow kids?

To start with I’d like to introduce you to our JavaScript framework, Flight. Right now it’s used by twitter.com and TweetDeck although, as a company, Twitter is largely moving to React.

Over time, as we used Flight for more complex interfaces, we found it wasn’t scaling with us.

Composing components into trees was fiddly and often only applied for a specific parent-child pairing. It seems like an obvious feature with hindsight, but it didn’t come built-in to Flight, and it made reusing components a real challenge.

There was no standard way to manage the state of a component; they all did it slightly differently, and the technique often varied by who was writing the code. This cost us in maintainability as you just couldn’t predict how a component would be built until you opened it.

Making matters worse, Flight relied on events to move data around the application. Unfortunately, events aren’t good for giving structure to complex logic. They jump around in a way that’s hard to understand and debug, and force you to search your code for a specific string — the event name‚ to figure out what’s going on.

To find fixes for these problems, we looked around at other frameworks. We like React for it’s simple, predictable state management and reactive re-render flow, and Elm for bringing strict functional programming to everyone.

But when you have lots of existing code, rewriting or switching framework is a painful and expensive option. You have to understand how it will interact with your existing code, how you’ll test it alongside existing code, and how it will affect the size and performance of the application. This all takes time and effort!

Instead of planning a rewrite, we looked for the ideas hidden within other frameworks that we could reapply in our own situation or bring to the tools we already were using.

Boiled down, what we liked seemed quite simple:

• Component nesting & composition
• Easy, predictable state management
• Normal functions for data manipulation

Making these ideas applicable to Flight took some time, but we’re in a much better place now. Through persistent trial-and-error, we have well documented, testable and standard techniques for creating complex component hierarchies, updating and reacting to state changes, and passing data around the app.

While the specifics of our situation and Flight aren’t really important, this experience taught me something:

Distill good tech into great ideas. You can apply great ideas anywhere.

You don’t have to use cool kids’ latest framework, hottest build tool or fashionable language to benefit from them. If you can identify a nugget of gold at the heart of it all, why not use it to improve what you have already?

Times, they are a changin’

Apart from stealing ideas from the new and shiny, how can we keep make the most of improved tooling and techniques? Times change and so should the way we write code.

Going back in time a bit, TweetDeck used some slightly outmoded tools for building and bundling. Without a transpiler like Babel we were missing out new language features, and without a more advanced build tools like Webpack, every module’s source was encased in AMD boilerplate.

In fact, we found ourselves with a mix of both AMD syntaxes:

define(["lodash"], function (_) {
  // . . .
});

define(function (require) {
  var _ = require("lodash");
  // . . .
});

This just wouldn’t do. And besides, what we really wanted was CommonJS, or even ES2015 module syntax:

import _ from "lodash";

These days we’re using Babel, Webpack, ES2015 modules and many new language features that make development just… better. But how did we get there?

To explain, I want to introduce you to codemods and jscodeshift.

A codemod is a large-scale refactor of a whole codebase, often mechanical or repetitive. Think of renaming a module or changing an API like URL("...") to new URL("...").

jscodeshift is a toolkit for running automated codemods, where you express a code transformation using code. The automated codemod operates on each file’s syntax tree – a data-structure representation of the code — finding and modifying in place as it goes.

Here’s an example that renames all instances of the variable foo to bar:

module.exports = function (fileInfo, api) {
  return api
    .jscodeshift(fileInfo.source)
    .findVariableDeclarators('foo')
    .renameTo('bar')
    .toSource();
};

It’s a seriously powerful tool, and we’ve used it to write a series of codemods that:

• rename modules,
• unify our use of AMD to a single syntax,
• transition from one testing framework to another, and
• switch from AMD to CommonJS.

These changes can be pretty huge and far-reaching. Here’s an example commit from when we switched to CommonJS:

commit 8f75de8fd4c702115c7bf58febba1afa96ae52fc
Date:   Tue Jul 12 2016

   Run AMD -> CommonJS codemod

 418 files changed, 47550 insertions(+), 48468 deletions(-)

Yep, that’s just under 50k lines changed, tested, merged and deployed without any trouble. AMD be gone!

---

From this step-by-step approach, using codemods to incrementally tweak and improve, we extracted a little codemod recipe for making significant, multi-stage changes:

1. Find all the existing patterns
2. Choose the two most similar
3. Unify with a codemod
4. Repeat.

For example:

1. For module loading, we had 2 competing AMD patterns plus some use of CommonJS
2. The two AMD syntaxes were the most similar
3. We used a codemod to move to unify the AMD patterns
4. Later we returned to AMD to convert it to CommonJS

It’s worked for us, and if you’d like to know more about codemods then check out Evolving Complex Systems Incrementally by Facebook engineer, Christoph Pojer.

Welcome aboard!

As TweetDeck has gotten older and larger, the amount of things a new engineer has to learn about has exploded. The myriad of microservices that manage our data and their layers of authentication, security and business logic around them make for an overwhelming amount of information to hand to a newbie.

Inspired by Amy’s amazing Guide to the Care and Feeding of Junior Devs, we realised it was important to take time to design our onboarding that each of our new hires go through to make the most of their first few weeks.

Joining a new company, team, or both, is stressful and uncomfortable. Everything you can do to help a new hire will be valuable to them. So please, take time to design your onboarding!

And as you build up an onboarding process, you’ll create things that are useful for more than just new hires; it’ll force you to write documentation, for example, in a way that’s understandable for people who are unfamiliar with your team, product and codebase. This can lead to more outside contributions: potential contributors feel more comfortable getting set up on your product without asking for help.

This is something that’s taken for granted in open source, but somehow I think we forget about it in big companies.

After all, better documentation is just a good thing. You will forget things from time to time, and you’d be surprised how often the “beginner” docs help!

For TweetDeck, we put together system and architecture diagrams, and one-pager explanations of important concepts:

• What are our dependencies?
• Where are the potential points of failure?
• Where does authentication live? Storage? Caching?
• Who owns “X”?

---

Of course, learning continues long after onboarding. The landscape is constantly shifting; old services are deprecated, new APIs appear and what once true can suddenly be very wrong. Keeping up with this is a serious challenge, and more than any one person can track.

To address this, we’ve thought hard about our knowledge sharing practices across the whole team. For example, we completely changed the way we do code review.

In my opinion, code review is the single most effective practice you can introduce to share knowledge around, and build the quality and consistency of your team’s work. But, if you’re not doing it, here’s my suggestion for getting started:

Every pull request gets a +1 from someone else.

That’s all — it’s very light-weight and easy. Just ask someone to have a quick look over your code before it goes into master.

At Twitter, every commit gets a code review. We do a lot of reviewing, so small efficiency and effectiveness improvements make a big difference. Over time we learned some things:

• Don’t review for more than hour ¹
• Keep reviews smaller than ~400 lines ²
• Code review your own code first ²

After an hour, and above roughly 400 lines, your ability to detect issues in a code review starts to decrease. So review little and often. The gaps around lunch, standup and before you head home are ideal. And remember, if someone’s put code up for a review, that review is blocking them doing other work. It’s your job to unblock them.

On TweetDeck, we actually try to keep reviews under 250 lines. It doesn’t sound like much, but this constraint applies pressure to make smaller, incremental changes. This makes breakages easier to detect and roll back, and leads to a very natural feature development process that encourages learning and iteration.

But the most important thing I’ve learned personally is that reviewing my own code is the best way to spot issues. I try to approach my own reviews the way I approach my team’s: with fresh, critical eyes, after a break, using a dedicated code review tool.

It’s amazing what you can spot when you put a new in a new interface around code you’ve been staring at for hours!

And yes, this list features science. The data backs up these conclusions, and if you’d like to learn more about scientific approaches to software engineering then I recommend you buy Making Software: What Really Works, and Why We Believe It. It’s ace.

For more dedicated information sharing, we’ve introduced regular seminars for everyone who works on a specific area or technology. It works like this: a team-member shares or teaches something to everyone else, and next time it’s someone else’s turn. Giving everyone a chance to speak, and encouraging a wide range of topics, is starting to produce great results.

If you’d like to run a seminar, one thing you could try to get started: run a point at the thing you least understand in our architecture session — thanks to James for this idea. And guess what… your onboarding architecture diagrams will help (and benefit from) this!

More, please!

There’s a few ideas here to get you started, but there are even more in a talk I gave this year called Frontend Archaeology, including a look at optimising for confidence with front-end operations.

And finally, thanks to Amy for proof reading this and to Passy for feedback on the original talk.