As I was navigating the circuitous course from the dev team to the ops team, I thought back with nostalgia on my time at a younger organization, using a smaller team. That old shop was small, but it’s projects were still ambituous, with two live products handling > 1k CCU.
So what is it I was missing about the old days with a small team? Really it’s a matter of ownership and agility. When a developer has to work on several pieces, from client to server to infrastructure to monitoring, it leads to a team that is extremely competent and efficient. Having insight into the performance ramifications of that new server feature being designed, or having the forethought to instrument as you go are invaluable things! Unfortunately, trying to spread this discipline amongst a more strongly partitioned culture seems like a real challenge. Not to mention the reduction in meteor factor (aka information silos) and increased ownership. It also fosters system level thinking across the team - not just with the top of the pyramid (CTO, Sr Director Eng, etc)! This increased responsibility at the system level will automatically lead to simpler, easier to grok systems. No sane developer is going to knowingly trap themselves in a rats nest of a dozen disparate systems - instead you’ll grow something organic and integrated. Something that’ll be much easier to brain dump onto the next superstar that’s brought into your slowly growing, slightly-smaller-than-needed team.
This leads to the question, can a small team have large, successful products? Absolutely! There are a number of notable examples to be found around the ol’ interwebs. I’d wager that keeping a lean team of stellar developers will pay off in spades over time.
There’s also a psychological aspect that shouldn’t be overlooked here. When you have a small amount of people fully engaged, those people are going to feel like a crucial part of the organization, and feeling needed is really nice. Contrast this with a bloated team that only generates enough work to fill 50-65% of their time - websurfing isn’t going to inspire life altering, with-you-till-the-bitter-end loyalty.
It can be tricky to find legitimately competent generalists, but it’s well worth the search. I’ll take five well-rounded superstars over thirty siloed specialists any day of the week.
Let’s take a look at a hypothetical scenario:
“Damn, it’s really hard to identify bottle necks in this system…”
The Small Team Solution
- I set up a graphite installation (that I provision on AWS myself)
- I hook up metrics to the JVM based applications
- I wire up host level metrics in the chef recipes for deployment
- I instrument the other moving pieces to report
- Behold graphs, notice correlations
- I send out an email with some summary findings and suggestions for improvements
- Teammates interpret the data, make suggestions
The Bloated Version
- I ask around to see if the ops team is using configuration management software
- I find out they are, but it’s kept hidden and separate from developers
- I decide to at least instrument my piece of the system
- I hook up metrics
- I need a host to run graphite… where can I get a host?
- I bring up the need, an order is put in for another host - it should be available in a couple weeks
- Finally,the JVM piece is instrumented
- I discover the bottle neck is another team’s component
- That team doesn’t have the bandwidth to look at it, I can’t instrument their code for them
- I try to stub out a mockup of the slow components
- I find it’s too stateful, with too many dependencies and api entry points to mock
- Tears ;-(
Granted this is not exactly comparing apples to apples, the point remains that giving developers the independence, trust, and authority to just do things counts for a lot.
Mutiply the savings from this one scenario over the space of a project’s lifetime, and that really adds up.