Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Let’s look at some hypothetical examples you may have actually witnessed with your engineering teams.
We have one server, but probably need two.
Let’s use Kubernetes.
We need to count the number of messages
Let’s use Redis.
We want to build a batch-based system that updates twice a day.
Lets use Rust.
Do the engineers not know what they are doing? Yes and no. They could write code, sure. They could probably pass a few Leetcode tests, for what that’s worth, but they have lost a key aspect to what makes an engineer great. Mediocre engineers confuse difficulty with value. They want to prove their cleverness by deploying heavyweight machinery to solve lightweight problems. The result? Bloated systems that cost more to maintain than they ever save, brittle architectures that fail under real-world conditions, and a culture where complexity is mistaken for progress.
The need to ask why
Asking why is a critical skill. And it comes with experience.
Why Kubernetes? They might argue it was to future proof the product and ensure it could scale. The addressable user-space for this product was small. It would only need to be updated a handful of times a day. Maybe we would need a total of 2-3 servers, behind load balancers, maybe not? All managed by a single team.
Why Redis?They need for a cache. A cache that had to be fast. It would store the previous responses made by users and store a count of the messages. Given the infrequent access patterns and no specific performance requirements (in this hypothetical examples) this was puzzling. They might argue it would be needed for later parts of the product, but could not articulate what they would be. To compound this none, let’s imagine none of the team had experience in Redis.
Why Rust?They could argue it was to ensure we could again scale effectively. The team might be obsessed with future proofing a solution. But did they know what the future actually looked like? No, they just knew it had to scale and that it had to be fast. Neither of which were likely product features and never asked for.
Let’s now imagine this was a cloud deployment. That costs money.
Annual Run Cost
$8,500
Development Cost
$16,000
Cost after 5 years
> $100,000
Without asking why they embarked on a path that will cost the business over $100k in 5 years time (predicting that most enterprise solutions have a shelf life of between 5 and 8 years). This takes into account cloud storage and compute costs, the initial development costs and then the total cost and support after 5 years.
Take the same approach and scale this up in organisations having thousands of developers. We are not looking at some very large and worrying numbers.
Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.
Donald Knuth – Communications of the ACM, Volume 17, Number 12, page 671
He’s absolutely right. It’s nice when something is fast. It feels like we did the right thing.
I read the book Real-Time Java Programming many years ago (when I spent my days just writing code…oh how I miss those days). It talks about this and what “realitime” actially means. It’s not immediate and it isn’t about speed; it’s fundamentally about predictability and ensuring tasks complete within specified time constraints, or deadline. And that could be milliseconds or a month.
So why are the whys not asked or answered? It’s a mixture of poor management, wanting to play with the latest technical solutions and a lack of commercial awareness. What are the teams allowed to get away with? I’m a big believer in hiring smart people, giving them the tools they need to work and then getting out of their way. But that doesn’t mean it’s a free for all. Every why needs to be answered. Put a weak engineering manager in a role and the team will ride roughshod over them in no time as they accumulate a shiny new collection of cool new technology.
- Start small
- Demo often
- Ask why
You must be logged in to post a comment.