Posted by & filed under Leadership, XP.

Many software teams struggle with ever growing cost of change, and technical debt that risks overwhelming them. 

It’s always interesting to talk with folks and understand, how the system and incentives created this state. 

Often I hear from software engineers that management doesn’t give them permission for (or doesn’t prioritise) work such as tidying, refactoring, testing, observability.

Blaming management in this way may be accurate in many situations (they are indeed probably accountable for the system that created this outcome). However, the problem with blame is that it tends to short-circuit further thinking. How could the system be improved with the influence and control you do have, perhaps with minimal authority.

But what about from the management side? I often hear from engineering and product managers that they’re unaware, or learn too late, that folks were accumulating debt. It becomes visible when teams are no longer able to achieve goals.  

Management has a bad reputation in the tech community. I’ve been very fortunate to work almost exclusively with managers who genuinely care deeply about the people and teams they support; who want what’s best for them in the long term.

And yet, it’s still common, with the best of intentions, to create systems that result in unsustainable software development, and lots of technical debt. 

How does this happen? 

The risks are not very visible. If managers were given the option of twice as much this month, in exchange for nothing for the next 6 months, they probably wouldn’t take it (perverse incentives like annual bonuses notwithstanding). Would that it were so simple. 

The decks are stacked against sustainability. It requires active effort from managers to combat the natural incentives.

Here’s three practical ways you can help as a manager.

  1. Converse; don’t Coerce
  2. Encourage Healthy Habits
  3. Celebrate Sustainability

Converse; don’t Coerce

Get Curious

People will always want more than you’re able to achieve. Meaning we’re often starting from a position of saying no to lots of things that are very valuable; it’s faster to identify opportunities than it is to win them.

Most people don’t want to give bad news. Most people don’t want to hear bad news.  

This sets us up for miscommunication. 

<Manager> we need to do X this week
<Maker> Ok [unsaid: I’m going to risk a production outage by skipping testing]

If sufficient trust exists then it’s possible the risk will be vocalised, but the manager has not invited this feedback. Nor have they provided enough information for the maker to form their own opinion on the merits of the risk. 

It’s an instruction. There’s no curiosity as to what the impact or tradeoffs are, which shuts off thinking. 

It encourages risks. Some might infer an unsaid “…whatever it takes”. This could lead to taking risks such as skipping normal safety, or working long hours

It’s informationless. It’s a blanket “we need”. It doesn’t say what the value or tolerable costs are. The recipient has no information to make good judgements themselves with.

Things get a little better with even the slightest bit of curiosity.

<Manager> can we do X this week? 
<Maker> I think so [unsaid: cutting corners]
* next week *
<Manager> can we do Y this week?
<Maker> I think so [unsaid: that means they’ve decided not to clean up the debt from X]

At least the manager is now making it clear that there’s some degree of choice, although the power dynamic might make the choice seem illusionary

It’s a closed question. The manager’s likely to get yes or no answers. They’re not showing curiosity about the consequences of the team making X or Y happen. 

The manager is still asking the team to do things. Having to ask a team to do something is often a sign of a bug in the organisation. An organisational smell.

What’s stopping the team self-organising to the most important work? What information were the team missing to help them spot that X&Y were more important? What skills or perspectives are they missing to be able to make the best choices? 

Use Open Questions

Things improve further if we ask open questions:

<Manager> what would we drop if we tried to do X this week? 
<Maker> well we were planning on implementing the actions from last week’s post mortem, if we postponed those we might be able to do X.

or

<Manager> on a scale of 1-5, how risky would it be to release X this week? 
<Maker> Hmm, probably a 4, we’d not have any time to test or do a canary deployment so we’d lose our usual safety nets

Here the manager has asked open questions, and solicited information that might help make better decisions. Even if the manager then asks the maker to do the same thing, the risks have at least been taken more intentionally.

Prefer information over instructions

What if we didn’t ask the team to do X at all?

<Manager> I’ve learned that we could close a $n deal if we can ship X this week. 
<Maker> Hey that’s more than the total cost savings from what we were working on, let’s do X instead.

or

<Manager> I’ve learned that we could close a $n deal if we can ship X this week. 
<Maker> Hmm, but if we rush we risk breaking the service. Last time this cost us $10*n

However, this is somewhat optimistic. Even with all the information, people often have recency bias and eagerness to please. The power dynamic between manager and maker often exacerbates this.

Incentives conspire to eliminate slack time that would be used to keep development sustainable. 

Combining Curiosity and Information

<Manager> I’ve learned that we could close a $n deal if we can ship X this week. 
<Maker> Hmm, that sounds like a lot, I can get it done
<Manager> How risky is it on a scale of 1-5?
<Maker> Probably a 1, we’ve still got time to test it carefully. 
<Manager> What things are we dropping that we’d otherwise be doing?
<Maker> Oh we were going to refactor Y as changes in that area are slow
<Manager> Oh I thought Y was done months ago
<Maker> We shipped it months ago but have since noticed the design trips us up and we never have time to fix it with all these urgent opportunities.
<Manager> How much time has it cost us? 
<Maker> More than it takes to ship X just this month

Here the manager and maker have built a much richer understanding of the situation and tradeoffs. They are much better placed to make a decision than if the maker had simply been asked to ship X.

Encourage Healthy Habits

Team habits can help people escape the trap of needing permission from unsupportive or oblivious management. You will probably be unsupportive or oblivious at some point. Unintentionally coercing people is so easy. 

You can counter this by encouraging your team to adopt habits that will resist your efforts to coerce them into doing the wrong thing.

If your team practices TDD then you’d have to break a strong habit to encourage people to release a feature without tests.

If your team habitually drops everything to bring the build time down every time it creeps above 5 minutes, then they’re more likely to do the same even if you’ve asked them to deliver a feature.

If your team habitually prioritises work based on cost of delay, then they’ll likely ask you questions to find out the value before jumping on it. Even if you have omitted the information in your request.

Habits are hard to break. This can be a good thing. The right habits can add resistance to the forces that conspire to create creeping loss of sustainability. 

Celebrate Sustainability

If you publicly appreciate internal improvements you will signal for more.

If your only questions about the work are inquiries about when it will be finished, you’ll incentivise unsustainable working.

If your promo process only recognises features and customer impact, you’ll discourage sustainability.

Do you celebrate with the team as much when they halve their deploy time, as when the ship that flagship feature?

Do you thank folks for taking extra days to refactor and clean up the code?

Do you ask “how much more could we simplify it?” or “when will we be done?”

Do you ask “what if tried doing less?” or “how can we do more?”

Posted by & filed under Leadership, XP.

I’ve worked in and with teams who would identify as agile for many years. I’ve written about how I think the most interesting part of the agile manifesto is the “uncovering better ways” mindset. 

With this in mind, I’ve been pondering how the principles in the agile manifesto have aged. Where do (and don’t) I share the same principles. 

Given the “better ways” of developing software we’ve uncovered, can we “turn up the dials” on the principles as well as the ways of working? 

Principles tend to stand the test of time better than specifics. The agile manifesto still reads as an aspirational document today, but I think there’s room for improvement.  

Here’s my take on things I’d tweak in each principle; hopefully staying true to the values

Bettering Ourselves

Building working software is a powerful tool to achieve our ends. However, I wouldn’t consider it the highest priority of a team. If we can achieve our goals without software (which can be a liability) that’s even better. 

Moreover, if the software we’re asked to build is harmful for humanity, or counter to our organisational/personal values; avoiding that comes before satisfying the customer. The negative impact software can have on society is increasingly evident as more of our society is controlled by software.

“Bettering ourselves” (optimise for learning) is at the heart of agile to me. To borrow from Captain Picard, I’d state our highest priority as:

Our highest priority is to better ourselves, our organisation, and the rest of humanity. Continuous delivery of software is our tool to achieve this.

Learning we’re Wrong

We’re not passively waiting for requirements. We can take more ownership over our goals and understand customer needs ourselves. We’ve shifted from requirements to forming our own hypotheses of what customers need.

It’s not that requirements change, it’s our understanding that improves. We learn how we were wrong and adapt our goals in response.

We don’t need to build for a customer-proxy (stakeholder who wants the software). We’re can build products for people; they are our customers.  By meeting customer needs, we build a competitive advantage for ourselves.

We welcome learning that we’re wrong, even late in development. Changing our software in response to what we learn helps us meet customer needs and gives us a competitive advantage.

Continuous Delivery

The only thing worth changing in the continuous delivery principle, is that our tooling and platforms are so much better now than 20 years ago. We can deliver considerably faster and smaller.

A team can feasibly ship multiple times a day—given a good continuous delivery pipeline, reliable & fast tests, production safety net from canary deploys, and production observability. We’re no longer having to burn CDs with software builds.

Deliver increments to working software frequently, from a couple of hours to a couple of weeks, with a preference to the shorter timescale. 

All the Brilliant People

I’d replace project with product; due to the temporary-team fixed time/budget/requirements connotations of project.

I think longer lived product teams are more effective. I appreciate this won’t be for folks working in a project/consultancy context but I avoid those ;) 

I’d also expand to “all the brilliant people” as per Woody Zuill’s description of mob programming. Get everyone needed to achieve the goals together into the team, collaborating together every day. Minimising dependencies and handoffs.

All the brilliant people work together daily on the product. Developers, businesspeople, customers, designers, testers, ux, data scientists …  

Product Teams over Individuals on Projects

As per above, I prefer sustained investment in stable product teams over the project model.

Teams of motivated individuals build products. Give them the environment and support they need, and trust them to get the job done. 

Communication Quality over Efficiency

This is the only principle I feel like I’m watering down rather than turning up.

I think face to face conversation is still the highest bandwidth communication method. However, face to face communication is not always feasible for practical & inclusion reasons. Reasons such as geographic location, neurodiversity, family commitments, carbon footprints, and global pandemics! 

If face to face communication is infeasible then it’s not the most effective method of communication; even if it would be the highest bandwidth method.

I think the general principle is it’s better to trade off efficiency for quality of communication. Be willing to sacrifice time for better communication. You’ll save time in the long run. Prefer the highest bandwidth communication you can enable, even if it feels inefficient.

Prioritise communication quality over time efficiency. Learn the most effective method of conveying information to and within your team, for your context and each other’s individual needs. 

Results are the primary measure of progress

I don’t see working software as the primary measure of progress. We can ship oodles of working software every day and benefit nobody.

How are people’s lives enriched by our toil? How are our organisational goals furthered by our software? How is society better?

Have we achieved what we set out to? What have we learned from our experiments?

The less software needed the better!

The results of working software in production is the primary measure of progress. Get software into production early, starting with a walking skeleton.

Sustainability

I’d make the call for sustainbility more imperative. We don’t rely on static processes to keep development sustainable. It requires active effort and adaptation to combat the natural incentives in play.

We don’t go fast now at the expense of tomorrow. We strive to keep the cost of change curve low. 

Prioritise sustainable development over speed. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Technical Excellence

I find myself avoiding the words agile and agility since “agile” has become so semantically diffused it can create confusion.

Continuous attention to technical excellence and good design keeps cost of change low and lets us learn and adapt swiftly.

Simplicity

The principle of simplicity is timeless.

Diverse Teams

I’d add diversity to the description of teams that produce the best results. Self-organising teams with a monoculture are not going to produce the best ideas; there’ll be shared blind spots.

I think it can also be useful to expound on some of the things teams need to self-organise effectively.

The best architectures, requirements, and designs emerge from diverse teams that can self organise. Having the skills needed, clarity on their goals, with a high level of psychological safety

Double-loop Learning

I’d just add adjusting goals, in addition to tuning team behaviours. The most effective teams benefit from double loop learning. Learning-through-doing that our goals are wrong, or our assumptions when choosing our goals were wrong. 

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly, reviewing their goals based on what they have learned.

Your Team’s Principles

How do your principles differ? I’ve only thought about things I’d tweak from the manifesto principles above, not what I’d add. What would you add?

Posted by & filed under Leadership, XP.

What if we could visualise the cost of attrition?

Here’s a team. Someone leaves. We hire a replacement.
We get lucky and manage to find someone more skilled. Looks like we’re better off?

Really when someone leaves we lose all the relationships they had with the rest of the team as well. The team is a diminished more like 40% than the apparent 20% by their loss. It takes longer to rebuild the team than is apparent. Relationships take time.

It’s worse than that. The team probably wasn’t maximally-connected to start with. And it’s not just the interpersonal relationships that matter but the knowledge of tech and domain. A departure can break teams apart and organisational knowledge needs to be rebuilt.

Your organisation probably has multiple teams. Someone leaving your team reduces its connectedness to the rest of the organisation. Increasing the time to recover even with a swift new hire.

Internal mobility is less of a hit to the team’s connectedness due to pre-existing relationships. It also increases the whole organisations resilience by establishing more inter-team relationships.

Teams following the Isolated-individual model of work… (as opposed to collaborative work like pairing and collective ownership) …are particularly brittle & significantly impacted by staff churn.

How would we think about retention if we could visualise the full impact of someone leaving our team?

Beware looking at teams on a spreadsheet. If you have a hiring rate matching attrition rate it might look like the team health is maintained. It’s probably not.

Tracking tenure by team and average tenure in team can be interesting proxy indicators. Teams can be growing but have dropping tenure.

Bear in mind “All models are wrong, some are useful”. Sometimes teams benefit more from fresh ideas than the value of relationships lost in a change. Sometimes gaining someone who helps everyone else in the team form connections at a faster rate can accelerate the team.

 

This post is also available as a Twitter Thread

Posted by & filed under XP.

How many better ways of working have you uncovered lately?

This past year a lot of people have been forced into an unplanned and unwanted experiment. Many teams have had to figure out how to work in a remote-first way for the first time. 

I was privileged to be working in a team that was already distributed, making the transition for us easier than many found it. But adapting to work without office equipment was still new. 

Going without the offices that we were used to was uncomfortable in many ways. It was particularly difficult for those not privileged enough to have good working environments at home. 

While nobody would wish the challenges of the past year upon anyone, disruption to the way we were used to working, helped us uncover some better ways. Nothing groundbreaking. But some things were driven home.

Like how much easier video discussions are when everyone is fully visible, at the same size on the screen, and close-mic’ed. Better than the hybrid experience when part of the team are using a conference room with shared microphone and camera.

We also learned how bad our open plan office environment was for collaborative working. Compared with the experience of two people pairing over a video chat, each in their own quiet spaces.

We had some resilience because we were already distributed. How much more resilient might we have been, had we been more used to experimenting and adapting the ways we worked, safely.

The opening sentence of the agile manifesto doesn’t seem to be as much discussed as the values & principles. I think it’s arguably the most interesting bit.

“We are uncovering better ways of developing software by doing it and helping others do it.”

The Agile Manifesto

This brings three questions to mind that might be useful reflection prompts:

  1. How many better ways of developing software have you uncovered recently? Have we already found the best ways of developing software? I hope not!
  2. Are the people who are developing software in your organisation, the same people who are uncovering better ways of doing it? Or does management tell people how to work?
  3. When deciding how to work, are you thinking foremost of what’s easiest for you? or what’s most helpful for others?

We didn’t need a pandemic to try the experiment of no office, but for many folks it was the first time trying that experiment. What if we experimented more? Experimenting with self-selected, smaller, low-risk experiments?

How much has your team changed the way it works lately? Could you be stuck in a local maximum? Are your ways of working best for your team as it is now, or tradition inherited from those who came before, who may have been in a different context?

Every person is different. Every team is different. Transplanting ways of working from one team to another may not yield the best working environment. 

Ways of working often come from “things I’ve seen work before” or your preferences. Why would they be best for this team now? What are you doing to “uncover” what works best for you now?

Uncovering is a great metaphor. Move things around, remove some things, see what you discover! 

Teams often limit themselves by only trying experiments where they believe the outcome will be an improvement. Improvement is then limited by the experience and imagination of those present.

How about trying experiments where you expect the result to be no different? Or where you have no idea what will happen? Or even where you expect the result to be worse?

Here’s some practical experiments you could try in your team to see what you learn:

Try removing something

What activities, ceremonies, processes, and practices do you have? What would happen if you stopped doing one of them? They may seem like purely beneficial things. Even if they are, by trying to remove them you might learn more about their value (or lack thereof) to you. As well as their influence on your behaviour.

  • Imagine there’s No Offices (We’ve had this experiment forced upon us lately)
  • Imagine there’s No Meetings. Maybe we’d expect more miscommunications and wasted effort. Perhaps worth testing—maybe we’ll understand the value of meetings more clearly by abstaining for a time. Maybe we’d also learn to communicate asynchronously through writing more clearly. 
  • Imagine there’s No Estimates. Do you estimate all your upcoming tasks to measure progress? What would happen if you stopped doing it? If you’re required to give estimates what if you estimated everything as one day?
  • Imagine there’s No Staging. Do you test your changes in a staging environment for a time before promoting to prod? What if you didn’t? How would you be forced to adapt to mitigate the risks you had been mitigating in staging.
  • Imagine there’s No Branches. Do you work independently from others on your team? What if you had to all work on the same branch? How would you adapt?
  • Imagine there’s No Backlogs. What if you discarded everything you’re not doing now or next? What would happen? Maybe worth trying.
  • Imagine there’s No Managers. Do you know what your manager is doing? What would happen if they were unavailable for a while? How would your team need to adapt?

Removing things might seem risky and likely to make things worse. But what can we know about our resilience if we only do experiments where we think we know the outcomes.

It can be interesting to try removing something and see if we achieve similar outcomes without it.

It can help you understand the real value of the practice or process to your team. Help you understand your team better, and how things normally go right.

Try constraining something

Add an artificial limit on something you do often and see what you learn. Here’s some popular examples.

  • Keep tests green at all times
  • Implementation code may only be factored out of tests, not written directly. 
  • No more than 1 test in a commit
  • Deploy every time tests go green
  • Only one level of code indentation, or one dot per line
  • Only mock types you own
  • No fewer than 3 people work on a task if you usually work alone, or no more than 1 person working on a task if you usually work paired.
  • No more than n tasks in progress
  • Don’t start a new task until previous is in production.
  • No changing code owned by another team. 

Try the opposite

What ways of working do you tend to choose? What if you tried the opposite for a time and see what you learn? Try to prove it’s no worse and see what happens.

  • Do you always take on the team’s frontend tasks? What if anyone but you took them?
  • Work together in pairs? Try working apart individually.
  • Work individually? Try working together in pairs.
  • Feel we need to plan better? What if we just started without planning, together.
  • Hiring to go faster? What if you tried having fewer people involved instead.

Try the extreme 

An idea from extreme programming is to look for what’s working and then try to turn it up to the maximum; turn the dials up to 11. 

A few ideas:

  • Pair programming on complicated tasks? Try pairing on boring tasks too.
  • Pairing? Try the whole team worked together as an ensemble/mob.
  • Deploying daily? Try to deploy hourly.
  • Do you declare a production incident when there’s a problem with customer impact? What if every “operational surprise” were treated as an incident.

Reflecting

Only run experiments that your team all consent to. Set a time limit and a reflection point to review. 

Ask yourselves: What was surprising? Are we having more or less fun? What were the upsides? What were the downsides? Are we achieving more or less? 

Make it the default to revert to the previous ways of working. Normalise trying things you don’t expect to work, and reverting. Celebrate learning that something is not right for you. If every experiment becomes a permanent change for your team, then you’re not really experimenting you’re just changing. 

Be bold enough to try things that probably won’t work. You might uncover new insights into your team, your colleagues, and what brings you joy and success at work.

Posted by & filed under Leadership, XP.

Instead of starting from “how do we hire top talent?”, start from “what are our weaknesses?”

Why are you hiring? Are you hiring to do more, or are you hiring to achieve more?

Design your hiring process to find the right people to strengthen your teams’ weaknesses, rather than trying to find the best people. 

Instead of “how can we find the smartest people?” think about “how can we find people who will make our team stronger?”

People are not Talent. An organisation can amplify or stifle the brilliance of people. It can grow skills or curtail talent.

Often the language used by those hiring betrays their view of people as resources. Or, to use its current popular disguise: “talent”. 

“We hire top talent” “the candidate didn’t meet our bar”, “our hiring funnel selects for the best”. “we hire smart people and get out of their way”. We design “fair tests” that are an “objective assessment” of how closely people match our preconceptions of good.

The starting point for the talent mindset is that we want to hire the smartest people in the “talent pool”. If only we can hire all the smartest people, that will give us a competitive advantage!

If you focus on hiring brilliant people, and manage to find people who are slightly smarter on average than your competitors, will you win? Achievements in tech seldom stem solely from the brilliance of any one individual. Successes don’t have a single root cause any more than failures do. 

We’re dealing with such complex systems; teams that can capture and combine the brilliance of several people will win. 

With the right conditions a team can be smarter than any individual. With the wrong conditions a team can be disastrously wrong; suffering from over confidence and groupthink, or infighting and undermining each other. Hiring the right people to help create the right conditions in the team gives us a better chance of winning than hiring the smartest people.  

Talent MindsetWeaknesses Mindset
Find top TalentFind skills that unlock our potential
Grow TeamTransform Team
Help teams do more thingsHelp teams achieve greater things
People who can do what we’re doingPeople can do things we can’t do
Hire the best personHire the person who’s best for the team
People who match our biased idea of goodPeople who have what we’re missing
Fair tests & Objective assessmentEqual opportunity to demonstrate what they’d add to our team
Consistent processChoose your own adventure
Hire the most experienced/skilled candidateHire the person who’ll have the greatest impact on the team’s ability.
Number of candidates & Conversion rateHow fast can we move for the right candidate?
Centralised Process & BureaucracyTeams choose their own people
Grading candidatesCollaborating with candidates
Prejudging what a good candidate looks likeReflecting on our team’s weaknesses
FungibilitySuitability
Smart peoplePeople who make the team smarter
IntelligenceAmplifying others
Culture FitMissing Perspectives
People who’ve accomplished great thingsPeople who’ll unblock our greatness
Exceptional individuals. People we can grow and who will grow us
What didn’t the candidate demonstrate against checklistWhat would the candidate add to our team

Talent-oriented hiring

If we start out with the intent to find the “best people” it will shape our entire process.

We write down our prejudices about what good looks like in job descriptions. We design a series of obstacles^Winterviews to filter out candidates who don’t match these preconceptions. Those who successfully run the gauntlet are rewarded with an offer, then we figure out how to best put them to use. 

Hiring processes are centralised to maximise efficiency & throughput. We look at KPIs around the number of candidates at each stage in the funnel, conversion rates. 

Interviewing gets shared out around teams like a tax that everyone pays into. Everyone is expected to interview for the wider organisation.

Hiring committees are formed and processes are standardised. We try to ensure that every candidate is assessed as fairly as possible, against consistent criteria. 

We pat ourselves on the back about how we only hire the top 1% of applicants to our funnel. We’re hiring “top talent” and working here is a privilege. We’re the best of the best. 

Weaknesses-oriented hiring

There’s another approach. Instead of starting from our idea of talent and the strengths we’re looking for, start from our weaknesses. What’s missing in our team that could help us achieve more? 

Maybe we have a team of all seniors, frequently stuck in analysis. We need some fresh perspectives and bias for action.  

Maybe we are suffering from groupthink and need someone with a different background and new perspectives. Someone who can help generate some healthy debate?

Maybe we have all the technical skills we need but nobody is skilled at facilitating discussions. We struggle to get the best out of the team.

Perhaps we’re all fearful of a scaling challenge to come. We could use someone experienced who can lend the team some courage to meet the challenge.

Maybe those in the existing team specialise in frontend tech, and we’ve realised we need to build and run a data service. We could learn to do it, but bringing someone in with existing knowledge will help us learn faster.

Maybe we are all software engineers, but are spending most of our time diagnosing and operating production systems. Let’s add an SRE in the team.

Maybe we don’t even know what our weaknesses are—until we experience collaboration with the right person who shows us how to unlock our potential.

Identify your weaknesses. Use them to draft role descriptions and design your interview processes. 

Design your interviews to assess what the candidate brings to your team. How do they reduce your weaknesses and what strengths would they add? 

Leave space in the process to discover things people could bring to your team that you weren’t aware you needed. 

A talent-oriented process would assess how the candidate stacks up against our definition of good. A weaknesses-oriented process involves collaborating with the candidate, to see whether (and how) they might make your team stronger. 

If possible, have candidates join in with real work with the team. When this is not feasible for either side, design exercises that allow people on your team to collaborate with them. 

Pair together on a problem. Not what many companies call paired interviews where it’s really an assessor watching you code under pressure. Instead, really work together to solve something. Help the candidate. Bounce ideas off each other. Share who’s actually doing the typing, as you would if you were pairing with a colleague. Don’t judge if they solved the same problems you solved; see what you can achieve together.

A weaknesses-oriented process might mean saying no to someone eminently qualified and highly skilled; because you have their skills well covered in the team already. 

A weaknesses-oriented process might mean saying yes to someone inexperienced who’s good at asking the right questions. Someone whose feedback will help the experienced engineers in the team keep things simple and operable.

Why not both? 

It’s often worth thinking about when something good is not appropriate. There are rarely “best practices”, only practices that have proven useful in a given context.

At scale I can see the need for the talent-oriented hiring approach in addition to weaknesses-oriented.

Exposing all of your teams’ variety of hiring needs to candidates may create too much confusion.

You may well want a mechanism to ensure some similarity. A mechanism to select for those who share the organisational values. To find those with enough common core skills to increase the chances that they can move from team to team. Indeed, long-lived and continuously funded teams are not a given in all organisations.

If you’re getting thousands of applications a day you’ll probably want a mechanism to improve signal to noise ratio for teams wishing to hire. Especially if you don’t want to use education results as a filter. 

I suspect a lot of hiring dysfunction comes from people copying the (very visible) talent-oriented top-of-funnel hiring practices that big companies use. Copy-pasting as the entire hiring mechanism for their smallish team.

Start from reflecting on your weaknesses. Whose help could you use? Some of the practices from the talent-oriented approach may be useful, but don’t make them your starting point. Start from your weaknesses if you want strong teams.

Posted by & filed under Leadership, XP.

If you’re struggling with how to get to tidy code, fast feedback loops, joyful work. How to get permission, or buy-in. Try team habits.

Create working agreements of the form “When we see <observable trigger>, instead of <what we currently do> we will <virtuous action>”. This is a mechanism to pre-approve the desired action.

In last week’s “level up” newsletter Pat Kua likened developers asking product managers whether to do activities such as refactoring and testing, to a chef offering washing and cleaning to customers. i.e. it shouldn’t be a discussion.

I strongly agree, but I can see that some people might still be stuck with the how. Especially if you’re the only cook in the kitchen who seems to care that there is mess everywhere.  

I recently suggested every team should be paying attention to the delays in their feedback loops. Keeping time to production under 5 minutes. Running all programmer tests in a matter of seconds. 

Measuring what matters is a good step towards being able to improve it. But it’s not enough. Especially if your manager, product manager, or fellow developers seem unperturbed by the build time creeping up, the flaky tests in your build, or the accumulating complexity.

How we get stuck

Maybe you point to graphs and metrics and are met with “yes but we have a deadline next week” or “just do something else while you’re waiting for the slow build”. We easily get stuck at how to break out of seeking permission.

How can we become a team with code we’re proud of? How can we become a team that has fast effective feedback loops?

I’m a big fan of radiating intent rather than asking for forgiveness, but what if you’re the only person on your team that seems to care? Acting autonomously takes courage. It sends a positive signal to your coworkers, but it could easily be shut down by an unsupportive or unaware manager.

You might give up when your small solo efforts make little headway against a mammoth task. It’s easy to become despondent when a new teammate rants about the state of the huge codebase you’ve been refactoring bit by bit for weeks. “Ah, but you should have seen it before” you sigh. 

People get stuck asking for permission. Asking permission, or taking a risk to do the right thing every time is wearisome. “Just do it” is easy to say but is risky. Especially if you have leaders who are reluctant to let go of control. 

Is quality code only something that teams with sympathetic managers can achieve? Are fast feedback loops something only very talented engineers can achieve? By no means!

A trap that lots of teams fall into is making the safe, sustainable, faster path improvement path the special case. We create systems whereby people have to ask for permission to clean and to tidy. We need to change the expectations in the team. It should be that tolerating a messy workspace is seen as the deviance that needs permission or risk taking. Not vice versa.

How to change this? Propose a new a beneficial habit to your team. 

When we see <observable trigger>,
instead of <what we currently do>
we will <virtuous action>”

How can you convince your team to try a new way of working? Make a prediction of the benefit and propose running an experiment for 2-4 weeks, after which the team can review. 

Good habits give teams superpowers. Habits can pre-approve the right actions. Habits unstick us from “doing things” to take necessary actions, when they’re needed, continuously. 

What do I mean by good habits? Refactoring constantly, making the build a bit faster every day, talking with customers. Feedback loops that increase our chances of success. 

Make them team habits and the onus will be on managers and team members to convince each other to not do the good things rather than the other way round.

Examples of good habits

TDD

TDD is great for habit forming—it creates triggers to do the right thing. When we see a red failing test we trigger the habit of implementing new behaviour. We never have untested code because the habit we’ve formed for writing code is in response to a failing test. When we see a green test suite we trigger the habit of refactoring. Every few minutes we see a green test suite and trigger our habit: spending some time thinking about how the code could be tidier, easier to understand, and then making it so.

Sure we could convince our pair to skip the refactoring step, but the onus is on us to make the case for deviating from the sensible default, habitual path.

When we write code, instead of starting to implement the feature we will write a failing test first”

When we see a red test suite, instead of writing lots of code we will write the simplest possible code to make it pass”

When we see a green test suite, instead of moving on to the next capability we will stop and do at least one refactoring”

Zero Tolerance of Alerts

Are you suffering from over-alerting? Do you have flickery production alerts that go off frequently and then resolve themselves without action? Use them as a trigger for a habit. Agree as a team that every time an alert fires the whole team will down tools, investigate and work together. Every time an alert happens: come up with your best idea to stop it going off again for the same reason.

If that’s too extreme you could try with a nominated person investigating. The point is to make a habit. Pre-agree expected, good, behaviour in response to a common trigger. Then there’s no need to seek permission to do the right thing, plus there’s some social obligation to do the right thing.

When we see an alert fire, instead of waiting to see if recovers by itself we will stop and investigate how to stop it firing again”

Max Build Time

Is your build getting slower and slower? You’ve made it measurable but it’s still not helping? Create a habit that will improve it over time. e.g. Every time you see the build has got a minute slower the next task the team picks up will be making the build at least two minutes faster.

Now the onus is on someone to argue that something else is more important at the time. The default, habitual, behaviour is to respond to a trigger of a slowed build with investing time in speeding it up.

When we see the median build time cross the next whole minute, instead of ignoring it we will prepend a task to our ‘next’ queue to reduce the build by at least 2 minutes”

Time Waste Limit

Do you feel like your team is drowning in technical debt? Maybe you’re dealing with such a mountain of technical debt. Perhaps it feels like nothing you do will make a difference? Build a habit to make it better in the most impactful areas first. e.g. Make wasting an hour understanding an area of the codebase trigger improvement. When you realise you’ve wasted an hour, update a shared team tally of hours wasted by area of code. When you go to add a tally mark and there’s already a couple of marks there, drop what you were going to do and spend the rest of the day tidying it up.

If you pre-agree to do this then the default behaviour is to make things a bit better over time. Permission seeking is needed to skip the cleanup.

When we notice we’ve spent an hour trying to understand some complicated code , instead of pressing ahead we will record the time wasted”

When we notice our team has wasted 5 hours in one area of the code instead of finishing our feature we will spend the rest of the day tidying up”

Spend Limits

Does your team’s work always take far longer than expected? Are you struggling to deliver a slice of value in a week? Maybe even when you think a story can be completed in a day it ends up taking two weeks? Make a habit triggered by length of time spent on a given story. Keep a tally of elapsed days. When it hits double what we expected, the team stops and has a retrospective to understand what they can learn from the surprise. 

When we have spent 5 days on one story instead of carrying on we will hold a team retrospective to understand what we can learn to help us work smaller next time”

How to help your team form a good habit

Asking your team to try to form a new habit is something anyone can do. You don’t need to be a manager. 

  1. Look for a suitable trigger mechanism, in day to day working.
  2. Think of a virtuous action that could be taken upon that trigger.  
  3. Propose an experiment of a habit to try, combining items 1+2
  4. Write it down, as a working agreement for the team.

    When we see a green test suite, instead of moving on to the next capability we will stop and do at least one refactoring”
  5. Review your working agreements next retrospective.

There’s lots of good habits that you can steal from other effective teams (such as TDD).

A team committing to try to form a habit will make a more effective experiment than one person trying to do the right thing against the flow. 

Let me know how it goes. What resistance do you run into? What are your team’s most powerful habits?

Posted by & filed under sql.

I love SQL, despite its many flaws.

Much is argued about functional programming vs object oriented. Different ways of instructing computers.

SQL is different. SQL is a language where I can ask the computer a question and it will figure out how to answer it for me.

Fluency in SQL is a very practical skill. It will make your life easier day to day. It’s not perfect, it has many flaws (like null) but it is in widespread use (unlike, say, prolog or D).

Useful in lots of contexts

As an engineer, sql databases often save me writing lots of code to transform data. They save me worrying about the best way to manage finite resources like memory. I write the question and the database (usually) figures out the most efficient algorithm to use, given the shape of the data right now, and the resources available to process it. Like magic.

SQL helps me think about data in different ways, lets me focus on the questions I want to ask of the data; independent of the best way to store and structure data.

As a manager, I often want to measure things, to know the answer to questions. SQL lets me ask lots of questions of computers directly without having to bother people. I can explore my ideas with a shorter feedback loop than if I could only pose questions to my team.

SQL is a language for expressing our questions in a way that machines can help answer them; useful in so many contexts.

It would be grand if even more things spoke SQL. Imagine you could ask questions in a shell instead of having to teach it how to transform data 

Why do we avoid it?

SQL is terrific. So why is there so much effort expended in avoiding it? We learn ORM abstractions on top of it. We treat SQL databases as glorified buckets of data: chuck data in, pull data out.

Transforming data in application code gives a comforting amount of control over the process, but is often harder and slower than asking the right question of the database in the first place.

Do you see SQL as a language for storing and retrieving bits of data, or as a language for expressing questions?

Let go of control 

The database can often figure out the best way of answering the question better than you.

Let’s take an identical query with three different states of data.

Here’s two simple relations with 1 attribute each. a and b. With a single tuple in each relation. 

CREATE TABLE a(id INT);
CREATE TABLE b(id INT);
INSERT INTO a VALUES(1);
INSERT INTO b VALUES(1);
EXPLAIN analyze SELECT * FROM a NATURAL JOIN b;

“explain analyze” is telling us how postgres is going to answer our question. The operations it will take, and how expensive they are. We haven’t told it to use quicksort, it has elected to do so.

Looking at how the database is doing things is interesting, but let’s make it more interesting by changing the data. Let’s add in a boatload more values and re-run the same query.

INSERT INTO a SELECT * FROM generate_series(1,10000000);
EXPLAIN analyze SELECT * FROM a NATURAL JOIN b;

We’ve used generate_series to generate ten million tuples in relation ‘a’. Note the “Sort method” has changed to use disk because the data set is larger compared to the resources the database has available. I haven’t had to tell it to do this. I just asked the same question and it has figured out that it needs to use a different method to answer the question now that the data has changed.

But actually we’ve done the database a disservice here by running the query immediately after inserting our data. It’s not had a chance to catch up yet. Let’s give it a chance by running analyze on our relations to force an update to its knowledge of the shape of our data. 

analyze a;
analyze b;
EXPLAIN analyze SELECT * FROM a NATURAL JOIN b;

Now re-running the same query is a lot faster, and the approach has significantly changed. It’s now using a Hash Join not a Merge Join. It has also introduced parallelism to the query execution plan. It’s an order of magnitude faster. Again I haven’t had to tell the database to do this, it has figured out an easier way of answering the question now that it knows more about the data.

Asking Questions

Let’s look at some of the building blocks SQL gives us for expressing questions. The simplest building block we have is asking for literal values.

SELECT 'Eddard';
SELECT 'Catelyn';

A value without a name is not very useful. Let’s rename them.

SELECT 'Eddard' AS forename;
SELECT 'Catelyn' AS forename;

What if we wanted to ask a question of multiple Starks: Eddard OR Catelyn OR Bran? That’s where UNION comes in. 

SELECT 'Eddard' AS forename 
UNION SELECT 'Catelyn' AS forename 
UNION SELECT 'Bran' AS forename;

We can also express things like someone leaving the family. With EXCEPT.

SELECT 'Eddard' AS forename 
UNION SELECT 'Catelyn' AS forename 
UNION SELECT 'Bran' AS forename 
EXCEPT SELECT 'Eddard' AS forename;

What about people joining the family? How can we see who’s in both families. That’s where INTERSECT comes in.

(
  SELECT 'Jamie' AS forename 
  UNION SELECT 'Cersei' AS forename 
  UNION SELECT 'Sansa' AS forename
) 
INTERSECT 
(
  SELECT 'Sansa' AS forename
);

It’s getting quite tedious having to type out every value in every query already. 

SQL uses the metaphor “table”. We have tables of data. To me that gives connotations of spreadsheets. Postgres uses the term “relation” which I think is more helpful. Each “relation” is a collection of data which have some relation to each other. Data for which a predicate is true. 

Let’s store the starks together. They are related to each other. 

CREATE TABLE stark AS 
SELECT 'Sansa' AS forename  
UNION SELECT 'Eddard' AS forename  
UNION SELECT 'Catelyn' AS forename  
UNION SELECT 'Bran' AS forename ;
 
CREATE TABLE lannister AS 
SELECT 'Jamie' AS forename 
UNION SELECT 'Cersei' AS forename 
UNION SELECT 'Sansa' AS forename;

Now we have stored relations of related data that we can ask questions of. We’ve stored the facts where “is a member of house stark” and “is a member of house lannister” are true. What if we want people who are in both houses. A relational AND. That’s where NATURAL JOIN comes in.

NATURAL JOIN is not quite the same as the set based and (INTERSECT above). NATURAL JOIN will work even if there are different arity tuples in the two relations we are comparing.

Let’s illustrate this by creating a relation pet with two attributes.

create table pet as 

CREATE TABLE pet AS 
SELECT 'Sansa' AS forename, 'Lady' AS pet
UNION SELECT 'Bran' AS forename, 'Summer' AS pet;

Now we have an AND, what about OR? We have a set-or above (UNION). I think the closest thing to a relational OR is a full outer join. 

CREATE TABLE animal AS SELECT 'Lady' AS forename, 'Wolf' AS species UNION SELECT 'Summer' AS forename, 'Wolf' AS species;
SELECT * FROM stark FULL OUTER JOIN animal USING(forename);

Ok so we can ask simple questions with ands and ors. There are also equivalents of most of the relational algebra operations

What if I want to invade King’s Landing?

What about more interesting questions? We can do those too. Let’s jump ahead a bit.

What if we’re wanting to plan an attack on Kings Landing and need to consider the routes we could take to get there. Starting from just some facts about the travel options between locations, let’s ask the database to figure out routes for us.

First the data. 

CREATE TABLE move (place text, method text, newplace text);
INSERT INTO move(place,method,newplace) VALUES
('Winterfell','Horse','Castle Black'),
('Winterfell','Horse','White Harbour'),
('Winterfell','Horse','Moat Cailin'),
('White Harbour','Ship','King''s Landing'),
('Moat Cailin','Horse','Crossroads Inn'),
('Crossroads Inn','Horse','King''s Landing');

Now let’s figure out a query that will let us plan routes between origin and destination as below

We don’t need to store any intermediate data, we can ask the question all in one go. Here “route_planner” is a view (a saved question)

CREATE VIEW route_planner AS
WITH recursive route(place, newplace, method, LENGTH, path) AS (
	SELECT place, newplace, method, 1 AS LENGTH, place AS path FROM move --starting point
		UNION -- or 
	SELECT -- next step on journey
		route.place, 
		move.newplace, 
		move.method, 
		route.length + 1, -- extra step on the found route 
		path || '-[' || route.method || ']->' || move.place AS path -- describe the route
	FROM move 
	JOIN route ON route.newplace = move.place -- restrict to only reachable destinations from existing route
) 
SELECT 
	place AS origin, 
	newplace AS destination, 
	LENGTH, 
	path || '-[' || method ||  ']->' || newplace AS instructions 
FROM route;

I know this is a bit “rest of the owl” compared to what we were doing above. I hope it at least illustrates the extent of what is possible. (It’s based on the prolog tutorial). We have started from some facts about adjacent places and asked the database to figure out routes for us.

Let’s talk it through…

CREATE VIEW route_planner AS

this saves the relation that’s the result of the given query with a name. We did this above with

CREATE TABLE lannister AS 
SELECT 'Jamie' AS forename 
UNION SELECT 'Cersei' AS forename 
UNION SELECT 'Sansa' AS forename;

While create table will store a static dataset, a view will re-execute the query each time we interrogate it. It’s always fresh even if the underlying facts change.

WITH recursive route(place, newplace, method, LENGTH, path) AS (...);

This creates a named portion of the query, called a “common table expression“. You could think of it like an extract-method refactoring.  We’re giving part of the query a name to make it easier to understand. This also allows us to make it recursive, so we can build answers on top of partial answers, in order to build up our route.

SELECT place, newplace, method, 1 AS LENGTH, place AS path FROM move

This gives us all the possible starting points on our journeys. Every place we know we can make a move from. 

We can think of two steps of a journey as the first step OR the second step. So we represent this OR with a UNION

JOIN route ON route.newplace = move.place

Once we’ve found our first and second steps, the third step is just the same—treating the second step as the starting point. “route” here is the partial journey so far, and we look for feasible connected steps. 

path || '-[' || route.method || ']->' || move.place AS path;

here we concatenate instructions so far through the journey. Take the path travelled so far, and append the next mode of transport and next destination.

Finally we select the completed journey from our complete route

SELECT 
	place AS origin, 
	newplace AS destination, 
	LENGTH, 
	path || '-[' || method ||  ']->' || newplace AS instructions 
FROM route;

Then we can ask the question

SELECT instructions FROM route_planner 
WHERE origin = 'Winterfell' 
AND destination = 'King''s Landing';

and get the answer

                                 instructions                                   
-------------------------------------------------------------------------------
Winterfell-[Horse]->White Harbour-[Ship]->King's Landing
Winterfell-[Horse]->Moat Cailin-[Horse]->Crossroads Inn-[Horse]->King's Landing
(2 rows)

Thinking in Questions

Learning SQL well can be a worthwhile investment of time. It’s a language in widespread use, across many underlying technologies. 

Get the most out of it by shifting your thinking from “how can I get at my data so I can answer questions” to “How can I express my question in this language?”. 

Let the database figure out how to best answer the question. It knows most about the data available and the resources at hand.

Posted by & filed under Leadership, Star Trek, XP.

Growing up, an influential television character for me was Jean Luc Picard from Star Trek the Next Generation.

Picard was a portrayal of a different sort of leader to most. Picard didn’t order people about. He didn’t assume he knew best.  He wasn’t seeking glory. In his words: “we work to better ourselves, and the rest of humanity”. The Enterprise was a vehicle for his crew to better themselves and society. What a brilliant metaphor for organisations to aspire to.

My current job title is “Director of engineering”. I kind of hate it. I don’t want to direct anybody. I represent them as part of my first team. People don’t report to me; I support people. My mission is to clarify so they can decide for themselves, and to help them build skills so they can succeed. 

Director is just a word, but words matter. Language matters.

Picard was an effective leader in part due to the language he used. Here’s a few lessons we can learn from the way he talked.

“Make it So!”

Picard is probably best known for saying “make it so!”

This catchphrase says so much about his leadership style. He doesn’t bark orders. He gives the crew the problems to solve. He listens to his crew and supports their ideas. His crew state their intent and he affirms (or not) their decisions. 

I think “Make it so” is even more powerful than the more common “very well” or “do it”, which are merely agreeing with the action being proposed.

“Make it so” is instead an agreement with the outcome being proposed. The crew are still free to adjust their course of action to achieve that outcome. They won’t necessarily have to come back for approval if they have to change their plan to achieve the same outcome. They understand their commander’s intent.  

And of course it’s asking for action “wishing for a thing does not make it so”

“Oh yes?” 

Picard’s most common phrase was not affirming decisions, but “oh yes”. Because he was curious, he would actively listen to his crew. He sought first to understand.  

It’s fitting that he says this more than make it so. Not everything learned requires action. It’s easy to come out of retrospectives with a long list of actions. I’d rather we learned something and took no actions than took action without learning anything. 

“Suggestions?” 

In “Cause and Effect” (The one with Captain Fraiser Crane) there’s an imminent crisis. The Enterprise is on a collision course with another ship and are unable to maneuver. What does Picard do? Resort to command and control? No; despite the urgency he asks for suggestions from the crew. Followed by a “make it so” agreement to act.

Asking for suggestions during a crisis requires enough humility to realise your crew or team is collectively smarter than you are. Picard trusts his team to come up with the best options. 

He is also willing to show vulnerability, even during a crisis. His ego doesn’t get in the way. 

In this episode, the crew did not automatically follow the suggestion of the most senior person in the room. The solution to the crisis is eventually found in the second suggestion, after they tried the first. They succeeded because they had diverged and discovered options first, before converging on a solution.

The crew were aware of the other options open to them, and when the first failed they acted to try another (successful) option. Crucially, they did not wait for their captain to approve it. There wasn’t time to let the captain make the decision, but they were free to try the other options because they’d been told to “make it so” not “do it”.

To achieve the best outcomes as a team, intentionally seek divergent ideas first before converging on a decision. Avoid converging too soon on an idea that sounds promising, when others in the group may have better ideas. If your first choice doesn’t work out you will be able to try the other ideas that came up. 

“Nicely done!”

Picard made sure his crew knew when he thought they had done well. Even when they violated orders! He was not one to blindly follow orders himself and he praised his crew when they violated orders for good reasons.

Data’s violation of orders is met not with a reprimand but a “nicely done!”; when Data questions it Picard responds “The claim ‘I was only following orders’ has been used to justify too many tragedies in our history”

How different might the tech industry be if more people carefully considered whether doing what they’ve been told is the right or wrong thing to do? 

Posted by & filed under Java.

Java 16 brings Pattern Matching for instanceof. It’s a feature with exciting possibilities, though quite limited in its initial incarnation. 

Basic Pattern Matching

We can now do things like 

Object o = "hello";
if (o instanceof String s) {
    System.out.println(s.toUpperCase());
}

Note the variable “s”, which is then used without any casting needed.

Deconstructing Optionals

It would be even nicer if we could deconstruct structured types at the same time. I think it’s coming in a future version, but I’m impatient. Let’s see how close we can get with the current tools.

One thing we can do is use a function on the left hand side of this expression.

Let’s consider the example of dealing with unknown values in Java. There’s a fair amount of legacy to deal with. Sometimes you’ll get a value, sometimes you’ll get a null. Sometimes you’ll get an Optional<T> , sometimes you’ll get an Optional<Optional<T>> etc.

How can we make unknowns nicer to deal with?

We could create a little utility function that lets us convert all of these forms of unknown into either a value or not. Then match with an instanceof test.

Object unknown = Optional.of("Hello World");
assertEquals(
       "hello world",
       unwrap(unknown) instanceof String s
               ? s.toLowerCase()
               : "absent"
);

Thanks to instanceof pattern matching we can just use the string directly, without having to resort to passing method references i.e optional.map(String::toLowercase) 

The unwrap utility itself uses pattern matching against Optional to recursively unwrap values from nested optionals. It also converts nulls and Optional.empty() to a non-instantiatable type for the purposes of ensuring they can never match the above pattern.

static Object unwrap(Object o) {
   if (o instanceof Optional<?> opt) {
       return opt.isPresent() ? unwrap(opt.get()) : None;
   } else if (o != null) {
       return o;
   } else {
       return None;
   }
}
static class None {
   private None() {}
   public static final None None = new None();
}

Here’s several more examples, if you’d like to explore further.

Deconstructing Records

What about more complex structures? Now that we have record types, wouldn’t it be great if we can deconstruct them to work with individual components more easily. I think until more powerful type patterns exist in the language we’ll have to diverge from the instanceof approach. 

I previously showed how we could do this for records we control, by having them implement an interface. What about records we do not control? How can we deconstruct those? 

This is about the closest I can get to what I’d hope would be possible as a first class citizen in the language in future.

record Name(String first, String last) {}
Object name = new Name("Benji", "Weber");
If.instance(name, (String first, String last) -> {
   System.out.println(first.toLowerCase() + last.toLowerCase()); // prints benjiweber
});

It takes a record (Name) and a lambda where the method parameters are of the same types as the component types in the record. It deconstructs the record component parts and passes them to the lambda to use (assuming the record really matches).

We could also use as an expression to return a value, as long as we provide a fallback for the case when the pattern does not match.

Object zoo = new Zoo(new Duck("Quack"), new Dog("Woof"));
 
String result = withFallback("Fail").
    If.instance(zoo, (Duck duck, Dog dog) ->
       duck.quack() + dog.woof()
    ); // result is QuackWoof

So how does this work? 

If.instance is a static method which takes an Object of unknown type (we hope it will be a Record), and a lambda function that we want to pattern match against the provided object.

How can we use a lambda as a type pattern? We can use the technique from my lambda type references article—have the lambda type be a SerializableLambda which will allow us to use reflection to read the types of each parameter. 

static <T,U,V> void instance(Object o, MethodAwareTriConsumer<T,U,V> action) { 
 
}

So we start with something like the above, a method taking an object and a reflectable lambda function.

Next we can make use of pattern matching again to check if it’s a record.

static <T,U,V> void instance(Object o, MethodAwareTriConsumer<T,U,V> action) {
   if (o instanceof Record r) {
	// now we know it's a record
   }
}

Records allow reflection on their component parts. Let’s check whether we have enough component parts to match the pattern.

static <T,U,V> void instance(Object o, MethodAwareTriConsumer<T,U,V> action) {
   if (o instanceof Record r) {
       if (r.getClass().getRecordComponents().length < 3) {
           return;
       }
 
	 // at this point we have a record with enough components and can use them.
   }
}

Now we can invoke the passed action itself 

action.tryAccept((T) nthComponent(0, r), (U) nthComponent(1, r), (V) nthComponent(2, r));

Where nthComponent uses reflection to access the relevant component property of the record.

private static Object nthComponent(int n, Record r)  {
   try {
       return r.getClass().getRecordComponents()[n].getAccessor().invoke(r);
   } catch (Exception e) {
       throw new RuntimeException(e);
   }
}

tryAccept is a helper default method I’ve added in MethodAwareTriConsumer. It checks whether the types of the provided values match the method signature before trying to pass them. Avoiding ClassCastException

interface MethodAwareTriConsumer<T,U,V> extends TriConsumer<T,U,V>, ParamTypeAware {
   default void tryAccept(T one, U two, V three) {
       if (acceptsTypes(one, two, three)) {
           accept(one, two, three);
       }
   }
   default boolean acceptsTypes(Object one, Object two, Object three) {
       return paramType(0).isAssignableFrom(one.getClass())
               && paramType(1).isAssignableFrom(two.getClass())
               && paramType(2).isAssignableFrom(three.getClass());
   }
 
   default Class<?> paramType(int n) {
       int actualParameters = method().getParameters().length; // captured final variables may be prepended
       int expectedParameters = 3;
       return method().getParameters()[(actualParameters - expectedParameters) + n].getType();
   }
}

Then put all this together and we can pattern match against Objects of unknown type, and deconstruct them if they’re records matching the provided lambda type-pattern.

record Colour(Integer r, Integer g, Integer b) {}
 
Object unknown = new Colour(5,6,7); // note the Object type
 
int result = withFallback(-1).
    If.instance(unknown, (Integer r, Integer g, Integer b) ->
       r + g + b
    );
 
assertEquals(18, result);

Degrading safely if the pattern does not match

Object unknown = new Name("benji", "weber");
 
int result = withFallback(-1).
    If.instance(unknown, (Integer r, Integer g, Integer b) ->
       r + g + b
    );
 
assertEquals(-1, result);

Code for the record deconstruction and several more examples all in this test on github. Hopefully all this will be made redundant by future enhancements to Java’s type patterns :)

Posted by & filed under ContinuousDelivery, XP.

“We got lucky”—it’s one of those phrases I listen out for during post incident or near-miss reviews. It’s an invitation to dig deeper; to understand what led to our luck. Was it pure happenstance? …or have we been doing things that increased or decreased our luck?   

There’s a saying of apparently disputed origin: “Luck is when preparation meets opportunity”. There will always be opportunity for things to go wrong in production. What does the observation “we got lucky” tell us about our preparation? 

How have we been decreasing our luck?

What unsafe behaviour have we been normalising? It can be the absence of things that increase safety. What could we start doing to increase our chances of repeating our luck in a similar incident? What will we make time for? 

“We were lucky that Amanda was online, she’s the only person who knows this system. It would have taken hours to diagnose without her” 

How can we improve collective understanding and ownership? 

Post incident reviews are a good opportunity for more of the team to understand, but we don’t need to wait for something to go wrong. Maybe we should dedicate a few hours a week to understanding one of our systems together? What about trying pair programming? Chaos engineering?

How can we make our systems easier to diagnose without relying on those who already have a good mental model of how they work? Without even relying on collaboration? How will we make time to make our systems observable? What would be the cost of “bad luck” here? maybe we should invest some of it in tooling? 

If “we got lucky” implies that we’d be unhappy with the unlucky outcome, then what do we need to stop doing to make more time for things that can improve safety? 

How have we been increasing our luck? 

I love the extreme programming idea of looking for what’s working, and then turning up the dials

Let’s seek to understand what preparation led to the lucky escape, and think how we can turn up the dials.

“Sam spotted the problem on our SLIs dashboard”

Are we measuring what matters on all of our services? Or was part of “we got lucky” that it happened to be one of the few services where we happen to be measuring the things that matter to our users? 

“Liz did a developer exchange with the SRE team last month and learned how this worked”

Should we make more time for such exchanges or and personal learning opportunities? 

“Emily remembered she was pairing with David last week and made a change in this area”

Do we often pair? What if we did more of it?

How frequently do we try our luck?

If you’re having enough production incidents to be able to evaluate your preparation, you’re probably either unlucky or unprepared ;)

If you have infrequent incidents you may be well prepared but it’s hard to tell. Chaos engineering experiments are a great way to test your preparation, and practice incident response in a less stressful context. It may seem like a huge leap from your current level of preparation to running automated chaos monkeys in production, but you don’t need to go straight there. 

Why not start with practice drills? You could have a game host who comes up with a failure scenario. You can work up to chaos in production. 

Dig deeper: what are the incentives behind your luck?

Is learning incentivised in your team, or is there pressure to get stuff shipped? 

What gets celebrated in your team? Shipping things? Heroics when production falls over? Or time spent thinking, learning, working together?

Service Level Objectives (SLOs) are often used to incentivise (enough) reliability work vs feature work…if the SLO is at threat we need to prioritise reliability. 

I like SLOs, but by the time the SLO is at risk it’s rather late. Adding incentives to counter incentives risks escalation and stress. 

What if instead we removed (or reduced) the existing incentives to rush & sacrifice safety. Remove rather than try to counter them with extra incentives for safety? 🤔