For many years, Agile has mainly been about the iterative and incremental process, a couple of new roles, etc. Over the last years, as Agile has increasingly become about companywide transitions[note]To understand why I consistently use the term agile transition where many other use agile transformation you should read: Forget All About Agile Transformations[/note] in large companies, scaling this process and roles has gotten a lot of attention. Agile has gone from what could be illustrated like in the diagram to the left[note]www.scrum.org[/note] to the one to the right[note]www.scaledagileframework.com[/note] on the figure below.
However, I have observed that while we try to deal with the increased organizational complexity by applying scaling frameworks, we still struggle with the obvious misalignment between agile and traditional company culture and complex IT-architectures. After agile transitions in large companies, we often still have:
- very large projects with many people
- people working on several projects at the same time
- component teams handling a systems layer or a component determined by system architecture rather than user perspective
- people in many locations working together
- handovers from business to IT in the portfolio process
- longer release cycles
- etc
We call whatever it is we do Agile, even though it is not significantly more flexible or productive than what we had before.
Having so much focus on scaled agile processes and numerous roles, I think that we have somewhere on the way forgotten the very early rules of thumb/prerequisites for productive agile teams: self-directed, cross-functional teams of 7±2 people delivering value to user in 1-4 week cycles.
In my opinion, the initial scaling focus is the wrong way to attack the complexity of large companies. We should instead start focusing even more on the core of agile: the agile team — rather than making the existing structure and projects agile by scaling.
This is no vendetta against scaling frameworks of any kind, because I am totally aware that not all products can be delivered by a single team. I am also aware of the complexity of the IT stacks in large companies and how they resemble a “blackberry thicket,” as I have heard it mentioned. IT-systems are complex and tightly coupled in most large organizations.
I am also aware that the dominating management paradigm over the last centuries has dictated functional silos and all decisions are made up the organizational hierarchy. I accept that it is the culture in which the team initially live and that all teams are part of something larger and therefore many decisions are made above team level.
But I think it is scary how far from agile, we can end up, when faced by the “invisible” company culture and the complexity of IT architecture, we resign. It is as if, we do not see how interlinked culture, process, architecture, organizational structure etc are, so we try to solve them one at a time. Since Agile is the new black, we start there and “solve” the rest by making compromises in regard to “how agile we want to be”.
The result is that the agile, we get, does not deliver on the flexibility and productivity we were promised, and the employees in our organization, who really understands agile, get very frustrated.
If we want to gain higher agility and productivity, we must be ambitious and solved our basic issues instead of designing a process that attempts to handle them. The results are delivered by agile teams and that is a very different organizational construct compared to functional hierarchies, so we must focus on nurturing an organizational structure designed to support agile teams. I therefore suggest that we reverse the order of solution: form the teams as ideally as possible and let them solve the problems — cultural or technological — together with leadership.
If a hierarchical organization could create the culture and IT-architecture we have today, then an agile organization will also change those over time. We must accept that Agile is an organizational design, not just a process.
In this article I will concentrate on giving you the reasons why it is a good idea to approach an agile transition as a process of organizational redesign rather than a change of process. The organizational redesign is the foundation for gaining agile productivity and it is a prerequisite for establishing an optimal agile process. The latter will be the subject of my next article.
Organizational structure for delivering value
From an organizational design perspective, what are the primary factors for teams’ ability to deliver value?
Disregarding the option of just hiring the most brilliant people in the world and instead accepting that most of us have to start working with whatever we got, let’s focus on what’s possible. I have found that primarily five structural factors decide a teams ability to deliver:
- size
- competences/skills
- allocation
- location
- stability
All are important and ignoring/sacrificing just one can ruin all efforts in the other dimensions. Let’s start by addressing them one at a time.
Team size
Lot’s of research and experience reports has been made in the area of optimal team size (many are collected here[note]https://agilepainrelief.com/notesfromatooluser/2016/10/choosing-the-team-size-in-scrum.html[/note] ). Hackman and Vidmar (1970), were among the first. Based on team member satisfaction, they reached the optimal team size of 4,6 (left figure).
This finding has been repeated several times since and everybody seems to agree that you shouldn’t put more than 8-9 people in your teams. Actually if you only aim for highest productivity, you should aim for a much lower team size as the research of Larry Maccherone[note]https://www.infoq.com/presentations/agile-quantify[/note] shows to the right.
Productivity per team member is highest with three members and drops to below on third in teams of more than 20. As you can see other parameters like quality, responsiveness, predictability grow up till nine team members and then flatten before they even drop. The sum of all parameters point to an optimal team size of 5-9 being about 40% better than +20 people.
When Jeff Bezos talk about 2-pizza teams at Amazon, he is talking exactly about this.
Of course team size has everything to do with finding the best compromise between on one hand having the skills and knowledge needed to make the best decisions, and on the other hand minimizing the coordination overhead entailed in being too many. The figure to the rights illustrate, how coordination increases by square of the number of people in the team ((n2-n)/2). In a group of 20, there are 190 connections, so it should come as no surprise that efficiency will be low and people by instinct try to split up in smaller teams.
Creating teams of 7±2 people is typically not scary in itself — until the next two factors are added, so let us take the discussion on how to do this after discussing those.
Team member competences
According to Troy Maggenis[note]http://focusedobjective.com[/note] at Agile 2017 only one scenario justifies having more than 5-9 people on your team: If the only alternative is moving parts of a teams’ tasks to another teams’ backlog. Handling dependencies between teams is more expensive than the internal communication overhead of being more people. Solution: Add a person with the competences needed or provide what is needed for the team to learn those skills, if you want to get most out of your team.
Maybe it is intuitive for you that being dependent on another teams planning when you need someones competences, creativity etc, will slow you down a lot, otherwise the next couple of paragraphs will add to that understanding.
A consequence of the high price of inter team coordination is that a team must be truly cross-functional to perform. It must be able to do all aspects needed to bring value to the users — from customer/market understanding to all parts of development and releasing the product technically and commercially.
This is probably a big mouthful if you’re sitting in a large, mature organization with complex IT-architecture and highly specialized people. Both the business side and the IT-organization probably have more than 9 functional silos, which means that there is not room for every specialized competence in teams of 5-9, and in several of the silos there are not people enough to be part of each team.
The answer to this is as obvious as it is hard to implement: Agile team members naturally need to be much more T-shaped[note]https://officeninjas.com/what-makes-t-shaped-employees-at-work/[/note] than they are today, to be able to handle every aspect of the teams’ responsibility within the team. To make this competence transition, leaders have to be able to trust that the people they hired are capable of learning more things, and let them learn on the job in their new cross-functional team. Leaders should also ensure expert assistance while employees are learning. I have seen this handled by keeping some experts outside teams during the transition to agile and then later, let them join the teams full time. Another way is to form guilds around each competence and invest time in letting team members across teams share and learn.
It is not a painless transition, but necessary, and in my experience the vast majority of employees will thrive on getting the opportunity to learn new skills and take on a broader responsibility in their team. As for the company, it will benefit from an improved productivity higher than the difference between small and large teams.
Team member allocation
The third important factor, when creating optimal conditions for your teams, is looking at how much you allocate each team member to the team. Being part of more than one project imposes context switching costs. Weinberg’s[note]Weinberg, G.M. Quality Software Management: Vol. 1 System Thinking. New York. Dorset House, 1992. [/note] research found switching costs as in table below. These numbers were confirmed in Larry Maccherone’s large study, however his data only contains 50-100% allocated team members.
What Weinberg proves, is that you get less than half the productivity from the people, you ask to be on four projects, because they pay context switching costs: Every time they get back to one project, they not only have to recap what they did the last time they were in that project. They also have to catch up on what everybody else have been doing while he/she was away. No wonder they have difficulties in engaging fully in any project and contributing with their full experience. This is truly waste. To make it even worse: In every organization I have asked, they realize that the people on four projects or more, are the most experienced and qualified people in the organization.
Trying to create teams across a whole organization with all team members 100 percent allocated, is typically also a great challenge to large organizations even though you should think that they have enough people to take from. The reason is that traditional organizations have practiced to build deeply specialized employees for years and make sure to use the most qualified person every time they have a task that requires specialized knowledge.
Forming teams with 100 percent allocation will reveal every bottleneck in your organization, because you won’t know in which team to put your specialists — they are needed several places. You should take this transparency as a gift and force yourself to allocate even specialists 100 percent to one team, because your reluctance is just the sign of the vulnerability you have already in your organization.
The result of this exercise is not only the saved switching cost, it is also creating a much more robust organization that no longer need to plan the work around bottlenecks or fear that specialists will leave the company.
Co-located teams
In understanding the importance of co-location of teams, I am particularly inspired by Alistair Cockburn[note]http://alistair.cockburn.us/ &
Alistair Cockburn : “Agile Software Development – The Cooperative Game”, 2007[/note]. He has illustrated the richness of communication using different channels based on research by Scott Ambler (figure to the right). The underlying data prove that face-to-face at a whiteboard is four times as effective as an email conversation and more than twice as effective as video conversation. Who hasn’t tried being in an email conversation for days, just to find out that that it took five minutes to solve in front of the whiteboard?
Communication is important for productive development — much more than most realize. In his book, Cockburn highlights the cost and energy associated with detecting that someone else has information you need or has the need what you know.
If the information you need is on a different floor/building, it takes much more work and time to get it, compared to sitting next to each other (see figure below). You are actually much less likely to ask questions, if you cannot see the person. Distance makes us choose to struggle longer alone and settle for a poorer solution, rather than getting out of our office.
This pertains to the knowledge that you’re conscious about sharing, but often you’re not conscious about how you can help others or others can help you. When we are working, we subconsciously pick up traces of sounds etc around us, which sometimes will trigger us to ask for or provide information, that would otherwise not have been passed on. Cockburn calls this osmotic communication, and it is also about non-verbal communication eg sensing your colleagues’ joy or frustration and reacting to it.
Larry Maccherone has also looked into the effects of co-location in his research, and to his (and my) surprise, he has not been able to measure these effects. Looking closer into his own data, he has found that the best performing distributed teams are within the same time zone and are mainly distributed because they often work from home. His explanation of what contradicts his own (and my!) experiences of being co-located is that there are too many distractions at work. So be aware of those.
When organizations form teams across multiple locations, it is not because they do not know that it has a cost. In my experience, they do it, because their organization is spread around the globe as a result of acquisitions, or the desire to have access to more/cheaper competent employees. When they at the same time have the assumption that the people at the headquarter know the business/systems/technology etc better, they want them on every team to secure that insight is present on the team. I won’t neglect that it can be necessary for a period of time, but in my experience both headquarter and distant employees suffer from the distance.
In some of the agile transitions, I have been part of, we have started out with distributed teams, because we shared the assumption that people from the headquarter were needed on all teams to build up competence. Every time regardless if the second location was Ukraine or India, we found huge productivity gains, when we dared to create single location teams. Once we merged two Denmark-India distributed teams into two co-located teams, the danish team alone kept the combined velocity of the two previous distributed teams from sprint one. The Indian team took a little longer to gain speed but ended higher than a distributed team.
There may be many reasons why it is not be possible to co-locate all of your teams, even though you wish to. I am certainly no expert in distributed teams, but if you have distributed teams and want more out of them, I can refer to one of the best books on the subject, “Work Together Anywhere” written by Lisette Sutherland[note]Lisette Sutherland > Work Together Anywhere: A Handbook on Working Remotely—Successfully—for Individuals, Teams, and Managers[/note].
Stable teams
The fifth and final factor important for teams’ ability to create value, is the time they have been working together. There are numerous team maturity models and they are more or less all inspired by Bruce Tuckman’s model[note]Tuckman, Bruce W (1965). “Developmental sequence in small groups”. Psychological Bulletin. 63 (6): 384–399.[/note] from 1965 with the stages: Forming, Storming, Norming and Performing.
The curve to the right illustrates how you can expect performance to grow over time as the team develops: Establishing rules for collaboration and decision making, learning about each other’s strengths and weaknesses, beginning to trust each other, valuing team goals over individual aspirations etc
Again, Larry Maccherone has tried to put numbers on the improvements over time. He has found a 20 percent increase in performance within the first year together, and that it can be increased by about 50 percent if you conduct regular retrospectives.
Maccherone finds that teams stagnate after a couple of years and need new blood to continue growth. This is confirmed in lots of other research. However, having stable teams for too long time is rarely the issue in most companies.
Yearlong stability might be the hardest of the five factors to implement in reality — especially if you have large growth which naturally entails bringing new people on board into teams, where they can learn from more experienced team members.
In my experience, keeping teams together can be particularly difficult in companies without a clear vision and corporate and product strategies. A lack of clear direction results in rapidly shifting priorities and an urge to reallocate people very often. I hope than the promise of 20-50% more outcome from stable teams is just another reason to be even clearer about the direction of the company.
One way to improve this, is by involving Product Owners and teams, and delegating influence on the direction of their product to them. Let them be the de facto owners of their products, not just resources that are shifted around projects.
No matter for what reason you want to reteam, you should be aware that it can be done more or less intelligently in regard to team dynamics. Tuckman says that breaking up a team will force the team to start over again in the forming phase. Heidi Helfand however shares a lot of insight into how to do it better in the book she is writing[note]https://leanpub.com/dynamicreteaming[/note].
Conclusions on the five structural factors
Redesigning your organization to small, autonomous, 100% allocated, cross-functional, co-located and stable teams is not easy. Actually it is very hard. However, I hope, I have provided evidence enough to make you give it a hard try to begin your agile transition by creating optimal conditions for your teams, rather than making many compromises like those I mentioned in the beginning.
If you add the potential of each factor, a rough guess is that creating optimal conditions for a team alone more than doubles value creation compared to a traditional organizational designs. Then add the effects of the agile process, which I will come back to in my next article.
Another way I have often tested the validity of the research above is by asking people to think back to the period of their working life, where they felt happiest and most valuable as an employee. The time from which they tell the most happy stories. The time where they think best of their colleagues. After thinking for a few minutes and having thought through, how it was at that time, I ask them to tell me how many of the 5 factors were present, and a very huge majority remember a small, cross/functional, focused, co/located and stable team — or at least 4 of the factors.
I have been so lucky to be part of teams like that both as a developer, a leader and an independent agile coach and I still clearly remember how productive we were on all occasions compared to when the factors were not present. To me that’s as good proof as any formal research.
What follows from redesigning your organization?
The organizational structure that follows establishing the teams in concord with the five factors is very different from the traditional functional hierarchy (figure below) with project constructs. Organizational focus and employee loyalty will follow the products and value streams that the teams are responsible for, rather than the functional silos that they are used to belong to.
Changing this is a very clear signal from leadership that they are committed to a cultural change. They clearly show that their own role is also under change and that influence is delegated to the forefront of the organization: the teams that deliver the product.
Besides sending this very clear signal about starting to change themselves, leadership provides the best possible conditions for the organization to flourish. Teams can in fact succeed within clear boundaries because they have the competences and mandate to deliver maximum value.
Reorganizing also forces the organization to face complex IT-portfolios and start to decouple the “blackberry thicket”. Otherwise teams risk breaking each others’ chance for success. It cannot be postponed because teams are empowered to remove impediments, and system dependencies will probably be their #one impediment. If they cannot decouple themselves they will raise impediments, and following the commitment to the teams, the change of IT-architecture must happen.
Culture and IT-architecture will change as a result of the organizational redesign.
How to redesign your agile organization?
Concrete methods for organizational redesign of course is a topic of an article in itself and I hope to get back to share my experiences, arguments, techniques, hits etc in more detail. Until then, let me give you the headlines of my approach to organizational redesign that enables you to create optimal conditions for teams — and therefore for your business:
- Start with user empathy. Identify which different users you have and then how they look at your product(s). Split your product(s) into sub-products as far as you can — based on the users’ needs.
- Go through all known future deliveries and assign them to the sub-products to get an idea of needed capacity
- Do the same for systems maintenance
- Take the sub-products that require more than two teams and try to split again
- Then an only then choose an appropriate scaling framework for the multi-team sub-products
- Assign team members to create best possible compliance to the rules of thumb above
- Plan for building T-profiles
- Keep reacting to impediments raised by the teams
Good luck redesigning your agile organization.
—