The Agile Manifesto [1], created in 2001, brought about a significant shift in the development of (software) products. The values and principles in the manifesto have since evolved and expanded, and we continue to discover better ways to implement them. Overall, the changes have been positive and continue to benefit the industry.
This article discusses the journey we have collectively taken over the past few decades and predicts what is to come. It also highlights the varying levels of maturity that different teams possess, as well as the obstacles that may impede their ability to create value now and in the future.
This discussion is relevant to every team and organization, whether they face challenges typical of previous decades, current challenges, or future challenges. Those who fail to rise to the challenge risk being outperformed by those who do.
In this article, I will refer to the decades as if they represent clear milestones of progress. However, this is a rough oversimplification of what has happened in the Agile community, as well as for individual organizations and teams.
My intention is not to suggest that organizations and teams struggling with issues associated with earlier decades are old-fashioned or doing things wrong. Each organization and team faces unique challenges and has its legacy to contend with. However, I have learned that referring to 2000s-level teams, from a change perspective, is much more potent than addressing structural team problems. The latter is too easy to ignore or make excuses for.
I’ve found that simplifying things can also be helpful for organizations and teams when assessing themselves and planning for the future. Understanding ultimate agility can also help to skip some of the intermediate steps that those of us who have been onboard all along had to take because we didn’t know any better. That’s also why I try to predict what future decades will bring.
As you read through this, I encourage you to be honest about where your organization or team currently stands and to consider the most ambitious predictions before making your next move in Agile. Going all-in will be a significant challenge, as it requires technological, organizational, and personal changes, as well as unlearning old habits.
The first decade of the century
The software development industry underwent a significant transformation when agile development methods challenged the traditional waterfall process.
One of the most significant changes was the shift in focus towards the development team (see table below). Previously, products were developed in projects where developers were typically assigned part-time from their department based on their technical specialization. However, with agile development, a small, co-located, full-time allocated team that could build the product independently of others became the center of gravity. Those four characteristics form the optimal conditions for teamwork. As a result, the development process became much more efficient in delivering the features [2].
Another significant change was the focus on working software, where especially Extreme Programming [3] paid attention to automated testing and continuous integration and delivery. That meant that instead of getting one final verdict from the users at the end of the yearlong project, the team could get continuous feedback from frequent releases. In the 2000s, a great team did that every four weeks. The team got daily feedback on the quality of what they had built from both automated test and integration, which ensured a very high quality for those teams that reacted to it and kept bug rates close to zero. The higher quality leads to higher efficiency because the team was rarely hit by “technical debt.”
The frequent releases enabled a much closer interaction with users, but few benefitted from that. This is mainly because they didn’t take the last step and put their work into production but had to settle with feedback from proxies for the real users—typically the Product Owner or internal business people.
That helped move away from a fixed requirement scope, which was the default of pre-2000, to letting the Product Owner continuously adjust and prioritize the backlog to maximize (their perception of) the user’s value. The backlog consisted of a long list of User Stories (the format below was popularised by Mike Cohn in 2004 [4]). These small independent pieces of user-focused functionality were prioritized and delivered in increments, allowing user value optimization (see below).
However, the POs’ degrees of freedom were typically quite limited because the development resources were still subject to traditional project governance, where problems were analyzed, scoped, and estimated as projects prioritized at a corporate level.
As a result, the development teams were never part of the initial analysis and, therefore, rarely had a deeper understanding of the user’s needs. Consequently, the team contribution was developing (the green/yellow circle) what was defined elsewhere in the organization—just like a traditional pre-2000 project organization. (illustration below, a simplified version of Agile Planning Circles [5].)
At that time, it was really challenging to get to the point where we systematically released working software every four weeks. As I remember it, we struggled with
- resource bottlenecks, meaning that team members were partially allocated to our team
- the technology to do continuous delivery (CD) hardly existed yet
Since we’ve never worked with a user, we still thought that our most valuable contribution was typing fast at the keyboard.
The CD issues were solved long ago, but many organizations still struggle to create optimal conditions for teams, as mentioned above, within their project paradigm. On the other hand, who’s not caught in the pre-2000s in some part of their life? I struggle to keep up with the development in fashion, music, sports, etc.
The twenty-tens
The 2010s agile community was dominated by three different waves: DevOps, Discovery, and Scaling.
DevOps
DevOps brought operations into the development team, allowing the team to iterate even faster than in the 2000s. Releases every two weeks, every week, and even continuously became the norm, and the high frequency of feedback improved learning about the business and quality.
DevOps removed the typical handover from development to operations, which positively affected quality because it became so easy for those who wrote the code originally to fix and release. This also created a larger shift in the feeling of ownership for the software they were developing because their commitment was not defined as the duration of a project but as the entire product lifecycle—at least in principle.
Discovery
Through the first decades of the century, research showed that traditional development has failed to meet user needs. Well-known examples are the Standish Groups research (2002) showing that 2/3 of features were rarely or not used in 4 big products (see right), numerous similar research from Microsoft, Booking, Netflix [6], and even a successful company like Google is “bragging” about failing applications. Three hundred products that they’ve terminated after years of development [7].
The conclusion is that it is hard for everybody to foresee what the best solution is for the user. Consequently, much focus was put into involving users in the development process.
Eric Ries published Lean Startup in 2011, and Jeff Gothelf & Josh Seiden published Lean UX in 2013, both describing a user-centric approach with collaboration with real users in rapid iterations to validate every assumption. Design Thinking primarily attributed to IDEO/Stanford was incorporated into software development, especially with the book “Sprint: How to solve big problems and these new ideas in just five days” by Jake Knapp (2016), who describes a process used in Google Ventures (and now all over the world) to get from a problem statement through understanding, ideation, prototyping and validation with real users in just five days. Parallel to this, many methods supporting continuous discovery have emerged in the following years.
By the turn of the decade, Discovery had become a de facto part of a modern development process. This means that the development team has been further expanded by people with insight into design and creative skills. They no longer work in parallel with the software developers, but all team members are involved in helping the user get their job done. Focus shifted from delivering features to creating user outcomes, i.e., ensuring users can achieve what they need.
Jeff Patton introduced a new way to order the team backlog in 2014: The User Story Map. It transformed the one-dimensional list of User Stories from the 2000s to two dimensions, where each horizontal slice was intended to hold the user stories that created outcomes for the user—not just some features.
To fully take advantage of discovery, organizations should ask teams to deliver desired outcomes or let them figure out and decide how to create the outcome for the users, as illustrated in the figure below.
Progress from 2000s to 2010s
Now, one should think that adding DevOps and Discovery to every team would make them grow. However, that has not been the case since tooling for development, operations, and discovery has improved enormously over the decade and software developers master more technologies. So fewer specialties are needed.
The average team size in 2020 is smaller than 20 years before. Their iterations are much shorter, and they learn much about the problem they solve from real users and validate that the solution fits user needs within the iterations.
DevOps and Discovery is a natural extension of what happened in the 2000s (see table below):
- more parts of developing a software product are integrated into the team itself
- iterations with feedback have become shorter
- the teams have gotten closer to understanding the user’s need
- the team takes larger responsibility for the value creation for users
Despite significant advancements in creating user value, many organizations struggled to implement DevOps and Discovery in the 2010s and beyond.
One reason was that the needed skills were rare: designers were rare, and operations in a team are very different from those in a traditional Operations department: in DevOps, as much operation as possible is automated, hence done by a developer.
Another was that organizations stuck to traditional governance. An alternative type of governance that benefits from DevOps and lets teams learn from discovery and delivery was not well described at that time (illustrated below).
I also suspect that managers felt uncomfortable by handing decision power to teams. The project triangle of fixed scope, cost, and time feels more like being in control.
Scaling
The third development in the 2010s, scaling, tries to solve a different problem: Most software products today are much larger and more complex than a single team can handle. The best-known frameworks for scaling are LeSS (2008), SAFe (2011), Nexus (2015), and Scrum@Scale (2017), with SAFe coming out as the winner.
With the risk of oversimplifying things, it is common for all these approaches to take their outset in the winning agile approach of the 2000s: Scrum. They all attempt to solve the scaling challenge by adding further events, roles, and organizational structures to optimize the development process. SAFe has gone even further and attempts to solve every problem in a development organization, including a seemingly traditional management layer. That may be the reason why it was the most successful: it looked familiar to top management and, therefore a SAFe solution 😉
Scaling is definitely an issue that must be handled in most organizations, but is extending Scrum the right way to go?
The apparent consequence of the way the frameworks are structured is that
- several parts of the development process are taken out of the team again – or more correctly, Discovery and DevOps did not become integral team responsibilities, but rather roles outside the teams
- iterations are longer because many teams must coordinate. In SAFe, for example, 8-12 weeks
- teams are further away from their users, for instance, because of roles added outside the development team
- teams have no direct line of sight to their value creation because they only build part of the value delivered
I believe that these approaches are not a step forward but rather a step backward, as many of the advancements made during the 2000s and 2010s are being lost. I understand that those who sell and use these frameworks may disagree, but I have not yet witnessed teams within agile scaling frameworks who can learn about a potential user outcome one day and release a solution the next, as some of the best teams in the 2010s were able to do.
I acknowledge that, in some cases, organizations and software can be complex, making it necessary to implement a framework to release a cohesive solution. However, this means that they may not be taking full advantage of the advancements made during the 2010s. There is nothing inherently wrong with this as many organizations are caught here. Similarly, as mentioned earlier, many organizations are still struggling to create optimal conditions for their teams.
The twenties
Although many organizations have difficulty taking advantage of the technological advancements made in the 2000s and 2010s, those that can should continue striving to maximize the value they create with the resources they have available.
If we look beyond the financial success of Scrum and scaling hereof, what would then be the steps to take into the twenties for development organizations?
I believe that it is worth considering the progress we’ve made over the last twenty years: broader teams, shorter iterations, closer to users, and more responsibility for value creation.
Following these four trajectories, my predictions for the 2020s and beyond will be:
Broader team
With the introduction of Agile, development transformed from something controlled by a process and a manager to become a team responsibility. The development team was only responsible for a little part of the whole process, as illustrated in green in the figure below.
In the 2010s, the team’s responsibility expanded to operate the solution more smoothly and efficiently (DevOps) and better fulfill user needs (Discovery). The figure below illustrates this by adding the orange and blue boxes to the team’s responsibility.
Looking in both directions of the process, the next natural step is for teams to take responsibility for the strategic direction of the (part of the) product they are working on (red box in the illustration below). This means that the team needs skills in product management, i.e., knowing the market, the competitors, the customers, etc., and deciding on desired outcomes themselves.
The team should also take responsibility for making money on the product (purple box) and, hence, master marketing and sales.
Both are substantial steps. Hard? Probably just as hard as adding discovery and operations in the 2010s. Advantageous for value creation? Let me get back to that shortly.
Shorter iterations
Are shorter iterations possible when teams of the 2010s can iterate shorter than two weeks? Naturally, not substantially in time, but many teams already release daily to get fast feedback.
However, notice what happened beyond shortening iterations in time through the 2010s. What was validated in an iteration went from building a feature to discovering user needs and validating that the new functionality also solved their actual needs. Is that an improvement? Remember the research from Standish Group, Microsoft, etc., above.
As suggested before, embedding the product strategy in the team’s responsibility will also allow them to verify strategic assumptions at a speed never experienced before when strategic decisions were taken at a slower cadence, e.g., yearly, and then subject to traditional governance.
Let me give a real-life example: I worked for an insurance company where insurance had a base coverage with supplementary coverages. The strategic assumption was that it would cost revenue if they allowed users to deselect supplements through their website. So initially, it was only allowed for users to add supplements, but out of their own initiative, the team responsible for the website created an experiment where deselection of one supplement on one insurance was allowed, and they got the surprising result that amount of additions of supplements doubled as a result of this while very few were deselected. Needless to say, the strategic assumption and, therefore, the whole solution changed after that.
Doing experiments at a strategic level allows for unprecedented strategic agility in shorter iterations. However, that requires a tight coupling of product strategizing and discovery/delivery to get real user feedback, so why not move that responsibility into the team?
I’ve written about this before based on what I’ve named the Agile Planning Circles (APC). These articles describe how strategizing can be tightly integrated with product development to create more value [8].
Strategic planning can change, as project planning was challenged around the turn of the century.
Closer to users
Can the team get closer to users than when they have already done discovery? First of all, the number of methods for discovery has grown enormously over the years. A great reference is Bland/Osterwalder: Testing Business Ideas). Also, tools to support discovery have improved to the degree that teams can see and analyze how users interact, from individually in real time to detailed statistics on cohorts.
The real difference is, however, what these insights can be used for. Having access to data on operations, user data, user behavior, etc., a development team of the 2020s has unique opportunities to understand what drives revenue and cost either directly, through data, or by doing small experiments like the one described above.
Coupling the team’s ability to create fast feedback with the strategizing process means that the time for validating a business opportunity is short—there is no long governance process to delay learning.
Therefore, the team backlog should not only contain what has been decided to do but also any relevant opportunity identified. As I described in an article, one of the artifacts that can do that is the Impact Map by Gojko Adzic (2012). This and other similar maps fx. Teresa Torres Opportunity Tree (2021) adds another layer to the team backlog.
The Impact Map (illustrated right) expresses the assumptions about how features (the red squares) can fulfill goals (yellow). For example, it is assumed that push updates will grow earnings from mobile advertising because push updates help super-fans to come back more often. The illustrations also show how the map can maintain different paths to the goal with other deliverables or via other stakeholders.
In the same article, I describe how the simple backlog of the 2000s is being replaced with an impact map, user story map, and sprint backlog going forward.
More responsibility for value creation
All three trajectories discussed above have the potential to increase the team’s value creation, so could the 2020s bring more? Imagine that we extend the responsibility of online marketing and in-application sales.
As argued above, developers have direct access to many data types and create low-cost experiments to learn faster. They can monitor the buying decision process for individual users or test different options in parallel, e.g., different pricing, different presentations, different payments, etc. Because they are developers, they can combine data without having to wait for others.
Is this going too far? This is already the case for many B2C products, where marketing and sales processes are driven by technology, e.g., social media and new sales models like freemium. Blake Bartlett is the frontrunner, and he named it product lead growth.
Will that extend to more complex B2B products? Technology will undoubtedly develop and make it possible to manage with a team, but will buying behavior change this decade? It will likely happen more slowly for traditional B2B, which has a legacy of mature technology, personal sales relationships, etc. Still, we see it already in the native software-as-a-service solutions. Examples of this are the collaboration tools we implemented to cope with COVID.
I have also seen older B2B solutions start to implement upselling within an existing contract, landed in traditional ways—a land’n’expand strategy? To a high degree, this is influenced by the team’s insight into usage and possibilities for changing technology to support it.
If teams take responsibility for marketing and sales, they take responsibility for business impact. They become the actual owners of their (part of the) product. They generate revenue themselves; they are the cost. They will move toward becoming a small enterprise in itself.
Progress from 2000s to 2010s
Continuing the direction that product development has taken since the pre-2000s, my prediction is that teams will expand from being a development team focussing on optimizing the outcomes for the user through their product to becoming the actual owners of the product. These product teams will take care of the entire lifecycle, from strategizing, attracting new users, continuously improving outcomes, operating the system, etc.
Teams can no longer be regarded as the cost of producing a product; they are the business and generate business impact directly.
This development is driven by very short learning loops, eliminating delaying dependencies on other parts of the organization, user intimacy, etc. These teams will have any opportunity to outperform teams with different conditions.
As with the jumps in progress in previous decades, this will not be easy. Each team will have more capabilities and responsibility—similar to previous jumps. It is also built on the foundation of previous decades, such as real teams, continuous delivery, and discovery habits. So, are you still struggling with those? The 2020s of product development are far away.
The Thirties
If we continue to develop as described above, the teams within a company in the 2020s will resemble small enterprises collaborating to achieve the overall company strategy. In extreme cases, the traditional company structure will be almost entirely replaced by micro-enterprises sharing a few cross-cutting capabilities such as legal and security.”
Hard to imagine? Definitely, but if you’re curious, I recommend that you look into the work of Corporate Rebels to see what alternative organization structures are starting to appear around the globe and might come your way—probably not until the thirties if your company is still struggling with the challenges of the 2000s, 2010s, and 2020s unless you make a giant leap.
Conclusions
The year 2030 is still far away, but I feel confident that the changes in how we develop (software) products will continue as described, with the goal of creating more value faster, in smaller units, and closer to the users.
We will all move in that direction, but not at the same speed, not meeting the same challenges, and definitely not in the same order. There is no reason why you should be a 2000s team before implementing the good stuff from later decades. On the contrary, there are better ways to start. Your organization’s path is individual, but it should give you one piece of advice: Get your continuous delivery running. That is a prerequisite for many other things I’ve mentioned.
We’re close to the end, and the attentive reader will have noticed that I haven’t returned to the challenges of scaling development of larger, complex products after discarding the current agile scaling frameworks as the way forward.
The prerequisite for gaining the advantages is that each team can have its own sub-product with a well-understood user and value creation. That equals descaling the big, complex product that may seem inseparable into parts. That is by no means simple, but I believe that Conway’s law works both ways. If we insist on a different organization and invest in changing what needs to be, our systems will eventually have a structure that resembles the newly descaled organization.
Over the last few years, I’ve been a firsthand witness to transformations in quite large organizations with many teams and complex legacy systems. My testimony is that descaling products like that is not the easiest task, but it pays off.
Instead of giving you a superficial summary here, I believe it’s better to start writing a more comprehensive article about what I’ve learned about descaling products as an alternative to scaling your agile teams in Scrum-based frameworks. Frameworks, I believe, limit your possibilities for taking advantage of the progress made during the last decades.
…
References:
[1] http://agilemanifesto.org
[2] http://www.managecomplexity.dk/blog/2018/06/28/agile-is-organizational-re-design/
[3] http://www.extremeprogramming.org/rules.html
[4] http://www.amazon.com/User-Stories-Applied-Software-Development/dp/0321205685
[5] http://www.managecomplexity.dk/blog/2019/05/23/agile-planning-circles/
[6] https://www.amazon.com/Experimentation-Works-Surprising-Business-Experiments/dp/B085KB2KDX
[7] https://killedbygoogle.com/
[8] http://www.managecomplexity.dk/blog/2019/05/23/agile-planning-circles/