Incorporating software architecture into agile projects

Firstly, I should start by calling out the fact that there are different types of architecture. For example, enterprise architecture focuses on strategy rather than code. It is the strategy for how people, processes, technology etc. is utilised across an organisation in order to be effective and efficient. The type of architecture I will be focusing on in this post is software architecture. This is a lower level of abstraction, which focuses on the technical detail.

In Simon Brown's book on the subject he states that the software architecture has a dual meaning. Firstly, as a noun it signifies structure. It is the result when solutions are decomposed into its constituent parts, including all cross-cutting concerns and relationships. As a verb, it translates to vision. It means to take the project drivers and turn them into a technical roadmap. It is about making the "significant" design decisions (i.e. the ones which will be costly to change your mind about!). These can include (but are in no way limited to) levels of abstraction, technology choice, framework choice, design approach and patterns, etc.

He also states emphatically that it is the job of the architect to introduce technical leadership into a project.

Architectural principles

The main architectural drivers which drive the vision are as follows:

• Functional requirements

You need to have a clear understanding of the project goals in order for the solution to meet them.

• Constraints
  • Time and budget
  • Technology (approved, existing, maturity, internal IP)
  • People
  • Organisational

Though these constraints can be extremely limiting (especially the ones around time and budget) they can allow narrowing down of choices, which can be helpful when trying to design a solution. They are also often for good reason and recognising them is crucial in constructing an appropriate solution.

• Quality attributes
  • Performance
  • Scalability
  • Security
  • Availability
  • Privacy

The requirements surrounding quality attributes can be difficult to obtain. Partially due to the fact that they are often not as well defined from the outset of a project. There is also the issue that if you ask someone, for example, how secure they need their solution to be, they will invariably say something along the lines of "very". Including a cost estimate when capturing quality requirements can be useful – "to give only 0.1% downtime will cost X, however for only Y we can give 0.5%". This will make the process of weighing up cost and benefit clear from the offset.

Once these drivers are well defined and prioritised, an architect will use them to create a vision for the project. They will then communicate this vision to the developers who will turn it into reality. How much detail is needed in the architecture depends on the team who will be building the solution. There are two extremes: the developers have no freedom to make decisions, or the developers can change anything that they see fit. Usually the answer lies somewhere in the middle of these extremes.

The problems surrounding architecture

A lot of the problems with software architecture are around the perception of architects as sitting in an "ivory tower", being domineering lords who are detached from the realities of software development. This is partially due to the AaaS (architecture as a service) mode of working which is often seen. In this model, an architect will "deliver" a solution and then move on. Therefore, one of the main take-aways from Brown's books was that one of the key architectural pillars is technical leadership. This means that an architect is directly responsible for continuous technical decision making and planning throughout the project. It also means that they take ownership of the solution and many of the risks that surface throughout a project.

An issue which compounds this is that there can be a large gap between the architect and the developers. This distance can brew a lack of respect, and demotivation of the team because they see the architect as out of touch with real software development. There are several strategies for alleviating this. Getting hands on (writing specs, prototypes, PoCs) can help to reduce this gap as it reduces the separation between an architect and those building the solution (and it also helps in keeping up to date with the latest technology trends!). But crucially, open communication and constant involvement goes a long way towards addressing these issues. It is extremely important to set up a feedback loop between the architect and the developers. This feedback loop will allow knowledge gained through technical implementation to be fed back into the solution design. Critically the architecture role is a leadership role, and the importance of communication and management that goes along with this needs to be recognised.

Architecture and Agile

If you have read my previous blog post on Agile estimation and planning, then you'll know that here at endjin we have incorporated a lot of these techniques into our business model. A lot of people think that Agile working and software architecture are at odds with one another. After reading a lot on the subject, I can see why. On the surface, the idea of planning out the whole architecture of solutions seems to collide with the flexible, week by week iterative working involved when using Agile. However, the idea of not doing any software architecture also seems like insanity to me.

Luckily, I think that the way in which Agile and architecture overlap goes a long way in enforcing a more productive style of architecture. Instead of up-front architecture, Agile requires constant and ongoing architecture throughout the project. This means that an architect is not able to "throw a design over the wall" and move on to another project. The architect will be making design decisions throughout the project, governed by the architectural drivers. They will constantly be involved in the project, taking ownership of risks uncovered in the design throughout the process and adapting as necessary as the project progresses. The creation of feedback loops is also something which is critical in Agile development, and this goes hand in hand with the feedback loops necessary for effective architecture.

There are a few things which I think are worth calling out when talking about the relationship between Agile and architecture. The main one is about the order of work completed. Often the argument is made that architecturally things need to be done in a certain order. Whereas, the Agile process says that features which have the highest business priority, or highest risk, should be done first as to get the most value should the project start to overrun. I think the answer here really is doing "just enough" design in order to not close yourself into a corner where architecture suffers, while retaining the business prioritisation which leads to successful projects. The way this is implemented can vary. Howard wrote a blog post some years ago, which proposed a solution to the "how much design" problem. He suggested that the architect should have an overall view of how the software hangs together, how the pieces intertwine. But the actual implementation of each piece was not filled in until necessary. (This includes the cross-cutting concerns, which cannot be left out of any architectural design.) He uses the analogy of a colouring book, with the architect being responsible for the outlines and the exact shade of each section being filled in, by the development team, as the project progresses.

This approach can be combined with the concept of the "last responsible moment". This is the strategy of leaving important but costly-to-change decisions until they need to be made, or until it would be irresponsible not to make them. This means not filling in those colouring sections until they need to be filled in, allowing yourself the flexibility to change things if problems, or better solutions arise. It should be noted that "needing to be filled in" includes need due to business/technical risk factors. For example, if there is a large amount of risk around a certain area of the solution, then deciding how to implement it and beginning the work may be necessary for risk mitigation as the project progresses.

Alongside the idea of constant architecture, there is the possibility that you do not need one specific person to fulfil the architecture role. The role can be shared amongst all members of the team. This not only means that the "gap" between the architect and the developers is non-existent (they are literally the same people), but also means that the entire development team is necessarily thinking about the big picture.

This mode of working comes with its own dangers. It requires the team to be proactive about actually performing the responsibilities of the architecture role as a whole. Risks need to be uncovered, owned and mitigated, which involves individuals in the team working to keep the project on track. Cohesive architecture decisions also need to be made, which involves strong communication and a lack of ego. Simon Brown states that this kind of architecture is only possible in mature teams who can work together effectively.

Some key points

Solving many of the problems surrounding architecture (whether working in an Agile environment or not) hinges on good communication of the project vision, and the introduction of successful technical leadership. Essential for this effective communication of the vision is the idea of a ubiquitous language. This ensures effective communication and understanding of the vision. If there is not a collective understanding of the language being used to describe the model, then there is no hope for cohesive progression towards a common goal. Also key to communication is the production of useful and informative diagrams at different levels within the architecture. However, this is a subject which I hope to dive into a lot more detail in in another blog post, so for now I will just say that good diagrams are an essential aspect of communicating the model and vision.

Finally, whether you have a single or collective architecture role, and whether the design is happening upfront or throughout the project, the key to successful architecture is constantly keeping the drivers in mind. Risks are prioritised relative to these drivers, and they are the things which drive the project towards successfully completing its goals. Without clearly defined drivers, the architecture role is impossible to carry out, and projects can end up wildly off track.