Friday, May 18, 2018

Thinking about Software Architecture & Design : Part 8

It is common practice to communicate software architectures using diagrams, but most diagrams, in my experience are at best rough analogies of the architecture rather than faithful representations of it.

All analogies break down at some point. That is why we call them “analogies”. It is a good idea to understand where your analogies break down and find ways to compensate.

In my own architecture work, the main breakdown point for diagrams is that architectures in my head are more like movies than static pictures. In my minds eye, I tend to see data flowing. I tend to see behaviors – both human and algorithmic – as animated actors buzzing around a 3D space, doing things, producing and consuming new data. I see data flowing, data flowing out, data staying put but changing shape over time, I see feedback loops where data flows out but then comes back in again. I see the impact of time in a number of different dimensions. I see how it relates to the execution paths of the system. I see how it impacts the evolution of the system as requirements change. I see how it impacts the dependencies of the system that are outside of my control e.g. operating systems etc.

Any static two dimensional picture or set of pictures, that I take of this architecture necessarily leaves a lot of information behind. I liken it to taking a photo of a large city at 40,000 feet and then trying to explain all that is going on in that city, through that static photograph. I can take photos from different angles and that will help but, at the end of the day, what I would really like is a movable camera and the ability to walk/fly around the “city” as a way of communicating what is going on in it, and how it is architected to function. Some day...

A useful rule of thumb is that most boxes, arrows, straight lines and layered constructions in software architecture diagrams are just rough analogies. Boxes separating say, organizations in a diagram, or software modules or business processes are rarely so clean in reality. A one way arrow from X to Y is probably in reality a two way data flow and it probably has a non-zero failure rate. A straight line separating, say “valid” from “invalid” data records probably has a sizable grey area in the middle for data records that fuzzily sit in between validity and invalidity. And so on.

None of this is in any way meant to suggest that we stop using diagrams to communicate and think about architectures. Rather, my goal here is just to suggest that until we have better tools for communicating what architectures really are, we all bear in mind the limited ability of static 2D diagrams to accurately reflect them.

Thursday, May 10, 2018

Thinking about Software Architecture & Design : Part 7

The temptation to focus a lot of energy on the one killer diagram that captures the essence of your architecture is strong. How many hours have I spent in Visio/Powerpoint/ on “the diagram”? More than I would like to admit to.

Typically, I see architectures that have the “main diagram” and then a series of detail diagrams hidden away, for use by implementation and design teams. The “main diagram” is the one likely to go into the stakeholder presentation deck.

This can works fine when there are not many stakeholders and organizational boundaries are not too hard to traverse. But as the number of stakeholders grows, the power of the single architectural view diminishes. Sometimes, in order to be applicable to all stakeholders, the diagram becomes so generic that it really says very little i.e. the classic three-tiered architecture or the classic hub-and-spoke or the peer-to-peer network. Such diagrams run the risk of not being memorable by any of the stakeholders, making it difficult for them to get invested in it.

Other times, the diagram focuses on one particular “view” perhaps by putting one particular stakeholder role in the center of the diagram, with the roles of the other stakeholders surrounding the one in the middle.

This approach can be problematic in my experience. Even if you take great pains to point out that there is no implied hierarchy of importance in the arrangement of the diagram, the role(s) in the middle of the diagram will be seen as more important. It is a sub-conscious assessment. We cannot help it. The only exception I know of is when flow-order is explicit in the diagram but even then whatever is in the middle of the diagram draws our attention.

In most architectures there are “asks” of the stakeholders. The best way to achieve  these “asks” in my experience is to ensure that each stakeholder gets their own architecture picture, that has their role in the center in the diagram, with all other roles surrounding their part in the big picture.

So, for N stakeholders there are N "main views" - not just one. All compatible ways of looking at the same thing. All designed to make it easier for each stakeholder to answer the “what does this mean for me?” question which is always there – even if it is not explicitly stated.

Yes, it is a pain to manage N diagrams but you probably have them anyway – in the appendices most likely, for the attention of the design and implementation phase. My suggestion is to take them out of the appendices and put them into the stakeholder slide deck.

I typically present two diagrams to each stakeholder group. Slide one is the diagram that applies to all stakeholders. Slide two is for the stakeholder group I am presenting to. As I move around the different stakeholder meetings, I swap out slide number two.

Tuesday, May 08, 2018

Thinking about Software Architecture & Design : Part 6

Abstractions are a two-edged sword in software architecture. Their power must be weighed against their propensity to ask too much from stakeholders, who may not have the time or inclination to fully internalize them. Unfortunately most abstractions require internalization for appreciation of their power.

In mathematics, consider the power of Euler's equation and contrast it with the effort involved to understand what its simple looking component symbols represent. In music, consider the power of the Grand Staff to represent musical compositions and contrast that with the effort required to understand what its simple looking symbols represent.

Both of these abstractions are both very demanding and very powerful. It is not uncommon in software architecture for the practitioner community to enjoy and be comfortable with constantly internalizing new abstractions. However, in my experience, a roomful of software architects is not representative of a roomful of stakeholders in general.

Before you release your killer abstractions from the whiteboard into Powerpoint, try them out on a friendly audience of non-specialists first.

Wednesday, May 02, 2018

Thinking about Software Architecture & Design : Part 5

Most architectures will have users at some level or other. Most architectures will also have organisational boundaries that need to be crossed during information flows.

Each interaction with a user and each transition of an organizational boundary is an “ask”. i.e. the system is asking for the cooperation of some external entity. Users are typically being asked to cooperate by entering information into systems. Parties on the far end of integration points are typically being asked to cooperate by turning around information requests or initiating information transfers.

It is a worthwhile exercise while creating an architecture, to tabulate all the “asks” and identify those that do not have associated benefits for those who are performing the asks.

Any entities interacting with the system that are giving more than they are receiving, are likely to be the most problematic to motivate to use the new system. In my experience, looking for ways to address this at architecture time can be very effective and sometimes very easy. The further down the road you get towards implementation, the harder it is to address motivational imbalances.

If you don't have  an answer to the “what is in it for me?” question, for each user interaction and each integration point interaction, your architecture will face avoidable headwinds both in  implementation and in operation.

Friday, April 27, 2018

Thinking about Software Architecture & Design : Part 4

Any new IT system will necessarily sit inside a larger context. If that context includes the new system having its own identity – from “the new system” to “Project Unity” - it will be anthropomorphised by its users. This can be good, bad or neutral for the success of the new IT system.

It does not matter what form the naming takes e.g. “Project Horizon”, “Bob's System”, “The new HR system” or even a visual identity such as "the new button", or even a tactile identity such as “the new panel under the desk at reception”. In all cases the new IT system may be treated by existing team members in much the same way as a new team member would be treated.

New systems get “sized up”, so to speak, by their users. Attributes such as “fast”, “unreliable”, “inflexible” or even “moody” might be applied to the new system. These may be factually based, or biased, depending on the stance the community of users adopts towards the new system arriving into their team.

One particularly troublesome possibility is that the new system may be seen as causal factor in events unrelated to it. “X used to work fine before the new system came along....” The opposite can also happen i.e. the new system get plaudits for events it had no hand or part in. Causality versus correlation can be a tricky distinction to navigate.

Takeaway: sometimes the human tendency towards the anthropomorphic can be used to your advantage. If you suspect the opposite may be true for your new system, it can be useful to purposely avoid elaborate naming and dramatic rollout events which can exacerbate anthropomorphisation.

Sometimes, new systems are best rolled out with little or no fanfare in a “business as usual” mode. Sometimes it is not possible to avoid big bang system switchover events but if it is at all possible to adopt a phased approach to deployment, and transition slowly, I would recommend it for many reasons, one of which is the sort of team dynamics alluded to here.

As AI/Robotics advances, I think this will become even more important in the years ahead.

Tuesday, April 24, 2018

Thinking about Software Architecture & Design : Part 3

In software architecture and design we have some pretty deep theories that guide us on our way. We know how to watch out for exponential run times, undetectable synchronisation deadlocks, lost update avoidance etc. We have Petri nets, state charts, entity/attribute diagrams, polymorphic object models, statistical queuing models, QA/QC confidence intervals....the list goes on....

...and yet, in my experience, the success of a software architecture & design project tends, in my experience, to revolve around an aspect of the problem domain that is not addressed by any of the above. I call it the experiential delta.

Simply put, the experiential delta is a measure of how different the “to be” system appears to be, to those who will interact with it – experience it – day to day.

A system can have a very high architecture delta but a low experiential delta and be many orders of magnitude easier to get into production than a system with low architecture delta but a high experiential delta. 

It pays to know what type of experiential delta your “to be” solution represents. If it has high experiential delta, it pays to put that issue front and center in your planning. Such projects tend to be primarily process change challenges with some IT attached, as opposed to being IT projects with some process change attached.

In my experience, many large IT projects that fail, do not fail for IT reasons per se. They fail for process change reasons, but get labeled as IT failures after the fact. The real source of failure in some cases, is a failure to realize the importance of process change and the need to get process change experts into the room fast. As soon the the size of the experiential delta crystallizes.

Indeed in some situations it is best to lead the project as a process change project, not an IT project at all. Doing so has a wonderful way of focusing attention on the true determinant of success. The petri nets will look after themselves.

Thursday, April 19, 2018

Thinking about Software Architecture & Design : Part 2

Technological volatility is, in my experience, the most commonly overlooked factor in software architecture and design. We have decades worth of methodologies and best practice guides that help us deal with fundamental aspects of architecture and design such as reference data management, mapping data flows, modelling processes, capturing input/output invariants, selecting between synchronous and asynchronous inter-process communication methods...the list goes on. 

And yet, time and again, I have seen software architectures that are only a few years old, that need to be fundamentally revisited. Not because of any significant breakthrough in software architecture & design techniques, but because technological volatility has moved the goal posts, so to speak, on the architecture.

Practical architectures (outside those in pure math such as Turing Machines) cannot exist in a technological vacuum. They necessarily take into account what is going on in the IT world in general. In a world without full text search indexes, document management architectures are necessarily different. In a world without client side processing capability, UI architectures are necessarily different, in a world without always-on connectivity.....and so on.

When I look back at IT volatility over my career – back to the early Eighties – there is a clear pattern in the volatility. Namely, that volatility increases the closer you get to the end-users points of interaction with IT systems. Dumb “green screens”, bit-mapped graphics, personal desktop GUIs, tablets, smart phones,  voice activation, haptic user interfaces..

Many of the generational leaps represented by these innovations have had profound implications on the software architectures that leverage them. It is not possible – in my experience – to abstract away user interface volatility and treat it as a pluggable layer on top of the main architecture. End-user technologies have a way of imposing themselves deeply inside architectures. For example, necessitating an event-oriented/multi-threaded approach to data processing in order to make it possible to create responsive GUIs. Responding sychronously to data queries as opposed to batch processing. 

The main takeaway is this: creating good software architectures pay dividends but they are much more likely to be significant in the parts of the architecture furthest away from the end-user interactions. i.e. inside the data modelling, inside discrete data processing components etc. They are least likely to pay dividends in areas such as GUI frameworks, client side processing models or end user application programming environments.

In fact, volatility is sometimes so intense, that it makes more sense to not spend time abstracting the end-user aspects of the architecture at all. i.e. sometimes it makes more sense to make a conscious decision to re-do the architecture if/when the next big upheaval comes on the client side and trust that large components of the back-end will remain fully applicable post-upheaval.

That way, your applications will not be as likely to be considered “dated” or “old school” in the eyes of the users, even though you are keeping much of the original back-end architecture from generation to generation.

In general, software architecture thinking time is more profitably spent in the back-end than in the front-end. There is rarely a clean line that separates these so a certain amount of volatility on the back-end is inevitable, but manageable, in compared to the volatility the will be visited upon your front-end architectures.

Volatility exists everywhere of course. For example, at the moment serverless computing models are having profound implications on "server side" architectures. Not because of end-user concerns - end-users do not know or care about these things - but because of the volatility in the economics of cloud computing.

If history is anything to go by, it could be another decade or more before something comes along like serverless computing, that profoundly impacts back-end architectures. Yet in the next decade we are likely to see dozens of major changes in client side computing. Talking to our cars, waving at our heating systems, installing apps subcutaneously etc.