Microsoft Fabric Is A Socio-Technical Endeavour

TLDR; We recommend that technology leaders approach adoption of Microsoft Fabric as a socio-technical endeavour. The most significant influence for successful adoption of Fabric is likely to be cultural and organisational in nature. In other words, successful adoption of Microsoft Fabric will be as much about changing hearts and minds, as it is about delivering technology. An immediate action should therefore be to assess your "organisational readiness" and use that to drive the actions to take over the next 6 to 12 months to prepare for Microsoft Fabric becoming generally available.

In May 2023, Microsoft announced Microsoft Fabric. It extends the promise of Azure Synapse Analytics integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together reporting, analytics, data science and data engineering on a new generation of lake house infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. See Ed Freeman's Introduction To Microsoft Fabric for more background.

The objective of this five part series of blogs is to help technology leaders assess the strategic implications of Microsoft Fabric. So, if you are a Chief Technology Officer (CTO), Chief Data Officer (CDO), Director of Data and Analytics, Director of Artificial Intelligence, Head of Data Science or any similar senior data leader role, this article is for you!

Over the coming months, Microsoft Fabric will move from public preview to general availability at which point it will be feasible to start deploying production workloads on the platform. This blog will help you to identify what actions you should be taking over the next 6 to 12 months, to get your organisation ready to successfully adopt Microsoft Fabric and maximise the benefits from it.

Success relies on people more than technology

Microsoft Fabric represents an opportunity to change how your organisation derives value from data and analytics. Technology leaders should think beyond adopting Microsoft Fabric as "the same way of working on a new tech stack" to identify how this new platform could help your organisation extract greater value from data. There are many sources of inspiration, but all case studies have a common theme:

In other words you have to master the cultural, organisational and human aspects as well as the technology in order to be successful in extracting value from data.

The simple model illustrated below fits any data project. A data team on the left, who create a "data product" to meet the needs of users on the right. Value flows from left to right. The data team need to be skilled in order to create the data product, the data product needs to meet the needs of the end users (it's often surprising how often they don't!) and the end users need to be prepared to put the data product into use in order to derive value from it.

Also important are the feedback loops that enable the data team to evolve the data product in response to the evolving needs of users. There's often a lot of uncertainty, especially if you're creating cutting edge, innovative, transformational data products where things are under constant evolution.

Where the flow of value or the feedback loops break down, significant issues will occur.

Microsoft Fabric has the potential to fundamentally transform this socio-technical system by seeking to:

• Empower the users on the right to become "citizen analysts" - enabling them to develop and evolve their own analytics and reports
• Streamline the work of the data team on the left through "SaaS-ification" of the underlying infrastructure
• Open up more agile ways of working, helping organisations to innovate at pace by iterating through the data product lifecycle faster

It is clear from the points above that cultural, organisational and human aspects are critical. Taking each in turn:

• Do users in your organisation want to be become "citizen analysts"? Do they have requisite skills? Are they prepared to take on responsibility for owning the analytics and reports that they create?
• Are your data team(s) geared up to give up major parts of their day job by moving from PaaS to SaaS? What new things will the capacity that Fabric promises to create enable them to do? Do they need to pick up new skills? Are they prepared to "give up control" to empower "citizen analysts"?
• Is the organisation ready to work in more agile ways? Do you have an entrepreneurial spirit, can adopt a data product centric mindset? Do you need to become more "stream aligned" in how you approach data projects?

Beware of Conways law

A major consideration in any socio-technical system is Conway's Law.

“Any organisation that designs a system will produce a design whose structure is a copy of the organisation’s communication structure.“

This powerful homomorphic force means that organisational lines of communication will prevail over the software architecture that you're trying to deliver. In other words, the system you are trying to build will take on an architecture that mirrors the organisation that delivers it. For example: if your organisation is fragmented, the system they create will be fragmented too.

If ignored, Conway's Law can have significant negative implications for data teams. I delve more deeply into this topic in a talk I gave at SQLbits in early 2023 "Creating a High Performance Data Team - Lessons Learned From The Field".

Apply an inverse Conway manoeuvre

Conway's Law can be leveraged for good by thinking about organisational design in conjunction with the desired target architecture to consciously keep the two in sync. This is an approach referred to as an "inverse Conway manoeuvre".

We would encourage you to consider your organisational design along side the target architecture you imagine being enabled by Microsoft Fabric. Map out how value flows through the organisation across the full data product lifecycle and seek to address any bottlenecks / unnecessary hand offs.

Data mesh is a socio-technical endeavour

Microsoft Fabric has been heavily influenced by Data Mesh. Data Mesh is Zhamak Dehghani's thought leadership about how to "deliver data-driven value at scale". The fundamental thing to note about is that it is not just about technology architecture, Data Mesh also describes important cultural and organisational principles that need to be applied in order to drive value from data in a safe, scalable and secure manner. Due to Conway's law, we know that these cultural and organisational concerns will tend to override our technology vision if we don't address them.

Assess your organisational readiness

We recommend that your preparation for Microsoft Fabric is informed by an organisational readiness assessment. This takes a data driven approach, with the objective being to highlight the cultural and organisational factors that could become barriers to success. Most technology leaders will have a good appreciation of what these potential barriers are from the outset, but it is often useful to augment this with 360 degree data capture by consulting senior stakeholders, peers and members of your team.

The benefits of this approach are:

• Engaging stakeholders - by conducting a 360 degree review, stakeholders will feel like they have been consulted and have had an opportunity to shape the strategy. This alone can help to lower barriers to change.
• Providing evidence - we are strong advocates of evidence driven decision making. By presenting evidence, you are more likely to build a more complete picture and to get support for any changes you are proposing to make. Existing metrics such as time to value, total cost of ownership can be augmented by more subjective insights from interviews and a questionnaire completed by a wider cross section of the organisation.

We typically perform an organisational readiness assessment as an integrated part of our Insight Discovery process in recognition of the socio-technical nature of any data initiative.

Three examples of where cultural change is likely to be required in relation to getting ready for Microsoft Fabric are:

1. Central data teams shifting from "center of excellence" to "centre of enablement" - this is important to overcome the unhealthy tension that tends to develop between a centralised data team trying to maintain control and ever increasing demands from the wider organisation. Microsoft Fabric is seeking to address this by enabling de-centralised self-service BI. This can be daunting for staff who have fulfilled more traditional roles (such as DBA's) who may worry about their relevance in this new world.
2. Adoption of a product mindset - in order to embrace the concept of "data products" with the objective of shifting the culture in your data teams from "order taker" to "innovation engine". I will write more about this in a forthcoming blog in this series.
3. Up-skilling the wider organisation to create a community of "citizen analysts", with an objective of keeping the analytics closer to where it will be used. Think of this as being as fundamental to your organisation as Microsoft Excel. Knowledge of Excel is taken for granted today, but it took

Final thoughts

Organisations should use the time available before Microsoft Fabric becomes generally available to get ready to adopt the platform and maximise the opportunities it will offer. See Fabric as an opportunity to make a step forward in operating model, not just doing the same thing but on a different technology stack.

Assess your organisational readiness by conducting a 360 review to capture input from senior stakeholders, peers and from within individual teams. The most significant gaps in readiness are likely to be related to culture and organisational design.

When it comes to making changes, we strongly advise an "evolution not revolution" approach. Avoid anything that looks like a "re-org". Take small incremental steps with feedback loops that ensure that future steps are informed by what you have learned. By making one small step every month, you will be amazed by what you will achieve as small changes build up across the organisation.

Think of this as a long game, Microsoft Fabric is a significant opportunity to sign post important changes that your organisation needs being to make to remain competitive years into the future. Dame Diane Coyle, professor of public policy at the University of Cambridge, a recognised expert on how we measure productivity was recently interviewed by the BBC in relation to the question "Why is technology not making us more productive?", in the article she reinforces that data is a socio-technical endeavour:

"It seems to be that if you have highly-skilled people, you have a lot of data and you know how to use the sophisticated software, and you can change your processes, so that people can use the information, your productivity is going through the roof. But in the same sector of the economy there are other companies that just can't do that."

In the article, the BBC concludes:

The technology is seemingly not the problem, and in some respects it is not the solution either. High productivity growth will come only to those that learn how to use it best.

We couldn't agree more!