Data is the new soil

This blog is inspired by the TED talk delivered by David McCandless in 2012. In the talk he gives examples of his design led approach to understanding data. By finding ways of compressing knowledge into a map or graphic he seeks to avoid information overload and enable our eyes (the most powerful of our 5 senses) to discover patterns and find the insights that matter.

When you are lost in information, an information map is useful.

Indeed David believes that "information design" is key to solving many of the information related problems in society. Whether that be issues of trust, increasing transparency, helping people to overcome information glut or engaging people in important issues such as climate change.

There is one specific quote from David that catches our attention:

Data is the new soil.

Note - soil not oil! He deliberately adapts the more common metaphor "data is the new oil", because he sees data as a fertile medium which can be enhanced and re-used over time (unlike oil), with data visualisations being the flowers that bloom from it.

This resonated with us, but we wanted to build on this metaphor to explore some of the factors that are key to establishing successful data and analytics strategies within an organisation, based on our real world experience over the past decade

Start with a clear goal

Farmers don't grow crops simply because they have fertile soil. They have a clear and compelling goal: to feed people.

Data and analytics initiatives are no different. They should be driven by a clear and compelling business goal.

We find that a clear and compelling business goal can have a dramatic impact on the success of a data and analytics initiative by giving it a compelling purpose and helping people to focus on delivering a meaningful impact. We would encourage you to take a similar approach: start with the goal not with the data!

People's natural tendency is to start data projects by looking at the data. We mitigate this by using our Insight Discovery process to take a top down, goal driven, persona centric approach to identifying requirements. This process engages stakeholders and it keeps the initiative focused on delivering high value actionable insights. These actionable insights are enabled by data. They support business goals by providing the evidence that prompts the actions that will result in a better outcome for the organisation.

If data is the new soil, then goal driven actionable insights are what organisations should be seeking to grow from it.

Establish strong roots

The roots support the data flows by providing the supporting infrastructure that enables the actionable insights to be successfully delivered and sustained. Just like roots, these are typically unseen, but they are absolutely critical - if the roots are not strong and healthy, the plant will wither and die. However, many organisations fail to realise the importance of establishing this infrastructure. They launch a data and analytics capability only to find that it withers and dies further down the line because it can't scale to meet the demands placed on it.

Furthermore, farms need activities farming such as unblocking irrigation channels, maintaining fences and applying fertiliser to be carried out to protect them and maintain their health over the long term. Data and analytics infrastructure also needs regular processes to be applied to keep it infrastructure operating reliably and secure.

The biome defines the environment in which a farm operates. It is defined by factors such as the nature of the soil, the landscape and climate. In a data and analytics context the biome is defined by by factors including the nature of the data (soil) you are working with (see the the 5Vs of big data) and other environmental concerns such as security, scalability, maintainability and flexibility.

This is where non-functional requirements (NFRs) come into play. Think of NFRs as your response to the biome in which you will operate - how you will keep the solution healthy and operational over its full lifecycle. The NFRs drive the type of infrastructure you will need to ensure the long term success of a data and analytics initiative - i.e. the more "harsh" the environment, the more you will need to invest in producing and protecting your crops. For example, it is straightforward to manually capture a small sample of data, to load it into a spreadsheet and to perform some ad hoc analysis on it. But an entirely different solution is required to be able to do the same at (small, medium or global) enterprise scale, in an automated manner, keeping the data secure, in a cost effective way over the full lifetime of the solution.

Build and maintain healthy roots by prioritising the analysis of non-functional requirements at the outset of your data and analytics initiative. Involve people who have the necessary experience to ask the right questions and who can design the infrastructure that will meet those requirements. For example, we help many organisations to establish strategic data platforms and through a decade of experience we know the right questions to ask in order to elucidate the NFRs, and can then bring tried and tested intellectual property that allows delivery of those NFRs to be de-risked.

Also be mindful that this is stage in the lifecycle that organisations can drop back into their comfort zone, where the prospect of spinning up new technologies can sometimes result in them straying from the original purpose. Avoid this pitfall by using the clear and compelling business goal to focus this analysis. This business goal driven approach can also have a significant bearing on the "total cost of ownership" (TCO) of a data and analytics initiative. For example in one recent project, by doing the up front work to articulate the business goal and define the NFRs, we were able to design and implement a solution that saved our client over £1M in infrastructure costs over the lifetime of the solution.

If data is the new soil, then your organisation should seek to establish strong roots by addressing the non-functional requirements from the outset.

Empower your farmers

Farmers have responsibility for tending to the land to keep it healthy and fertile year after year. In a data and analytics context, this would map most closely onto people commonly referred to as data stewards.

Many organisations will already have data stewards, but that role may not be formalised. These are typically people who have an operational role within the organisation and as part of their day to day job they work with data that is critical to the organisation.

As an organisation seeks to leverage the power of actionable insights, the role of data stewards will become increasingly important. As part of a data and analytics initiative, you should therefore seek to:

• Ensure that the role of the data steward is formally recognised;

• Involve data stewards in the initiative as expert users who can help to define the requirements and then act as super users to champion successful adoption in the wider organisation;

• Invest in training to enable your data stewards to upskill so that they can adopt modern data tooling and become "super human" in their role.

Indeed, along with data stewards, the role of any data initiative should be to upskill and empower the organisation as a whole with new data driven capabilities. Experience has taught us that the long-term success of any transformative project such as this is dependent on many aspects related to the organisation's readiness to roll out, adopt, embrace, support and evolve the data platform over time. Do not underestimate the amount of time and energy that this will take.

We use our Organisational Readiness Assessment process to help our clients identify gaps in their capability that should be addressed to safeguard the long-term success of the initiative. These gaps may include areas such as training and development, governance, programme management or the need to establish new career pathways to reflect the changes to roles driven by the transformation.

Think beyond the first years' harvest. If data is the new soil, then your farmers are the people who will champion and sustain the value of data and analytics within the organisation over many years ahead.

Secure the sponsorship and resources necessary to farm sustainably at scale

There's a big difference between someone being able to keep a plant alive on their windowsill, and being able to efficiently meet large scale demand for food! Modern farming is enabled by a long term commitment to investing in people and equipment to continually improve productivity and yield to offset downward market pressures. Data and analytics initiatives require a similar long term commitment to establish the infrastructure, make the necessary organisational changes and sustain the delivery value over the long term.

Many data and analytics initiatives often fail before they start because organisations don't commit to it in the right way. This is typically due to a lack of understanding of the scale of the value opportunity and the resources that will be required to deliver it.

In the extreme case, we see "vanity" projects: tactical investments that allow the organisation to claim it is investing in a data strategy but lacking the direction and long term investment that will enable it to make a real impact. Indeed, we have observed that Covid-19 has caused many organisations to re-focus leading to many such vanity projects being terminated.

To avoid getting off to the wrong start, you should:

• Start from the top, not the bottom - by having an executive in your organisation who is willing to sponsor the initiative from a leadership and budget perspective. Most importantly, they should understand that data is not an end to itself - it needs to be integrated into the wider business strategy. It has to be part of the non-technical goals of the organisation - just like sales / marketing / growth - acting as a lens, an enabler and a catalyst;

• It's a marathon, not a sprint - approach it as a strategic change initiative, with the objective of seeding a new level of data literacy across the entire organisation. To succeed, changes have to be made across the whole organisation - new skills, new roles and new ways of working. It's useful to think of spreadsheets as an example, organisations don't have a dedicated team of Excel experts. It has become a core IT skill and it is a tool available on every desktop. The same needs to happen with more modern analytics tooling. You should be seeking to do the same with more modern analytics tooling: to establish new tools and processes that become an impact the whole organisation. This type of transformational change will require time, patience and perseverance;

• Harness the talent, experience and perspectives across the organisation - successful data and analytics initiatives are "done in partnership with the organisation" NOT "done to the organisation". To achieve this, the initiative should be a multi-disciplinary endeavour involving a broad set of skills, including expertise in organisational change.

One useful exercise to help secure the right level of sponsorship and resources is to map out the end to end lifecycle of the project and to quantify the total cost of ownership (TCO). Just like a farm, data and analytics is a multi-generational affair. It is through a long term lens that any data and analytics initiative should be viewed.

The more data you capture, store and process, the more it will cost you. The TCO should therefore encompass the full lifecycle, spanning all generations from early exploration through to building, scaling, sustaining and ultimately retiring the data and analytics capability. This lifecycle should reflect the multi-generational nature of data and analytics. For example one phase of the lifecycle may involve getting the basics in place, potentially for years, to build up a big enough corpus of data to do more sophisticated data science or machine learning model development / training.

This analysis does not need to be onerous. Keep it high level by mapping out the resources that will be required in each phase of the lifecycle. Indeed if it does become onerous, the likelihood is that you've strayed too far away from solving a clear and compelling business goal and have gone down a rabbit hole.

In the illustration below we capture a generic view of what you typically expect to see coming out of this analysis: a multi-year, multi-million £ investment. This level of fidelity should generally be sufficient to get you in the right ballpark and to facilitate decision making:

Use this technique at the outset to set expectations with sponsors, to explore what the organisation can afford and to shape the goals of the initiative accordingly. The TCO does not necessarily mean net headcount or expenditure growth, it can often be supported by re-directing and re-focusing resources and budget from other areas of the business as part of a wider strategic prioritisation exercise.

This is a tool to enable the organisation to understand the value of data and analytics and the long term commitment required to make it a success.

If data is the new soil, then adoption at enterprise scale relies on a commitment to "farm" at scale through a long term multi-generational investment.

Ready to build your data farm?

Hopefully we haven't stretched the metaphor too far, and that this blog has been useful in highlighting that data and analytics is not a "bolt on" but something that should ultimately impact every person at all levels in your organisation.

It may also lead you to ask if your organisation is maximising the opportunity:

If data is the new soil, when you do nothing with it you get nothing out. However valuable it is.

If you are interested in exploring further, we would be happy to talk to you. For example, our 90 minute Data Strategy Briefing provides an end to end view of how endjin would approach establishing a data and analytics strategy and setting our client up for success.