How To Actually Make Decisions When Architecting a Data Platform
SQLBits 2024
Are you drowning in a sea of architectural options for your new data platform? Are there people in your organisation with strong opinions on which tools should be used, and for what? Are you unsure how to prove/evidence your gut feeling as to how things should be built? Well - this session's for you!
If you're in charge of architecting a new data platform, it can be difficult to make concrete decisions that pushes you closer to your end goal - building a successful solution. This session will introduce a tried-and-tested suite of strategies and tools that will help you efficiently navigate the unforgiving task of architecting a modern data platform.
We'll take a look at artifacts such as ADRs (Architectural Decision Records), NFR (Non-functional requirements) documents, and discuss strategies inspired from the Agile methodology (such as failing fast) that can help you in your endeavour.
Chapters
- 00:00 Introduction and Background
- 00:30 The Problem with Too Many Options
- 01:38 Understanding Platform Requirements
- 04:20 Organizational Factors and Conway's Law
- 07:53 Evaluating Products and Vendors
- 10:13 The Evaluation Process
- 15:05 Making the Final Decision
- 16:39 Closing Tips and Final Thoughts
Transcript
My name's Ed Freeman. I've worked at endjin as a Consultant and Data Engineer for eight years, ever since I graduated from a Maths Degree. I've done dozens of Greenfield and Brownfield projects across many industries, learnt many things. There's been an explosion in Power BI content, and that's why I curate a weekly newsletter called Power BI Weekly.
So, the problem. We have thousands of different options to choose from. This is a an analysis that's done by a company called FirstMark. It's a ML, AI, and data landscape and it's not just that. You go to conferences like Big Data London, SQLbits is quite good. They haven't got that many sponsors here, but for a free event, of course, there's gonna be hundreds of different products that they're trying to push at you. Lots of these things do exactly the same thing. Lots of them have lots of crossover. How do you know So where do you start?
Obviously you start from the top left and then you go one by one through everything until you figure out which one you want. No, just kidding. But really, where do I start? The way I see it is you've got three different things that are all at play. And all iteratively impacting one another. So you've got your platform requirements, you'll come in at the start of a project, you'll pretty much know the types of things that you need your platform to do, the product to do.
But these things are impacted by organizational factors and the products themselves, what the products can do, who they're, and who their vendor is. So let's talk about platform requirements. Platform requirements, most of you will know, generally split into functional and non functional requirements. Non functional requirements, also known as the "-illities".
The functional requirements are the things about what features do I need and how should the system behave. For example, the platform must support third party databases and APIs, or data must be stored in an open table format. Non functional, the "-illities" are more about the availabilities, the scalabilities.
And these are more about how the system should perform and how the system should scale. These things are generally quantified, so the data should be stored up to one terabyte, or the platform must cost less than a thousand dollars a month in operating expenses. The way we capture non functional requirements and functional requirements, but non functional particularly interesting, we keep it alongside our code.
So we keep it in a version controlled repository and we render it as marked down so that we can have a nice tabular view of the data. It helps to go back and revise and go back and figure out when things change, you need to update that and you have that version tracked. But how to capture everything to do with your project requirements?
It's not an, it's not a new a new thing, but software requirements specification documents capture all the different things that you might need to know. when you're when you're designing your platform and designing your project. Now the main thing the main difference between this and old waterfall methodologies and Agile, this is a living and breathing document.
You don't specify this once up front and then you're done. You want to go back and revise it once you learn about organizational factors that you need to consider or even certain features that products may have that might influence what you think your product should do internally.
So we've got the introduction, so we're setting the context description and assumptions of our platform. All the way through functional and non functional requirements, user stories. Now in data models and data dictionaries are paramount. You need to know what data you're after, and know what format it's going to be in, how it's going to map from source systems, because that could impact which product you go for.
And lastly, you've got the architecture diagrams. Now, this is a bit difficult. If you're trying to figure out what your architecture is, you might not know that. But the chances are, you're integrating with other things. Try and figure out, map out what those other things are, so that you can have a placeholder for the actual product you end up choosing in the middle.
Now one thing I think you should take inspiration from is Microsoft publishes a well architected framework. It's just it's a design framework for for trying to build a high quality software solution or data solution. So let's have a look at some organizational factors. And the first thing I want to quickly mention here is something called Conway's Law and homomorphic force.
Without going into too much detail, Conway's Law is, essentially says that organizational design prevails over software architecture design. So your software and your data products that you build will likely mimic the the organizational communication structures. Therefore it's important, really important to understand the organizational factors that are at play.
When you're designing your project. And one tool that we love using at endjin is something called Wardley Maps. So this is, this helps you map your business strategy to build situational awareness around the strategic direction of the organization. Excuse me. Sorry. So in this instance what we've done is we've mapped the evolution of Azure Synapse to Microsoft Fabric.
Most of you will be aware that is essentially The evolution that Microsoft's been pushing. And we have two axes. We have the evolution axis and we have the value chain. The value chain, essentially the higher you go that, you go on that, the more visible the component is to the end user.
And the evolution is how much of commoditization has been applied to each component. So you can see Power BI has become even more commoditized, and it's even closer to the end user than what it was before with Fabric. I can't go into too much more detail on this, but if you want to learn more about it, my colleague Barry has written a great blog about it, and that there.
But what questions should I ask? Organizational factors. Okay, what products or clouds are already being used internally that we could potentially think about using for our use case here? Are there any constraining commercial agreements? Are you, do you have a CSP, for example, that has to provide you with certain bits of software?
Or are you only allowed to provision certain bits of, certain products from certain cloud vendors? What's the structure and skill sets of the teams that will own the platform? So I have, do you have multidisciplinary teams already that could potentially take a single product and own it themselves?
Or is the product you're after going to have to be maintained by multiple different teams? And what are the skill sets? Are you looking for a no, no or low code platform? Or you're looking for a pro code platform? Or is it a mixture of all of them? Are your data providers aligning with any particular products?
So certain data providers, especially in finance, are actually trying to align themselves with certain products. And it might be that it might be easiest if they are your sole kind of data provider that you align with the way that they're going. That's not necessarily you have to do it that way, but it's one way of doing it.
What would other departments think about a new product? Do they even know that you're looking for a new product? Legal, finance, procurement, all should have a say in, in this process. And the sooner you bring them along for the ride, The sooner you The easier it will be further down the line. Pretty simple one.
What's your budget? So both CAPEX for the original, the initial project you're having to migrate your data platform or improve it, but also OPEX for when you actually go live. How much do you get to spend a month on this platform? Because that could, again, very quickly rule out some vendors. What regulatory requirements does your organization have?
Obviously everyone who has data in the EU needs to think about GDPR, but other more regulated industries, like finance and health, would have to think about does this product that we want to use actually align themselves with those regulations? So that's all the organizational factors, and the idea is, after the organizational factors, you'd go back and you'd revise your your functional, non functional requirements accordingly.
So if we think about products and vendors, Now the first thing I'll say is, in my opinion, products are converging. The vendors are in a race to add value, and what we find is that actually features are converging. So you'll see Databricks, Snowflake, Fabric, and Synapse, they all have both combined data engineering, data warehousing functionalities now.
All of these products are getting AI features, so Fabric has Copilot, Databricks has Databricks Assistant, Snowflake is acquiring companies left and center with different AI capabilities. So actually the importance of getting the product right is lessened because lots of the products are starting to do the same thing.
Anyway, what questions should we ask about the products that we're looking at? What core features does it offer? Does it align with our functional requirements? What pricing models are offered? If we know our OPEX budget is X, are we going to be able to have a provision a server for a hundred grand a year?
When we're meant to be running at a grand a month, for example. How mature is the product? If the product's really new, it's less likely to be as stable, it's less likely to have a community around it. How extensive is the product's ecosystem? As I said, if it's a new product, there's probably, it's probably not very extensive ecosystem, there's probably not many learning resources.
But if there's a product that's 5, 10 years old, there's probably a huge ecosystem there, huge scope for learning. They might have certifications that you can have and therefore the talent pool for actually hiring people to maintain that platform going forward will be will be bigger. Then on the vendor side, what's the vendor's reputation?
Do they have a reputation for building products and pulling them a couple of years later when their funding stops? What is their funding? Are they a big company that's not going to go away for the next couple of decades? Or are they a startup that are relying on VC funding that, that you can't guarantee is going to be around for a while?
What's their support model? If you're going to have issues with your platform what is the reputation of their support team? If it's going to be frustrating, you're going to run into a lot of issues, you might not want to go for them. Do they publish product roadmaps? And this is a big thing.
If you're having a data migration or a transformation project, you're looking to bet for five, ten years into the future. Having some sort of product roadmap, even if it's only for a year or two, you're Makes have faith in the vendor that they're actually going, they're looking into the future and they're sharing that vision with you.
Moving on to the evaluation process. Now, we actually, we need to pick a product. One, assemble an evaluation team. That shouldn't just be you, shouldn't be just your team. It should be you, your fellow data roles, IT team, project managers, other stakeholders from around the business. Then people like Procurement, Legal, Finance say, Hey, I'm looking to to maybe onboard a new product.
How much pain is this going to cause me? We've had procurement operations go on for 12 months, 18 months. And at that time your requirements might have changed. If it's going to be a high barrier to entry, you need to consider that. Step two and identify candidate products. So create a short list of products that are already used internally.
That's a great start. Products or cloud services that are used internally and then look at known alternatives as well. So if you're looking at Snowflake, think about Databricks. If you're looking at Fabric, think about Snowflake and Databricks. Alternatives that you're pretty sure are going to do the right thing even if even though you don't know much about them.
Consider those. And then if you need some more you want to do some more due diligence, time-box some research into additional options. Now this could be by speaking to your network searching the internet, or even reviewing industry analyses. So Gartner, Forrester, they all publish their own thing.
You've got to take that with a pinch of salt because their evaluation criteria might not match yours. But speaking to your network is a really good one. You, as data professionals, probably have people from past jobs or people from university that might be in data roles right now, and might be in quite high data roles.
They're probably using products that you might not even know about, or they're using products that you do know about, but you don't use and you want to find out more about. Ask them, reach out to them, and organizational factors should be kept in mind here, right? You can't just start pulling in every single product under the sun and start evaluating that if they don't actually meet your organizational requirements.
Step three, evaluate these against your requirements. Establish your evaluation criteria up front. Now, that's really important. If you've got an evaluation team, chances are people are going to have their preferred vendors going in. You want to establish the evaluation criteria before you see or test out any of the vendor solutions so that you don't have any bias in the actual evaluation process.
Then capture the results in a structured form. So the way we've done this in the past is we've just created a matrix to capture the evaluation against candidate products. So here we've got product A, product B, product C, product D. All the dimensions on the left hand side, so requirements both functional and non functional, then product and vendor dimensions.
The components within those, which is it goes from maturity, pricing, scalability, And all the statements that we care about for this. And what we've done, we've just come up with a simple scale from minus two to two about how well that, that particular product aligns with the statement.
Step four, analyze and identify top products. Your data professionals do data stuff. And that's the, one of the most important things here. You will have an array, a huge, vast array of data at your disposal. And you need to be able to analyze that. So this is what we've done in the past.
We've taken those spreadsheets that I've shown you, we've created a notebook just to wrangle them into, unpivot them into a full kind of stacked structured format, and then we've visualized them in Power BI. So we've mapped the functional against the non functional requirements, which ones are further to the top right and therefore most trusted by all of the people who've answered this questionnaire.
But then we've got down at the bottom the product against the vendor. But actually, when we look at the product against the vendor, product B doesn't look that good.
We can also look at the breakdown against the statement. So we have product A, B, C, D. We have a heat map and we can see, oh the product B is really good on this bit. And actually, we've done our prioritization. We've done some Moscow analysis on our requirements. And that's the most important thing.
So if you wanted to, you could then bring in the prioritization that you've assigned and have a weighted average. And then it'll be even clearer. What you have which ones are your preferred products. So step four, identify and investigate those top products further. Run POCs, attend vendor demos. If you're feeling brave, even host the vendors if you want.
If you're worth enough money, they will send everyone they can to come and try and convince you to get their product. It's not a bad thing. You get to see the thing in real life. But what we would do is you should choose a simple workload and have them implement it. Even if it's not over your data, have a, have as close a workload you can as possible.
Then once you've done that, you can revisit the product evaluation if necessary. Oh, actually, their marketing didn't really live up to the hype. So we're gonna have to go back to the drawing board or go back to the other things that didn't quite make it because then maybe they're more realistic with what they can actually do.
And step five, make your selection. So trust the process that you've been through. Let the data speak. But throw out all this if we step back. We're making decisions. And these can be hard to keep track of, hard to remember why you made certain decisions. This is why we really love using something called Architectural Decision Records, ADRs.
What these do is they set context for the decision. They list out the assumptions and decision drivers, and they highlight and scrutinize the considered options. They also, if you do choose an option, they will capture any mitigations for any shortcomings of the option you chose. So you've admitted that, oh, it doesn't do that quite well, but here's how we could work around that.
But these ADRs, they serve as an invaluable source for future you, and future team members. Why did we make that decision at that point? That seems wrong now, but it might be wrong now. You can go back and revise that. Because the way we structure ADRs is you version them, you treat them as append only, so that you can keep track of of the changes.
And you can see on the right hand side, this is just a normal a normal repo that we're using. But we've got an ADR over here, which I've truncated a little bit, because that's not what I'm trying to show. But all on the left hand side, we've got all the ADRs that we've written about different decisions that we've had to make for different parts of our solution.
You capture this here, and people can review it. And then you establish a team of reviewers, because you don't want to just force through your own work. You need to have some sort of review. So that other people, and you can get the consensus from other people. Some closing tips.
Iterate over the requirements and the organizational factors in parallel. So build that whilst you're figuring out the the context of your organization, so that you can build the best requirements you can. Involve multiple stakeholders in the architecture, design, and selection process. You don't want this to be just your decision.
You want to make sure that it's a decision of anyone that's impacted by the process you're going through. Avoid the sunk cost fallacy. So you might make the wrong decision, and it might be six months down the line you realize that you have sunk a lot of cost into trying this platform out. Don't just continue because you've spent a lot of money on it.
You can always go back to the drawing board, go back to the products you had before, because remember this is usually a 5 10 minute gamble, 5 minute would be great, 5 to 10, 10 year gamble that you're making. So you want to make sure you're getting it right, and 6 months is trivial when you think of it that way.
Upgrade platforms incrementally, don't try and do a whole migration at once, pick certain small bits of functionality and upgrade the platform to try and prove your value. And finally, keep your architecture as simple as you can. I'm not saying only have kind of one product in the mix. You might have a handful, but think about the maintenance going forward into the future.
If you have five different products that are all just doing a small part of your architecture, that's going to be hard to hard to manage, hard to keep track of. Thank you very much. Please give us feedback. I know SQLbits are really pushing this year to give online feedback. I'd love to hear what you have. If you have any questions, please come up to me after because we're out of time now. Thank you very much.