Skip to content

Decision Makers Guide to Microsoft Fabric - The Pace of Innovation

Ed Freeman Ian Griffiths Ed Freeman Ian Griffiths

Decision Makers Guide

Welcome to the final part in a new series of interviews with real-world Decision Makers (CTOs, CIOs, Heads of / Directors of Software Engineering, Data & Analytics) about how they manage their strategic roadmap and evaluate new technologies to simplify their portfolio, deliver better outcomes for stakeholders, or give them a competitive advantage. In a 3-part interview we talk to Tom Peplow about his assessment of Microsoft Fabric.

TLDR;

Tom Peplow, Principal & Senior Director Product Strategy at Milliman, chats with Ian Griffiths & Ed Freeman from endjin about how Microsoft Fabric is a disruptive technology. The insurance industry is slow moving, heavily regulated, and products have a long lifespan. Cloud platforms, on the other hand, are fast-paced and evolve quickly. How do you deal with these two opposing forces? Can you make good bets by betting on open source and open standards? Can you avoid vendor lock in, without the multi-cloud pitfall of building against the lowest common denominator? Or should you go all-in and bet on a single vendor and take advantage of their innovation investments?

The talk contains the following chapters:

  • 00:00 Introduction
  • 00:30 Mismatched timescales?
  • 01:30 Betting on open source
  • 02:07 What will the market make of Fabric?
  • 03:47 Is there lock-in with Fabric?
  • 05:06 Shielding users from technology changes
  • 06:25 When to bet on a platform
  • 09:23 Multivendor hedging
  • 09:51 Homogenization
  • 11:09 Final thoughts

Microsoft Fabric End to End Demo Series:

Microsoft Fabric First Impressions:

Decision Maker's Guide to Microsoft Fabric

and find all the rest of our content here.

Transcript

Ian Griffiths: This is the third and final part of a discussion of what Microsoft Fabric means for your existing skills and investments in Microsoft Azure's data and analytics services. You can find a link to the first two parts in the description. In this final part, Tom talks about how Milliman resolves the long-term outlook required in the insurance industry with the very fast pace of change that occurs in cloud platforms.

Milliman works in the insurance industry, which tends to have quite a long timeframe perspectives; a lot of the products have a very long life cycle, it's a heavily regulated industry, so things often tend to move quite slowly, whereas the nature of cloud platforms tends to be very fast iteration and everyone's driving to innovate and get the next thing out there. But the dark side of this is that things often go away quite quickly as well. We have already had two versions of Data Lake and Microsoft Azure and now we have got OneLake. Technologies in the cloud seem to have a short shelf life. How does this affect Milliman?

Tom Peplow: This is going to sound self serving, but we've made some pretty good bets. We did manage to sidestep the early data products that did not make it. The underlying decision process for us has been "let's bet on open source" because we are not just relying on one vendor to support it. And let us bet on things that have high levels of adoption and are core to the platform. Our Compute Platform is something that we have invested heavily in ourselves. It is core to our IP. The ability to distribute our calculations at global scale is a hard problem to solve, but it means that we can go to our customers as a very robust solution.

We try our best to make bets where we are not getting blown out of the water. That is one of the hesitations with moving quickly with Microsoft Fabric. What is the market going to think of it? It is tough to look into the future and predict whether it is going to fail. Just because I work in insurance does not mean I understand the future!

But you must think about it. You have got to be careful and understand that past performance is not a predictor of future success. If you look at Microsoft's past performance, you may decide that you do not want to bet. Because they made mistakes, and people have been burnt by those mistakes. But what you could do is look at the decisions they made to move forward. Power BI as a platform is very well adopted. It is in every enterprise business in the world. Azure is not going anywhere. And if you think about where all the data in Azure is stored, OneLake's built on top of it. If OneLake were to vanish, your data would still be in Data Lake Gen 2. The APIs you used to get at it have been around for years, so other tools integrate with them. So you've got a migration path in and a migration path out. Delta Parquet is an industry standard, there was a great article about Apple using petabytes worth of data on Databricks in Delta Lake. So, the tech works, its proven; other vendors are using it. You have a migration path in, you have a migration path out. Where is the lock in with Fabric? I do not think there is one. There probably is. And Ed will tell me because he is used it more than I have. The decisions I'm looking at is that there's two standards based ways to store big data; Apache Iceberg and Delta Lake, and both are open source.

Apache Iceberg probably has more contributors because it's used by Snowflake. So, it has a big player data vendor reading in and out of it. At the end of the day, it is still Parquet at the bottom. So, it is an easy migration path and I would imagine there's connectors between the two of those things, Apache Iceberg and Delta Lake. You have got to choose one of the two. That is an important decision to make. That is where we are looking first; where do we keep all of our data? I am not as worried as I was in the past about making the wrong bet because the way we made bets in the past, thanks to endjin's coaching, as we have worked with you for 10 years, you've really helped us understand where to invest and not invest.

We have tried to shield our end users from technology. For example, we use SQL Server today to serve data to Power BI. They do not know that. It is just a Power BI report. They see the Power BI report, so it is an important investment decision for us that Power BI did not die. That would have been an expensive thing for us to have to deal with. But we waited to see if Power BI got broad adoption. And when we adopted it, it was early with Embedded we launched that. The last one of my data videos in public was of Josh Caplan on the launch of Power BI Embedded. Josh is now is "Mr. OneLake" but before he was "Mr. Power BI Embedded". We did have to decide early there, but we knew that Power BI was not going anywhere when we did that.

It is hard and it will be hard when something goes away because you must spend money to replace it. But you can abstract away complexity and hide things. SQL, I think, is going to go away for us one way or the other. With Direct Lake which removes latency from people being able to see numbers. It provides better performance than we were getting through SQL, has more feature rich things with DAX, so that could be a good decision, but customers would not notice they just see things get better, which is good.

Ed Freeman: I think the interesting point when you brought up Databricks and Snowflake and Fabric, and choosing between these data platforms is now that they are all opening up feature sets, which makes certain elements transferable.

That does not mean that you are in the clear. There is still benefits that you get from betting on a single platform. With Fabric, whilst there is no lock in because they are using open standards and they are using existing protocols. They are building optimizations that are you are only going to benefit from in Fabric, like Direct Lake, for example. It is a prime candidate because they are using this proprietary compression algorithm that if you are writing a table from Databricks, you cannot write it in that format; you will have to essentially re-optimize on the Fabric side.

So, you have got to think, is that additional overhead worth it to say, I am going to bet on all of them because I want to use because everything is open now? It is a little bit more nuanced than that. You will still get that kind of better returns if you bet one or the other at least for a specific kind of project or platform, because there are these kinds of optimizations that these vendors can apply because they have a bet on a specific format and they know their engines the best. The engine layer the VertiPaq, there are going to certainly be still things that are not going to be released into the open source.

So you've got to try and think what's there? What is their unique selling point really for this product and kind of the technical unique selling points are going away or they are all kind of converging because each of these vendors are providing kind of similar tool sets. It's almost more about the socio-technical side of how do our users use this platform, and use the additional kind of value add features to their benefits? It is a difficult question. I think it would be hard to say we can go totally vendor agnostic here because there are still benefits you get from choosing a specific.

Tom Peplow: And that's been key to up to my strategy with technology is the whole is better than the sum of its parts. If you do go with the whole thing with Fabric, you get a whole load of more value, because they have thought about how those things integrate. They have leveraged IP, they are not going to make open, it is their differentiated value. Do you believe in that? Are you locked into it is an important question. But if you feel that is worth locking in, then you go for it. And if you are not, then there is some pretty options to try it. There's the same problems on the Databricks side too. They have proprietary stuff that they are not open sourcing that makes their solution differentiated as well.

The other thing is betting on a vendor rather than trying to hedge. We have bet big with Microsoft over the years. We have been asked to be multi-cloud in the past. For me, then I am building to the lowest common denominator across the platforms, which is difficult because, to your point earlier, I am not leveraging the better bits of the platforms, or if I do, then I am doing it multiple times to leverage the better bits of the platforms. The other thing we did is, the homogeneity of it all, like last year we stopped supporting on premises for High Performance Computing. It was very difficult for us to be able to optimize for every different configuration of a High Performance Computing setup a customer would have. We have some super sophisticated HPC environments that we can run on and we've got some shadow IT "PCs under the desk". And making calculations run reliably in those two different settings is challenging because there so different.

Now in the cloud, we know what we are targeting. We know we can depend on InfinityBand networking. We can depend on highly-available storage. We can depend on these types of CPUs. We can optimize for different instruction sets. We know we are going to have a GPU in this. We know we are going to have an AMD chip that we can run our calculations, knowing the calculations are going to run well. That is massively valuable for our ability to go quickly and deliver value to our customers because we do not have to think what about that weird situation that could happen if they do this particular thing. So having homogeneity in control of the platform. Being able to influence the vendor and having a deep, meaningful partnership. And going big, they're scary bets, but they pay off because you have a lot of upside from being committed to an outcome.

Ian Griffiths: Okay. Thank you, Tom, very much for sharing your insights with us. You have given us a perspective that is obviously very deeply informed by years of experience with working with this stuff.

Do you have any last thoughts to share with us on Fabric before we go?

Tom Peplow: No, I would just encourage people to look. We are looking seriously looking and finding ways to prove it works at scale without burning a customer along the way. Try the same and be excited. Do not be scared by it. It is going to free your time up to do stuff that really matters!

Ian Griffiths: Okay. Tom, Ed, thank you both very much.