Modernizing Reactive Extensions for .NET
.NET Conf 2023
Endjin are proud to be a .NET Foundation Corporate Sponsor, as we are maintainers of Reactive Extensions for .NET (AKA ReactiveX AKA Rx.NET) which is one of the most well established and widely used open source .NET projects. Learn how this project is evolving to take advantage of changes in .NET 8.0
Transcript
Ian Griffiths
Okay hello and welcome. Thanks for coming, everyone. My name is Ian Griffiths. I'm a technical fellow at endjin, and I'm the lead maintainer of the Rx.NET project, and have been since the start of this year. So today, I'm going to talk about the work that we've been doing to modernize Rx.NET over the last year, and our plans For Rx's future.
So just in case you've never heard of Rx.NET, I'll start with a very quick introduction. Just one slide, so if Rx already, don't worry, I'll get to the point nice and quickly. The reactive extensions for Rx, usually just known as Rx, sorry. The React extensions for .NET, usually just known as Rx, were created by Microsoft's Cloud Programmability Group.
And this was actually formed back in the early days of Azure. And their job was to envisage a future in which cloud based programming was the norm. Can you imagine such a thing? And they were trying to ask back then, what would that mean for developers? What tools were we going to need to build software in this new world?
And one of the areas they identified that had been particularly poorly served was this. Dealing with things that happen, which may seem like a simple thing, but actually quite a lot of programs don't really tackle this. So if you're performing analytics on some data already in a data lake, or if you are dealing with things that are only in the past and don't re need to respond to new events, then this problem doesn't arise.
But a lot of systems are dynamic. Maybe you need to respond when. Financial information changes. Maybe you're working on industrial systems where you've got loads of sensors or machinery to keep track of. There's lots of kinds of systems that need to react swiftly to ever changing information or to be able to derive meaningful insights from streams of data.
Now, when it comes to data at rest, the kind of data that can just sit in a database we've long... had ways of representing tables, maps, and relationships and so on in our code, but live data representing actual events that are dynamic have always been a bit of a second class citizen. So although most languages have got a plethora of ways of dealing with tables and data and so on, they might have no intrinsic way to represent events.
Now, of course, C# and other .NET languages, these do have a way of supporting events, but they've always been a bit limited. So the number of things you can do with a C# event is pretty constrained compared to what you can do with a collection class. You can't even pass an event as an argument to a method, for example.
The Reactive Extensions for .NET, their goal was to change this by introducing a first class abstraction for event sources, elevating our ability to deal with things that are happening. to the same kind of level that we're used to having with just ordinary data in collection types. And they envisage this as a vital abstraction for connecting together layers in a cloud based system.
And of course, the cloud based future they envisaged back in 2008 has long since arrived. Now, .NET's design was copied into other languages, and it's been particularly widely used in JavaScript, and this kind of highlights Rx's basic soundness and usefulness and utility. Now, it's been widely used in its original platform, .NET, but it has had a slightly bumpy ride.
So now, I'm going to talk a little bit about how we got to where we are. Today, with Rx, in order to understand the context for what we've changed and what we're going to change in the future. So we need to understand how we got there today. This kind of shows the release timeline of Rx, all the different versions of Rx that appeared over the years.
Don't worry if the numbers are too small to read, they don't matter that much. The key points I'll highlight as we go along. Rx was first publicly unveiled back in the 2009 Microsoft Professional Developer Conference and the first supported release shipped in June of 2011, and then it was open sourced In 2012, it was actually one of the earliest Microsoft projects to do this and it was so early, it was on CodePlex.
This was back before Microsoft actually put things on GitHub. It was on CodePlex. Of course, it's moved over to GitHub now and initially Rx.NET was developed entirely by Microsoft, but once it was open sourced, it transitioned into a community maintained project. It's not exactly clear at what point the Rx.NET team went away, but it's been many years since there's been a team inside of Microsoft responsible for continued development of this. It's been a community supported effort for a long time, but development has continued. Version three was down here post this, downstream of this, and this added support for .NET Core and .NET Standard that were new back then, and then about a year and a half after that, Rx version four shipped, and this actually is going to be quite relevant, believe it or not.
It may be going back over five years, but this particular release has had an impact on the way Rx works today, and it's going to have a, it's quite significant to the work we're doing now. It added various performance enhancements, but it made one particular change that's very relevant to talk about.
And that was this. Before version 4, Rx was spread across lots of different NuGet packages, and it was actually a bit confusing working out what you needed to do. So one of the big ideas in Iron Rx version 4 was that it was going to unify this. There was going to be a single... Package, system.
reactive, that you add your reference to on NuGet and you... Add a reference to that, you've got all of Rx, and that's how it is, and it remains that way to this day. That was done back in version 4, and it's still the case now, but as we're going to see, it's actually been increasingly problematic in recent years.
It was a decent idea at the time, but it's now caused repercussions that we need to fix. That happens, whenever this was, around about 2018 or just before, there was a minor release coinciding with .NET Core 3.1 coming out. By the time we get to this point, it's relatively unusual for Rx to be getting brand new features because it's been around for quite a few years now, or about a decade by this point.
Bugs were not being found all that often any more. Now, the next big update was actually deliberately timed to coincide with .NET 5's. This added full support for nullable types in C# and a few other C# language features. Now, when this happened, the people who were maintaining Rx at the time took a decision to position Rx version 5 as being part of the .NET 5 release wave.
Now, it was a total coincidence that the version numbers actually lined up, that .NET happened to get to version 5 at the same time as Rx did. There was no reason for that, it just happened to work out that way. But the Rx team the Rx maintainers decided to say no, we're part of .NET 5. And with hindsight...
That was an unfortunate choice, because for various reasons, it was about this point that the people who'd been maintaining the project ended up moving on to other things. A year later, .NET 6.0 shipped, and no new version of Rx appeared. And that might have gone unnoticed if they hadn't made such a big thing of Rx v5.0 being part of the .NET 5 wave because it meant people expected Rx v6.0 to ship when .NET 6.0 came out.
So it looked like Rx was out of date. As it happens, Rx v5.0 works perfectly well on .NET 6.0 and on .NET 7.0 and it actually works fine on .NET 8.0 as well. By the time we get to this point, after .NET 6.0 and .NET 7.0 are shipped and there hasn't been a single new release of Rx, not even a minor release, people are starting to ask, has this project died?
And no one was responding to issues being reported on GitHub. So although... This wasn't technically the longest gap between major releases of Rx, it was the longest period that we'd ever gone without having any kind of a release. And people started to lose a certain amount of confidence in the project.
And although... Rx was working fine on .NET 6.0 and .NET 7.0. It didn't have support for, it didn't have explicit support for things like ahead of time compilation and trimming, which meant that it was starting to look like a less good fit for the way .NET runs today. At the start of this year, 2023, my employer, endjin, stepped in and offered to take over maintenance of the project, because it had clearly ground to a halt.
Now, we've actually already been involved in a related project, so back in May of 2021, we helped get the Reactor project open sourced. So that is a server side host for durable, reliable Rx subscriptions, which unfortunately I don't have time to talk about today. But our history with working on that.
Kind of paved the way for us to become the new maintainers of Rx round about the beginning of this year. So that's why I'm now the lead maintainer. So we published a roadmap of the goals that we had for reinvigorating Rx and I'm going to talk about what those are in a moment. We shipped Rx version 6 in May of this year, which addressed the first set of goals in that roadmap.
and we also made a preview of the async version of Rx available. I'll talk about that in a moment. And with .NET 8.0 having arrived, we've also... Added support for this release as well. If you go and look at the .NET repo up on GitHub, the Rx repo, sorry, up on GitHub, you'll see the most recent commits are around ensuring that the automated build system runs tests against .NET 8.0 as well as 7 and 6.
So we fully support .NET 8.0 today on Rx version 6. WHat about this roadmap then? What are the things we've changed? What are the things we're going to change? The first thing we actually looked at was the lifetime and support policies of several different frameworks that Rx is widely used in. So Rx is popular in user interface code, for example.
We can't just look at and .NET Framework. We also have to look at things like MAUI, Xamarin, Unity, Avalonia. Also, Rx has had support for UWP for a long time. And although people are discouraged from using that today, it is still supported by Microsoft and will be for years to come because it's a Windows support thing.
We need to define. Rx's position on that, and so this diagram is basically a snapshot of the foreseeable lifetimes of all the different platforms that Rx needs to be able to run on, and this informed our plans for what we're going to do next. What were those plans? At the start of this year, we published up on the repo, so GitHub.
com slash dot .NET slash reactive, that's where you can find it. We published a roadmap that was essentially in two phases. We had plans for a 6.0 release, which has already happened, that happened in May, and also plans for a 7.0 release. Now actually, with the 6.0 release, our main goal was really to unblock ourselves, because developments are basically ground to a halt with .NET 5.0, so Rx 5.0, and so the source code was stuck in that era and so It didn't actually build on the current version of Visual Studio.
You had to have an older version of Visual Studio just to build the source code. So our first job was to enable development to continue. So that meant we were deciding just to update the tooling, not to make any major changes. That in turn meant we were going to carry on supporting everything we already supported, including UWP, which had some consequences because it turns out that the XUnit project has effectively dropped support for that, so we needed to change to a different test framework just so we could carry on running the tests on Visual Studio 2022.
We removed references to old target frameworks, so .NET Core 3.1 and .NET 5 had been deprecated at this point, so we took out those target framework. moniker and we added .NET 6.0 as targets. We chose not to add .NET 7.0 as an explicit target. We just said we support the .NET 6.0 target on .NET 7.0 today because There's no .NET 7.0 functionality we want to take advantage of in Rx, so it didn't really need a separate distro for that.
It was mainly about getting things building again, which was actually surprisingly extensive. But we did some other things as well. We looked at the PR backlog and decided to take a few of the changes people had offered, but which had never been looked at because the previous maintainers had to go on to other things. So we examined those, we merged some of them in. We also had long discussions with the Reactive UI team. They're a big user of Rx and they have had some long standing problems associated with a long running Rx issue. So we had some discussions with them about that and added some functionality designed to address Fair scenario.
So mostly this was about tooling updates and unblocking the progress and fixing some bugs, but there was one new feature. So the one and only new feature that we have is we added explicit trimming annotations. So trimming is really important in certain deployment scenarios, right? So especially Blazor, if you're doing Client side code.
If you're building Blazor apps that run on the browser, download sizes matter a great deal because they affect page load types. And to give you an idea of the impact with Rx v5.0, if you added Rx v5.0 to a Blazor application, it instantly made your download one and a half megabytes bigger. In Rx v6.0, it's just 32 kilobytes.
So 1.5MB down to 32KB, because we enabled trimming properly. We added all the trimming support. So that was the one new feature added in 6.0. Okay there were some other things as well around Rx. AsyncRx, let's talk quickly about that. We sh A few years ago, Bart De Smet, who was one of the main developers on the original Rx team, added an experimental implementation of something called AsyncRx.
So you know in normal. .NET, we have IEnumerable
So this has been in the repo for a very long time, maybe four or five years, I think, but it had never been released to NuGet. So in May of this year, we made the first preview release available. Be aware. The code is experimental, so it's not of the same quality or maturity as the rest of Rx. So it's got no test suite, for example.
However, it's out there. People have started to try and use it and we've had some feedback. We've actually fixed some bugs on the basis of that. So it's out there and we plan to improve its quality over time. Another thing on they asked The first part of our roadmap was to improve documentation and actually the way we're going about this is we've updated a free online book.
So there's this website called intro to Rx. com that was created by Lee Campbell about a decade ago. And it's a great book that describes loads of detail about Rx, but he was never able to update it after writing the original edition. So it's all based on Rx version 1. But he has very generously given us.
The the rights to make a new version of the book. He's basically donated the book to the project and allowed us to do a new edition. We are going to publish that any day now. We were hoping to get it out by the time this talk started. It's not quite out there yet, but it'll be out there really soon.
That's another big deal. So much for what we've already done. Now, let's talk about what we intend to do next. The 7.0 plan on the roadmap, one of the things we put on there was .NET 8.0 support. Actually, we've been able to add that to Rx v6.0. Like I say, we've just published test suites for that, so Rx v6.0 is fully supported on .NET 8.0
what we might do... With the next version of Rx is add a .NET 8.0 target framework moniker if there's anything that we want to do that can take advantage of .NET 8.0 that won't work on older versions. We don't yet have anything identified for that, but it's something we're open to doing. So if it looks like there's a way we can make Rx better by building a .NET 8.0 specific version, then we'll do that. But If not, yeah, it'll work the same way Rx v6.0 does. It'll be fully supported on .NET 8.0 really, there's actually a much bigger goal for this release. We have one major, much more ambitious goal for the 7.0 release, which is to fix problems that have been afflicting certain Rx users who use certain .NET deployment models.
Anyone working with the Avalonia user interface framework will run into this, for example, but it can afflict anyone. The problem that you see... Manifests as a sudden huge increase in the size of an application deployment when you add a reference to Rx. Your deployable files suddenly grow by 30 megabytes, which is not good.
This only affects client side apps. You don't see this if you're building web apps or other server side code. This is a client side only thing, but we really want to fix this because Avalonia has actually stopped using Rx because of this. Maybe they'll never come back, but we would at least like it to be possible for them to start using it again.
So we want to fix this. However, um, it's complicated. And there's no good solution to the problem. We've actually got a few options, all of which have their problems. And we're actively soliciting feedback on what you think the best way for us to proceed is. Which is why I'm going to talk in a little bit of detail about...
What we're trying to do, and the problems we face, and what the options are. The origin of this problem is the thing I mentioned earlier. Remember I said that when Rx v4 shipped, they decided to simplify the way it was packaged, so you just have a single NuGet package called System.Reactive. And there were plenty of things that were good about this, it worked fine at the time, it actually happened to solve a slightly obscure But really important bug that would afflict certain kinds of plug in scenarios, so it's sold to stuff under the covers, and it was just simpler, and it worked great at the time, but at the time they did this WPF and Windows forms were only available on the .NET framework, and this consolidation all went a bit wrong once .NET Core 3.
I think that's it. added the ability to use WPF and Windows Forms in the .NET Core world. So this is what causes the 30 megabyte growth. So why is this happening? Rx detects when you're using a platform where WPF and Windows Forms are available, and if you are, it's It makes those features available.
It makes the Rx integration features for WPF and Windows Forms available. So if you use Rx on .NET Framework, it goes, oh, you're using .NET Framework. .NET Framework has WPF and Windows Forms pre installed. So I will give you the WPF and Windows Forms integration features of Rx. Makes sense, right? The tricky thing, though, is how do you handle that on .NET, modern .NET, .NET Core as was, and .NET 6.0 and 7.0 and 8.0 today?
With those... The way it works is it looks to see what your target framework is for your application. If you are on a Windows specific TFM, so if you're doing .NET80 windows0.0. 19401 or whatever, it'll go, Aha! You're on a platform where WPF and Windows Forms are available. Therefore, I shall give you the WPF and Windows Forms functionality, whether you want it or not.
Why is that now a problem? It's because of things like self contained deployments. If you add WPF and Windows Forms dependencies. That's what makes your self contained deployments 30 megabytes bigger. You're copying all of the DLLs of those two frameworks in as part of your application deployment, even if you're not using it.
That is not good. This also afflicts ahead of time compilation as well, because the same basic issues apply. So fundamentally, the problem we have is that if it thinks you could use WPF and Windows Forms, it forces you to have them. That's a problem. How do we fix this? In a way, it's really simple.
We just need to unwind that consolidation decision that was made back in Rx v4.0. We need to go back to a world where it's separate packages. You only get WPF if you ask for the RxWPF package and so on. So in theory, it's nice and straightforward. And actually, there's a couple of reasons it's not as straightforward as you'd hope.
And the big one... is that if you want backwards compatibility, you can't just take types out of an assembly. If we published System.Reactive version 7 and it just didn't have these types, any applications that are already using that package and are built against an older version of Rx are going to get missing method exceptions at run time.
And that's fine if you're running the app and using it directly because you chose to update yourself and that's, you can fix it. But if you're using other people's libraries and they're using Rx, And maybe you didn't even know they were. You find those suddenly break because you've added a different library that wants a newer version and that's taken features away.
You just can't take stuff out of an existing NuGet package, certainly not one that's widely used, without causing problems. So we think we're going to have to introduce type forwarders. There's also a more subtle problem that we might need to actually break backwards compatibility specifically for UWP because it's causing a specific kind of grief that I'm not going to go into.
Our view is that we're okay with that because that's a much less widely used bit of Rx, but you might disagree. We've... We've started a discussion about this on the .NET Reactive repo, so you can see the link down there at the bottom, and we're asking for people's views on this, and it's already, we've got a very wide range of opinions on how we should fix this so far, but everyone disagrees about what to do, so we really want to know what you think.
That needs to be done, but it's not entirely clear how. We've got a few more things we're going to do as well, so that's the big one. We want to fix this problem where installers become unnecessarily massive. What else? We're going to improve Async Rx.NET. Oh, I saw a question go past, by the way.
Someone said, hold on, I thought you already could await an IObservable. Yes, you can. You can await an IObservable. What you can't do is you can't use an await inside an observer. Observers are the things that receive notifications. You can await an IObservable, and that's fine if you expect that IObservable to produce exactly one thing.
That's fine. That works. But if you want to receive all of the items from an observable source, and you want to be able to use async await as you process them, you just can't really do that today with Rx. It was not built to work that way because of when it was designed. Async Rx adds that capability in.
So that's the difference. We want to improve Async Rx.NET so we want to add a proper set test suite. What we actually want to do is work out how to apply all the tests that we already have for normal Rx and apply them to async Rx. Because pretty much everything that's true about normal Rx should also be true about async Rx.
There are some technical challenges to doing that, but we think we can do it, and so that's a plan to do. So once we've done that, we'll decide whether it's time to change it from an alpha to a beta. We're not sure yet. There are some interesting questions around what happens with cancellation and async.
That means we're not completely sure that the design that's there today is actually the right one. So we're not sure if we're ready to turn it into a beta, but that's the hope. So that's another goal. Yet another goal. is we'd like to address a historical accident. If you go and look at the Rx repo, you'll find that it contains the source code for the async version of LINQ To Objects, which is a bit of a surprise.
LINQ To Objects is the link implementation for IObservable. There's also a package called system. LINQ.Async, which gives you a LINQ to Objects for IAsyncEnumerable. That turns out to be in the Rx Repo. What on earth is it doing in there? The Rx Repo is also home to something called Ix.NET, the Interactive Extensions for .NET, which provide LINQ to Objects style implementations of all of the non standard link operators that Rx has, and so it was a natural extension to add the async support in there.
However, David Fowler has approached us and suggested that could move into the .NET Runtime libraries. We'd really like that to happen, so part of the next phase is to try and work out how that happens. So that's also very much a goal. Okay, that's everything we've done and everything we're thinking about doing next.
So what can you do now? You can use Rx v6.0 today on .NET 8.0 that is fully supported. If you have opinions on the packaging issues that I raised please go to the discussion up on the GitHub repo because we really want to know what people think and to get views on this because it's going to be a significant decision.
And also... Go and find the free Intro to Rx book. That will be up on the the same repo. It'll have its own website as well, but you'll be able to get to it through the Rx repo, so that should be appearing any day now. So please go and look for that soon. And one last thing. I want to thank a bunch of people without whom we wouldn't have Rx.
net. So obviously there's the original Rx.NET team for creating it in the first place, and I'd like to give extra special thanks to Bart De Smet who has continued to contribute to it for many years after the Rx.NET team ceasing to be a thing. Also I want to thank all the people who continue to maintain it and contribute to it once it became an open source project.
I couldn't possibly list everyone, but I really want to thank in David Karnock and Brendan Foster. Also thanks to Lee Campbell for giving us the text intro to Rx. And also the ReactiveUI team have been really super helpful in this work. Ani Betts in particular have worked with us to help us try and resolve some of the issues there.
And they're being really positive about helping us bring Rx. Into the future. Go get .NET 8.0, go get Rx, read our book when it appears, and please let us know what you think, and thank you all very much for listening to me. Fantastic, Ian. Hey, we had one question come up pretty early in your presentation when you mentioned Reactor, and the question was, what are the best resources to learn Reactor?
Ah, there is a website called https://reaqtive.net, and there's also the source code repo itself, but yeah, reaqtive.net is a good one. We've got a history of how it came to be there, we've got various essays we've written about it. That's the best place to look. Excellent. Thank you so much.