Skip to content
· 74 min watch
Ian Griffiths By Ian Griffiths Technical Fellow I

Explore Reactive Extensions (Rx) for .NET, a library for async & event-based sources. Learn how Rx aids in data processing on edge devices & Azure IoT hubs.

About this talk

dotnetsheff

The Reactive Extensions for .NET ("Rx" for short) are a set of libraries for working with asynchronous and event-based information sources. While Rx has become widely adopted in client-side code, its roots are in highly scalable server-side processing.

In this talk, Ian Griffiths shows why Rx should be a critical part of any .NET developer's toolbox, and shows a real-world application that illustrates how the powerful processing and orchestration facilities Rx provides can be applied to collect and process data in edge devices and then stream the high-value events to an Azure IoT hub for further analysis in the cloud.

Read the transcript

Introduction to Rx

So thank you all for coming to listen. Thank you for having me to talk about this. I'm going to talk today about the Reactive Extensions for .NET, or Rx as they're known, which I think are one of the most important features available in .NET, and possibly one of the least well known as well. It's useful in a wide range of things. It's basically for any event driven system: it's useful in cloud applications, client-side stuff, IoT devices, so I'm hoping to share my enthusiasm about this today.

So just quickly, I'll tell you who I am. My name is Ian Griffiths, I am a Technical Fellow at a company called endjin. We are a Microsoft Gold Partner consulting firm. I am also a book author, so I'm partly here to plug my book, which has a chapter on Rx because I like Rx so much, so I might mention my book once or twice during the talk. But mainly I'm here to talk about Rx.

What is reactive programming?

So what is Rx? So the Reactive Extensions for .NET is the official original name, although Rx is now available for a wide range of other platforms as well. It was originally a .NET invention, but it's become popular in the JavaScript world, and there's various ports for other frameworks as well. So it's increasing in popularity, to the point that it's sort of become arguably better known outside of its original home.

So what is it? Well, it's an event-driven programming model, but really what's more interesting to begin with is what can you do with it, what's Rx for? If you are familiar with Rx at all, you may well know about its use in client-side code. This is arguably one of the better known uses, both in the JavaScript world but also in the .NET world. So in .NET there are WPF based frameworks based around Rx, and also in JavaScript there are a lot of projects using it to deal with the flow of information from some underlying source up into the user interface. So the sort of push driven nature of Rx makes it quite a natural fit for client side code.

But if you think that's all it's for, then you're missing out, because that was not what Rx was invented for. Rx originated from a group in Microsoft called the Cloud Programmability Team, and what they were trying to do when they invented Rx, their goal really was to come up with a way of representing computations that could move around from one computer to another. So they wanted to be able to have a way of representing some logic in a way that could potentially run on any number of different machines in a server cluster, but which could also be pushed out to devices or maybe run on the edge. And so the idea was to have a unified programming model that could run anywhere, and for it to be suitable for event driven applications.

So I would say Rx is useful in any system where things happen. That's really its remit. So if you don't build systems where things happen, then this might not be for you, but I would say that's a fairly broad church of applications.

The observable abstraction

So let's take a look at it. So the heart of Rx is an abstraction called observable. It has slightly different names in each different flavor of Rx, but I'm here to talk about Rx.NET, and it's called IObservable<T> in this world, and it's a very, very simple concept. The basic idea at the heart of Rx is simply a sequence of things. So from an abstract point of view, it's the same concept as IEnumerable. So IEnumerable is just one thing after another, and IObservable is the same thing, it's just the manner in which the things are delivered that is different.

So whereas IEnumerable is fundamentally a pull-based thing, your code says, I'd like the next thing now please, I'd like the next thing now. Typically you write a foreach loop and your code is in charge of when to go get the next item. With Rx, by contrast, your code is reactive, your code responds to things that happen. So you write basically callbacks that get invoked every time something is available from this sequence of things. So it's a very simple idea. It's not complicated, it's not hard, but there's quite a lot of stuff around this that makes it powerful and useful.

A basic Rx implementation

So I will start with some code, if I can find where my Visual Studio window has buried itself. There we are, right. I'm going to build a console application to show what this looks like from scratch. Let's call this RxDemo, and hope I don't already have one of those. Shouldn't do, because all the rest had "dry run" in the title.

So here's a very simple .NET Core console application, and the core type that we're going to work with, IObservable of something, is baked into the .NET class library. So it's been there since .NET Framework version 4, which shipped in 2010, so it's been in there for half the lifetime of .NET, and it's been in there for way longer than .NET Core has existed. And so we can declare variables of this type, but what are we actually going to assign in there?

It turns out that this interface is defined in the .NET Framework, in the .NET class libraries, but there are no implementations built in in any version of .NET you're likely to encounter. In fact the only version of .NET Microsoft shipped where there were any built-in public implementations of IObservable was Windows Phone, so probably, I'm guessing, not many people are still writing for that at the moment. So we need a NuGet package to do anything useful with this.

So I apologize for the tiny fonts on this. There was a plan to do the video completely differently, that has utterly failed, and the fallback plan means that the font here will be illegible, but I will tell you. I think I'm typing System.Reactive as the name of the NuGet package, and so once I've installed this we'll be back in the realm of legible fonts.

So System.Reactive is a NuGet package. The source code for it is available in a GitHub repo. Oh, that's not open where I wanted it to be, let me bring up my specially pre-zoomed web browser. So if you want to go and look at the source code, it's in this GitHub project, dotnet/reactive. It's a .NET Foundation project, so it has backing from Microsoft, and it originated inside of Microsoft as a closed source project. It was one of the earlier open source initiatives at Microsoft, and now it's kind of detached from Microsoft, in theory, and is mostly maintained by various volunteers.

Observable.Range and Subscribe

So I have asked for that package, System.Reactive, and now I have a Reactive namespace in here which offers various useful things. So with that in place I can say I would like an observable sequence... actually this is in the LINQ namespace, never mind... an observable sequence going from one to ten. So I am deliberately doing something that I could equally have done with IEnumerable at this point. So the Enumerable class built into .NET also has a Range operator that says give me numbers from n through m, just to show you something that should be broadly familiar.

But rather than being able to foreach over these things... you can't do this, x in xs. If I try this, the compiler will complain. It will say, in probably tiny, tiny writing, that this is not consumable by foreach, because foreach is used for collections where our code gets to decide when to take the next item out of it. foreach is designed for pull scenarios, where my code says, I'd like to pull the next value out of this source now.

Whereas with observable, the whole point is it's push based. The source of, well, numbers in this case, is going to push its values at me when it's good and ready, so I have to provide some sort of callback, which I can do by writing xs.Subscribe, and in this case I am just going to dump the values out to the console. So hopefully that's visible: numbers one through ten. Deeply exciting stuff. Let me try and get that window into position where it'll be legible again next time.

Observable.Timer and LINQ operators

So obviously nothing I couldn't have done equally easily with IEnumerable, so let's do something slightly more dynamic. So rather than Observable.Range, I'm going to use a different built-in source, Observable.Timer, which looks different. It wants to know when it should fire, so I'm going to say DateTimeOffset.Now, oops, hello, stop. Now I'm going to add one and a half seconds, so we'll wait for one and a half seconds, and then I can specify a period, so how often should it keep going off after that time? I'm going to say, please give me an event every half a second.

And that's complaining because this actually produces an observable of long, so this is now a sequence of longs. If I run this, it actually exits immediately, because I haven't caused my Main not to return. So I am going to do this: I'm going to turn this into an asynchronous main method, and for now I'm just going to do this. I'm going to await a new TaskCompletionSource object, so that will never return, because I've created a task that never completes, so this should keep running until I kill it.

So now we wait one and a half seconds, oh, my zoom got lost, and then the numbers start to appear. Now that will carry on forever. Another thing I could do is say, well, this is sort of in a LINQ-like world, so I could say actually I would just like to take the first five values that come out of that. So the Take operator, which you should be familiar with if you've used LINQ of any other flavor, gives you the first five items from a sequence. If I run this, we get zero, one, two, three, four, and then nothing more, so this will now just sit and wait forever.

Also, because this is LINQ-like, I can do other things. I could do .Where x is divisible by two, so I'm going to take the first five numbers that are divisible by two here. Zero... so it's skipping out every other number, they're coming out at half the frequency now, because it's chucking away each odd item as it comes out. So when I said this is a lot like IEnumerable, I meant it.

They've provided, in the System.Reactive library, they have provided most of the LINQ operators that you'll be familiar with, and also a bunch that you might not be. So there are additional operators that are not part of the standard set of operators in LINQ. So you can for example say, I would like to group the items together in batches of 10. Now this won't compile, because now the items that come out of here are actually an IList of long rather than just a long, so that won't actually work right now, but you can do that.

You can do temporal joins, so you can say, I would like to get events from two different streams, and I'm going to define a means of identifying the duration of each event, and I'd like to see all the events that overlap in time. You can do things like that, which obviously you can't do in any straightforward way with IEnumerable. So because it's inherently kind of dynamic, it's based on things happening when they want to happen, so in this case according to a timer schedule, but I could build an observable that runs off the keyboard, I could do all sorts of things.

Creating custom observables

Let's quickly do that. Let's build an observable sequence of keystrokes, just to show how this looks. So I'm going to write a static, oops, method that's going to return an IObservable<char>, and this wants to return an implementation of IObservable. Well, I could look at this and say, okay, what does it take for me to implement this thing?

It doesn't look that complicated: one method called Subscribe, which, for reasons I'll go into shortly, doesn't appear to match the signature of the call I've made up here. That's actually an extension method. This is the one I have to implement, and then I have to invoke methods on this observer class that I'm talking about momentarily. However, if you implement IObservable, it's your job to manage things like multiple subscribers. You're supposed to return an implementation of IDisposable so people can unsubscribe from your source before it's finished, so there's a fair amount of work involved in that.

So I'm just going to ask the System.Reactive libraries to do all the work for me. So I'm going to return an implementation created by Observable.Create, and this lets me write either a normal method or an async method, and let's do this. You get passed an implementation of IObserver, and if you want to generate data you just call OnNext. So if I wanted to fake up keyboard strokes, I could just do this, of course: call OnNext one item after another, and it will look like someone's typed that.

So if instead... oops, that was not what I meant to do... of subscribing to this timer, if instead I say my xs are just this keystrokes source, then, oh yes, this is going to expect me to return a thing, so let's just cheat, quick and dirty way of getting a task. So hopefully if I run this, it will look like I've typed hello. Obviously I've faked up that data, so now let's do a real bit of code here.

And just do while for as long as someone's listening. Well, how do I know that? Actually, I'm going to use an overload of this that takes a cancellation token that will get set if someone tries to unsubscribe. So I'm going to say, for as long as that cancellation token has not been set, then we're going to loop around and we're going to say... oh yes, we need to ReadKey, don't we, and then it's KeyChar, and then OnNext(c).

So if I've not made any stupid mistakes, if I run this and press a key, I get double of every letter, because of course the console is already echoing output for me, and I can't type well without a backspace key. So that seems to be working. So then if I wanted to, I could add more of these operators, so I could say, well actually, let's do Where char.IsUpper. So I'm going to filter to say I'm only interested in characters that are uppercase. So if I start typing, that's the echo, but if I do uppercase letters, they get fed through.

So the basics then are an observable sequence of things to which I can subscribe, and I can apply LINQ operators, and the Rx libraries give me an implementation of those operators, and they give me a fairly straightforward way of implementing IObservable. This takes care of all the work of having multiple subscribers and so on, although there are some caveats to that, but I'll talk about that when I get to my main demo. But this is the basics, so I'm going to return to my slide deck at this point.

Comparing .NET data models

So an observable is a sequence of things that you can subscribe to, and it will call you back each time it has a new thing for you. So at this point it's worth just thinking about where this fits in, in the world of .NET types. If you're writing something that provides information, so maybe a property on an object, or maybe something you're returning from a method call, you need to think about how to represent that. And in some cases it might be a very simple data type, if you've got one value to return.

If you've got an object representing a person and you'd like to represent their display name, then you could write a property of type string that just returns that value. Easy, you're done. So that fits in this diagram in the top left-hand corner box: we've got one piece of data to return, and we are able to support a pull style API. We should say that the code using my data is in charge of when it fetches the value. It's going to pull data out of my object by invoking the property getter, as opposed to having to sit and wait for that data to become available.

So what if I've got a single value but I can't necessarily provide it immediately? What if I've got to go and hit a web service to get hold of that thing? Well, there I'm going to have to push the value at the caller at some point, or, either that, I'd have to make them wait. I could make them wait, I could just say, okay, fine, I'm going to block your thread until I'm ready to give you the value, but that's not great in most applications.

You don't want to do that on server apps, because you will starve your thread pool. You don't want to do that on client apps, because you'll make your app unresponsive. So synchronous APIs for slow things are not good. A pull style API doesn't look a great bet in that case. So classically in .NET, the way to do this is to use the Hollywood principle: don't call us, we'll call you. You have some sort of API where you provide a callback that gets invoked with the information when it becomes available.

And actually the Task<T> class, under the covers, works in exactly this way. If you decide that the way you're going to present your data, which will become available eventually, is to present that as a Task<string>, which is a pretty common way of doing it, then under the covers the way Task<string> actually works, if the data isn't already available, is it has a method called ContinueWith, to which the caller provides a callback which will be invoked when the data becomes available. Now most of the time as C# developers we don't actually interact with the task that way. We use the compiler's async and await stuff to do it, but the actual underlying model is fundamentally callback based.

Or you could do it with events, or you could do it with an ad hoc thing where you just have a method that takes a callback as an argument. Lots of ways you can do it, but basically if you've got a single piece of data that arrives on a schedule determined by the thing supplying that data, then some sort of callback is the way it will go.

Multiple pieces of data: IEnumerable or IObservable

What if you've got more than one piece of data to return? What if instead of a name property you have a favorite colors, plural, property that wants to return lots of things? Well, if you're building a pull style API where the code using your object is in charge of when it's going to ask for the data, then you can just return an IEnumerable<string>, or possibly an IList<string>, one of the various collection types .NET offers. There are various different ways of returning collections of strings.

But what if it's going to take a while to produce the data? You might be tempted to say, well, let's do task of IEnumerable<string>, and that might be fine if the slow operation that is required to get the data gets all the data in one step. If you're going to get all the data in one big lump, then a task of IEnumerable<string> or a task of string array or a task of List<string> is absolutely fine. But if you have a data source that continues to provide data over time, then that's not such a good idea.

Because the problem with a task of collection of some kind is that you get exactly one transition between data not here yet and data here now. And if there's a single event corresponding to that, that's great, but if there's a constant stream of data coming in, that's a lousy model, because once you've completed the task once, then that's it, you've used up your one go at being async. But IObservable with Rx doesn't have this problem.

With IObservable, the source gets to provide its data items one at a time. You saw in that code I just wrote, I was able to call OnNext on the observer that has subscribed to my observable. I get to call OnNext on my schedule. I am pushing data out of my object into the receiver, so that's a much better model if your data is arriving on a schedule that is externally imposed.

Where does IAsyncEnumerable fit in?

So that used to be the whole story. It used to be that in talks like this I would say, well, for singular data you either have a string or a Task<string>, and for multiple pieces of data you either have an IEnumerable<string> or an observable of string, and that would be that. However, some of you might at this point be thinking, where does IAsyncEnumerable fit into all of this? So in .NET Core 3, a type called IAsyncEnumerable<T> shipped, and this muddies the water a little bit.

But it's worth talking about, because otherwise it's not really quite clear when you should and shouldn't use IObservable, because some of the scenarios that I would previously have recommended IObservable for in the past, I would now say actually IAsyncEnumerable is a better bet. So let's look at this. There are actually three dimensions, which sadly makes for a much messier diagram.

So we have, how many pieces of data are there, one, or are there several? Secondly, who's in charge? Is it a pull API, where the thing using the object is going to manage the flow control, it's going to say, I'm fetching this data now, I want to retrieve this information or this list now? Or is the data source in control, is it a push style API? And third dimension, is it synchronous or asynchronous?

And these overlap a little, because there aren't kind of separate answers as to what to use for all possible combinations. You'll see some of the types, the things I've mentioned here, show up in more than one box. So let's look at this. For pull, if it's a synchronous API, so the data can be supplied immediately, you just return a string, nothing changes.

If you want to support the use of the await keyword, so if you want it to look like it's a pull API but to support asynchronous population of data, so to actually have the caller be able to return, unblock the thread, let that thread get on with something else, and then continue when the data becomes available, so you have a pull programming model but under the covers is a callback based implementation of that, then Task<T>, Task<string> in this case, is the way to go. For push style of single data, then some sort of callback, doesn't really matter what. There isn't really one standard idiom for this in .NET, so do whatever you like.

But for the many case, this is where it gets a bit more complicated. If you want to support a pull style API, and again I mean that's one where the code using your object is in control of what happens when, then if you can provide the data immediately, if you don't have to block and wait while you fetch the data, then IEnumerable<string> continues to be a good choice. But if fetching the data is slow, and it might not even be a single operation to fetch it, it might be we fetch a page of 50 items, then we can start returning them, but then we've got to go and hit the web API again, get the next page of 50 items, start returning them, so it might be sort of slow operation, fast, slow operation, and that sort of thing. Or if every single thing is slow.

If you're in those situations, but you still wish to enable users of your code to act as though they're in charge, then that's where IAsyncEnumerable comes in. That gives you await foreach, is what that does. So the await foreach construct that was new in C# 8 is given to you by async enumerable, and that's a perfectly reasonable thing to do, and all the LINQ operators are available for that. So if a pull style API is what you want, then great.

However, it's not a natural fit for certain kinds of things. IObservable is a better model for certain kinds of processes, and I'm going to show you a real life demo in a bit where I'm using IObservable, and I will maintain that it's a better abstraction than IAsyncEnumerable would have been. Even though these three interfaces all fundamentally represent the same thing, they are all sequences of things, IObservable is, from a developer perspective, a more natural fit for autonomous things that throw out data from time to time on their own schedule.

Observables and observers

So let's look at these a bit more closely. So IEnumerable and IObservable are really, really closely linked together. They're not just conceptually similar, they're not just sort of the same in a hand wavy sense. They've actually been deliberately constructed to be, in a sense, the exact mirror image of one another, in that it is completely possible to transform an IEnumerable into an IObservable and vice versa, so long as you don't mind tying up a thread in the process of doing it. You wouldn't typically want to do that, but conceptually they are indistinguishable, and this goes quite deep.

So with IEnumerable, let's just quickly review how this works. Typically we consume it with a foreach loop. Each time a foreach runs, it will call GetEnumerator and you'll be returned, logically, a new IEnumerator. It might not necessarily be a new object, but logically it's a new one each time, and this enumerator remembers where you are in the list. And foreach walks through that enumerator and then disposes it when it's done, but each time you do a foreach you get another one of these, and for some sources you can have several of these on the go simultaneously.

So let's compare this with observable. With observable, the consumption model is you subscribe, you give it some sort of callback, and when you do this, you call the Subscribe method, you pass in something, and you get back an IDisposable. Now this, believe it or not, is there to model the break keyword in C# on the left-hand side. Because the thing about a foreach loop is that nothing forces you to go all the way to the end of the sequence. If you want to, you can say, I'm bored now, I'm going to break out of this loop, and that's fine, because it's a pull API, you're in control.

It can't force you to keep on calling MoveNext, MoveNext, MoveNext, so you're in control. Do you want to abandon it? You can, you just break out of the loop. The C# compiler has generated a try/finally block for you that will dispose the enumerator, and then you're done. With IObservable, we're not the ones in control of what happens when, but we still need some way of saying, I'm done now, I want to let go of this thing I subscribed to, because I no longer care.

And the way they do it there is that when you call Subscribe, they return you an object that represents that subscription, and it actually implements IDisposable. So if you want to hang up early, you can just call Dispose on that subscription and it will stop calling you back. And again you can have many of these simultaneously, and actually it's quite common to have multiple subscriptions attached to a single observable source in Rx. And the thing we actually have to pass to an observable source, ultimately, is a thing called an IObserver.

Demo: implementing IObserver

And I'm going to go into a demo now, just to show what these actually look like. So I did it the lazy way earlier in my demo. I just said, I'm going to use the extension method provided by the Rx libraries that lets me pass in a delegate, and it will then actually implement IObserver for me. But now I'm going to do the slightly longer-winded way. I'm going to write a class that implements IObserver of... well, let's make this generic, I don't really care what type it is, it's going to do a Console.WriteLine on it.

So this is the interface you must implement, so let me fill that in. And I don't like the order that it puts these in. I know it's alphabetical, but it doesn't make sense to me, so I'm going to reorder them in the way that does make sense to me. So, oh yes, I probably need to declare that this is generic, don't I? Right, so we have to provide three methods.

OnNext, so this corresponds to going around the loop body of a foreach loop once. So I'm saying enumerating an IEnumerable with a foreach loop is exactly equivalent to subscribing to an observable with an observer. So when I subscribe to an IObservable, it's like I'm looping over all the items in that source, and it will call me once for each thing. So essentially this code here is where I put what would have gone in the body of the foreach, so I can just write out the value here.

So what are these other methods about? Well, I need to know when I'm done. With an IEnumerable, you just keep calling MoveNext on the enumerator, and that eventually returns false to say there's no more data. So this is equivalent to that thing returning false. Basically we get a callback saying when we're done.

And the other thing is that when you enumerate over an IEnumerable, when you call MoveNext, it might throw an exception at you. That's a thing that can happen. And so because IObservable is trying to be a literal, if reshaped, translation of what IEnumerable looks like, they said, we've got to support that. That's a thing that can happen to you as you iterate over a collection of objects with IEnumerable, so we have to be able to represent that if you're using IObservable as well, and so we need to have a way of discovering when it's all gone horribly wrong.

So with this, I can now, instead of using the built-in implementation, sorry, instead of using the one supplied by the library, I can now just say I'm going to new up Obs<char> and pass that in. And now I'm not using an extension method, I'm just using the raw Subscribe method that's actually defined literally by the IObservable<T> interface. So this should do exactly, well, more or less the same. So remember we're just looking at uppercase letters, and I'm going to do one last thing. I'm going to say TakeUntil c equals uppercase Q, so that this thing now has an end to it.

It would help if I actually made that a proper lambda, wouldn't it? So let's run this one more time. So that's echoing, and when I hit Q, then I get the Q, but then it says I'm done. So the Rx libraries have noticed that the TakeUntil operator has said, we're finished now, and so I'm done, and now it's no more. And actually my loop for reading keyboard input's no longer running, because Rx has basically shut that down, because it knows no one's listening anymore, or at least it's not plugged into anything. So, right, so that's what observers look like.

Scheduling and bridging

So let's just talk a bit about this split between the NuGet package and the stuff that's built in. So just to clarify, the core interfaces, IObservable and IObserver, which are the counterparts of IEnumerable and IEnumerator, are built into the .NET Framework. And then you have the System.Reactive package, which is where most of the stuff that makes this actually quite powerful lives. This is where all the LINQ operators exist, Where and Select, and more exotic ones like Buffer and Join, temporal joins, temporal groupings and so on.

And it also provides some other services. It provides a scheduling facility. So you might have been wondering, when the timer ran, who exactly was running the loop that invoked my callback for that timer? How is that actually working? Well, the Reactive framework defines a notion of a scheduler, and it provides various implementations. There's a default one, but there are other ones as well.

There's a different scheduler to use if you're running in a user interface application, for example. You can bring your own scheduler. You can choose whether you want to have things run on the same thread as you, or to run on a thread pool, there's lots of different ways of doing it. And also the scheduler provides a virtualized notion of time, which is used for two reasons.

One, it means if you're doing time sensitive operations, such as the timer based thing, or if you're grouping things together in time-based windows, or whatever it is you're doing, you might want to be able to write tests that validate your code without actually having to wait for real time to elapse. So for testing purposes you can plug in a fake scheduler that makes time run at whatever speed you would like time to run. And in fact they even provide you with a built-in historical scheduler for faking up the passage of time.

That's useful not just for testing, though. I've worked on applications that do data processing on servers using this stuff, where we run the same queries over live streams of data but also historical streams of data that we've got stored in a database. So that can be useful if you want to go and do a what if with a new query you're developing. You want to say, well, I know I need to respond to these kinds of events, let's go and run that over last Tuesday's data, because we know that contains an interesting event that we wish to respond to, and just see if it works.

And if you're running over historical data, you can use a virtual scheduler that advances time at the rate at which the events originally happened, but you can churn through them as fast as you can possibly do it. So it's actually useful to be able to virtualize time in application code as well.

Bridging between Rx and the non-Rx world

And the System.Reactive package also defines various things that let you bridge between the world of Rx and the world of not Rx, and you've already seen one of these. Observable.Create lets me write a thing that returns a task, and I used Task.Run in my example, but you could equally do an async method. You can write a task that is handed an IObserver and just write code that generates the events. So that basically lets you get into the world of Rx, and it takes care of all the things like subscription management, cancellation and so on for you.

If you throw an exception out of your code without handling it, it automatically maps that into a call to the OnError method of every single subscriber to your event source. If you just return off the end of the method without calling OnCompleted, it will call OnCompleted for you. So it handles all those details that you have to get right with Rx.

And another thing it does... actually, quickly go back into my demo, just so I can show you this. I could do this. I'm going to do some magic which I'll explain in just a second, and then I'm going to say, rather than awaiting forever, I'm going to build something based on my keystrokes observable. I'm going to say, actually I just want the last item, and for various historical reasons that operator is called LastAsync in Rx rather than Last. It should be called Last, but it isn't.

And then, well, I could store this in something of type IObservable<char>, or I could just await it. It turns out that the Rx libraries define a GetAwaiter extension method for IObservable of anything, meaning you can just await the thing, and if I wanted to I could actually store that in a variable as well, that will work as well. So this awaits... actually, sorry, technically it waits for the first thing. In this case the first thing is the last thing, because I said give me the last item in my source.

So this will actually wait for the first thing the sequence produces, which in my example is the one and only thing the sequence produces. I don't really care about the outcome, but the significance of this should be that if I start typing and then hit Q, my program then exits, because I'm no longer waiting forever. I'm doing an await until that particular observable sequence reacts, to use the parlance, so reacts means either it produces an item or it throws an exception. So that is one of the bridging features that's enabled in the Rx library, the System.Reactive library.

So they provide this GetAwaiter that returns a specialized .NET type, so it doesn't return the usual Task<T> here. Oh, and IntelliSense is being very slow, oh, there we go. It returns a special thing called an AsyncSubject, and that provides the methods that C# expects to be able to implement the await for you. So that's bridging technology: it bridges between the world of IObservable and the world of async await. So let me put that back.

A real-world IoT demo: monitoring sleep apnea

So now I want to show you a real application, so I'm going to give you a little background with this, because this is not a commercial application, it's a personal project I've worked on for my own benefit. So earlier this year, around about the beginning of March, I was diagnosed with a condition called sleep apnea, which is a lovely medical condition in which you stop breathing in your sleep. This is not good. And I was going to get treatment for that in early April, and you can probably guess how that worked out.

So I still have this condition, I'm getting no treatment for it, so I decided I was going to at least try and monitor the situation by myself. So I am quickly going to show you a zoomed in thing, let me quickly do this. I apologize for showing the magic and hiding OBS, but this is a device called a pulse oximeter. It's a little kind of thing which you can put on the end of your finger, let me get that the right way around, and hopefully in a couple of seconds you may be able to see that it is drawing a little kind of bar graph thing along the bottom there, and it's also showing a couple of numbers.

99, that is the reported oxygen saturation in my blood, and then the other one is my heart rate, because I get very excited when I'm delivering talks. Let me move this back off the screen, because I don't particularly like this 1970s video effect. So I have this pulse oximeter, and I got it because I wanted to monitor my... let me put this back into less stupid mode... I wanted to monitor my oxygen levels in my sleep, so I could work out how severe this was, to see if I needed to seek more urgent medical attention.

And it's quite hard getting devices that are able to report their data reliably. So this particular device I'm using implements Bluetooth LE, Bluetooth Low Energy, and the app that comes with it is dreadful. Basically it doesn't work unless you have it as the foreground app on your phone and you configure the phone's screen to stay lit up at all times. If the screen shuts off, 20 seconds later the app gets killed and it stops logging data. If any sort of communication error occurs, it stops logging data. It's not very good.

Discovering Bluetooth devices with an observable

So I thought, I can do better than this, so I wrote an application for a Raspberry Pi that goes and talks to this device. So I'm now going to show you the source code for that application. So this is using .NET Core, and so it's running on a Raspberry Pi, a Raspberry Pi 4B as it happens, which has built in Bluetooth support, and I am using the Bluetooth APIs supplied in user mode by Linux. So I decided to model various things in this application using Rx, and the first thing was device discovery.

So the thing about Bluetooth is that to talk to a Bluetooth device you first have to see it. So my Raspberry Pi has to see my pulse oximeter actually broadcasting its presence to the world, and Bluetooth stacks and operating systems have a whole mechanism for device discovery, where they will tell you about devices that appear. And I thought, well, this is very naturally modeled by observable. So I've written a method which presents an IObservable of thing, I'll talk about the significance of that name in just a minute, but basically it presents an observable sequence of devices.

And this is using the same Observable.Create method you just saw me using in my demo, and then it disappears off into a load of weird calls into the API layer to actually do the work. But the point is this is going to report each new Bluetooth device that becomes known to the operating system. And actually it's a bit more complex than that, because when you first ask the operating system for all the Bluetooth devices it knows about, it reports all the ones it can remember as well. So anything it's ever had a conversation with in living memory, I've yet to establish exactly what defines living memory, but certainly it seems to be a good few days, it'll report stuff that it has seen lately as well as the stuff that comes online.

So when I first ask for this, this observable sequence is going to report 10 or 15 devices that it already knew about, and will then continue to monitor the airwaves to see if any new ones come in range, and will report those as well. So I think observable of something is the natural representation for this, because it's a sequence of things. It's like, oh, new device, oh, new device, and those things become available on a schedule that is totally outside of the control of any of my code. It's determined by when things are switched on or when things come into radio range, so observable is a very good way of modeling that.

And this means I can use things like the First operator to narrow it down. In this case my particular application is looking for a very specific device, so I can pass in as a command-line argument the device address, and actually I've coded the MAC address of my particular device into the app, so I don't even have to do that. But anyway, I'm looking for my pulse oximeter, and so this says, I'd like to look at this device discovery source until we find the first item whose address is the one I'm looking for.

And then I'm using the same await thing to actually grab the result that you saw earlier. This will stop as soon as it finds the first one of interest, it will automatically unsubscribe from that event source. So I've got my device, so that's my first use of Rx in this demo.

The Nordic UART as an observable stream of bytes

So then, once I've got the device... so I'm reporting these things as a class I've written called BtleNordicUart. What on earth is that? BTLE, that's Bluetooth Low Energy, that's the variant of Bluetooth being used. Nordic, that's a company that makes system on chip solutions. They sell various silicon chips that have everything you need to build a tiny computer, and they build the one that's in this pulse oximeter. It's an ARM-based device with a Bluetooth implementation.

It doesn't really matter what it is. The most interesting thing about it is that it presents streams of data in a particular way, so I've written a wrapper of this device to present the stream of bytes that come out of it. So what this pulse oximeter is going to do is it's going to send byte after byte after byte reporting various things about what it's measuring, and essentially it's pretending to be a serial port, is what it's doing, if you're old enough to remember them. That's why it's called a UART.

So they literally are modeling a universal asynchronous receiver transmitter, which is a fancy name for a serial port, simulated over Bluetooth, which is a triumph of technology over itself. It would have been much simpler actually just to use Bluetooth, but never mind. So in this class I've written, I decided, well, the device is generating byte after byte after byte, that's a sequence of things that arrives on a schedule determined by something external to my system. What on earth could I use to model that? How about IObservable of bytes? So that's what I've done. I have written an implementation that presents the data from the device using that.

I'm going to quickly show you the code, and you might want to tear your eyes out after you've looked at it. So again I'm using this Observable.Create thing, and this device is really flaky, so we basically have to loop around and keep reconnecting every time it fails, because it just sometimes stops talking to us. But then I attach to the property watching API provided by the Bluetooth stack, and I watch for changes on the device, which include the device randomly disconnecting for no good reason, so we watch for that.

But then the thing we're interested in is a particular service exposed by the device. This is the thing that's particular to the Nordic device, in theory at least, although this is such a popular device that there are a lot of libraries out there that enable you to emulate a Nordic UART on any Bluetooth device, so actually it could be anything that you're talking to. It seems to have become an ad hoc protocol for doing streams of bytes. Anyway, eventually down here we end up calling OnNext to report the data, and the rest of this is just error handling, really.

Decoding the byte stream into messages

So the net result is, once I've got the device represented by a UART, I can have a stream of bytes represented as an IObservable of bytes. However, I don't really want to write application code that looks at the bytes. I would rather have something pull those apart into the messages, because it turns out, after spending a couple of hours looking at the bytes coming out of the device, I was able to work out roughly what it was doing, and it turns out to send messages of various different kinds. So what I would like is not a sequence of bytes, I want a sequence of messages.

You'll notice I've got an observable of Pc60Message. PC60 is the device type, by the way. It is a Shenzhen Creative PC-60FW fingertip oximeter, it says here, so that's the device type, and these messages, as far as I know, are bespoke to that. And so I've got this extension method here which takes an observable stream of bytes and does stuff to turn that into an observable sequence of messages, and I'm using various Rx features here.

I'm using Rx's built-in Timestamp operator, because it turns out the device doesn't have a clock and doesn't send you any timing information. You're supposed to infer the timing information from when you receive the message. So the first thing I do is I timestamp every byte that comes in. Then I use this Publish RefCount magic thing I used in the earlier demo. Basically this is an Rx-ism that enables you to support multiple subscribers with only one underlying subscription to the underlying observable.

This basically says, I don't care how many people subscribe to this, I want this implementation of IObservable to see just a single subscriber. So notice on this occasion I've actually implemented IObservable myself, because it was actually more convenient than using the Rx wrapper classes, because I want an object that kind of maintains state. So this thing does the protocol decoding. It receives bytes and notifications from the source thing, and this is basically a state machine that decodes the incoming protocol, and eventually somewhere here it spits out a fully formed message, and has to deal with things like the device forgetting what it's doing halfway through and restarting and so on, because this is a real device and you have to deal with these things.

Azure IoT Hub and Time Series Insights

So we have a stream of messages, so then what? Well, at this point I'd like to just show you what the effect of all this is by going to this web browser. So I have here, up in Azure, a resource group that has an IoT Hub, so this is using Azure's Internet of Things features to receive streams of data from devices. So I've defined an IoT Hub, which is what this application on my Raspberry Pi is going to send events to, and then I've attached that hub to a thing called the Time Series Insights environment, which lets me go and look at the data.

So if we look at this, give it a couple of seconds to start up, this is going to report the data from my device. And if I go take a look at this one here, it lists all the different devices that have reported data, well, there's only one in this case, so that's fine. And I'm going to say I would like to see the blood oxygen saturation, the heart beats per minute, and the photoplethysmography values. That's the actual measurements the device is really taking, from which everything else is derived, and it shows me nice bar graphs for this thing.

So I am just going to zero in on this a little bit more, because it turns out there was a bug in my driver when I was doing this that means sometimes it fails to notice magic values that are way out of range, for various technical reasons. Let me just do this, we can see real looking data if I just hide most values. This is my blood oxygen level as I sleep, and I can see a minimum value of 92. That is actually the threshold of clinical significance, so I'm glad I was measuring this.

And this is all live. If I scroll over here and look at this end of the bar chart, you can see there has been some data recently. So if I zoom in and zoom in and zoom in, my talk started at 6:30, and, well, there's been some data fairly recently, let's take a look at that. So, yeah, occasionally the device slips off a finger and you get these dropouts, so that's what's going on.

I actually know, until I am not looking at the scale, every reading has been between 99, the maximum value it ever reports, and 97, all of which are perfectly fine. Let's show the heart rate on this as well. You can tell I get really quite excited when I'm delivering a talk. Gosh, that's high, even by my standards for a talk. I'm normally lower than that, but I get all geared up as talks go on.

And we can even look at the raw data. Now this looks like random noise until you zoom in on a little bit of it, so let me zoom in, and that is the boom, boom, the heart rate measurements actually being taken by the device. So this is all data that's been accumulated live while I've been talking, so this is a real demo. This is using exactly the same process that I would use when I'm doing a nighttime monitoring run, so this data has come out of the stream.

Data processing with Rx

So now you've seen where we're heading with this, let's go back and look at the code. So where we left it, we have a device which we've discovered, and we've then got its byte stream, and we've then plumbed that through this protocol decode method. I'm going to take the pulse oximeter off now, because it's quite hard to type with that on. And then I attach various subscriptions to it.

So the first thing I do is I say, I just want to know if any errors come out of this thing, or if my message stream completes. I actually don't expect that ever to happen. This is meant to run forever, because for as long as the device is producing messages I want to produce data, and I don't shut the stream off if the device switches off, because I want the thing to be able to pick up again if the thing falls off and I put it back on again. So it basically runs forever unless you deliberately unsubscribe.

Then, same message stream, as I subscribe to for this overall monitoring, I attach a second subscription. This one says, I am going to use the LINQ Where operator just to get the messages whose headers contain these bytes here, because I have determined by trial and error that those are the messages that report the blood oxygen saturation, the heart rate, and a thing called PI that I don't understand the clinical significance of, which is in there too. So this is just messages of those types.

There's a different bunch of headers for the photoplethysmography, so that's the actual raw readings the device is taking by shining a light into my finger and measuring what comes through it. So this is what the device is really taking. So the heart rate and O2, they're derived, calculated characteristics that the device is working out using an algorithm, but these are the actual readings, so that's what I saw when I zoomed in.

Interpolating timestamps with a sliding window

Now this is where it gets fun. You get quite a lot of these, and the data turns up in batches of five readings at a time, because five is a natural number for computing. I don't know why it's five, it just is. So you get five readings at a time, and some sixth number whose purpose I've yet to work out. And so the thing is, the first thing I'm going to do is work out what time each of those corresponds to.

So remember the device gives us no timing information, and I don't have any documented record of what the actual rate at which those things come out of it is. So what I'm doing is I am telling Rx that I would like it to deliver me messages from this stream of photoplethysmography messages in pairs. So the Buffer operator says I want to get more than one at a time. In this particular case I would like to get messages in pairs, but I want to advance through the sequence one at a time.

This basically gives me a sliding window. It gives me the first and the second message, then the second and the third, then the third and the fourth, and the fourth and the fifth. So I get pairs of messages, I'm going to see every message twice, and what I can then do is say, okay, what was the time between them? When did the first one arrive and when did the second one arrive? That's the time between messages.

And then I basically interpolate that to work out the nominal arrival time of each of the samples inside the message, to give me an actual annotated time series. So I end up with messages that say, okay, here's a bunch of values where for each value I have a time and, in this case, a reading. So I've got time series data now.

Batching messages for IoT Hub

So this is actually then going to spit out groups of batches of data, but it's coming in batches of five, and it turns out to produce about 20 readings a second, to produce that little blip blip bar graph thing. That's quite a lot of messages, and you get billed by the message with Azure IoT Hub, so you might want to batch these things up. There's a trade-off here to be made between how many messages you send in each lump and how quickly you can monitor things.

For demo purposes I've taken the decision that I want to send my high volume data in batches of 10. That's actually pretty small. That means I'm going to get two batches a second, so if I wanted to have a live readout on screen that was subscribing to the things coming out of the IoT Hub, I could do, and it would be, you know, half a second behind live, but it would be as close to live as... yeah, it would look quite reasonably quick. If you're just doing offline analysis, it would make sense to bump this up to a much higher number.

But again, notice I'm just using one of Rx's built-in operators. I'm saying, yeah, this thing's coming in in batches of this size, I would like to batch those into groups of ten. And actually, I've... sorry, I've mis-explained this. This is batching my batches, so actually I'm going to get groups of 10 groups of 20. This is actually sending 50 readings at a time, which actually makes a bit more sense, sorry, I forgot how that works.

So we then group those up into one great big batch, so this is yet another observable. So once again, everything's an observable. This is an observable sequence of great big chunks of data, big enough to be sent without massive expense to Azure IoT Hub. So then some stuff I commented out, let's ignore that, and also we're going to want to send that.

Sending the data to Azure IoT Hub

So next I create another subscription, this time to this output batch thing. I serialize each of those batches and I send them to Azure IoT Hub. So this is the point where I send them out to the cloud. This is why, when I showed you the web application earlier, it was able to show you data. This is why this particular orange bar graph is showing up, not bar graph, line graph.

And then the oxygen saturation and heart rates come in a separate sequence of messages. Remember I had a different observable for those ones, so I had the two observables filtered on different criteria in the headers. And this one says, okay, this is the analyzed data. This is a much lower rate, you get one of these a second, basically, and this is going to report blood oxygen, beats per minute. I don't know what's in byte number two. Byte number three is something that the phone app reports as PI, and I don't know what it means, but I report it anyway.

And again, I'm going to batch these up. Now here again I'm using Rx's Buffer operator. I like the Buffer operator, but here I've done something slightly different. I said, well, I don't have the manual for this device, I've reverse engineered the protocol, I don't actually know what rate it's going to send me the data at. So for this one I'm going to say, actually, let's just batch the data up into chunks of five seconds.

So every five seconds I'm going to send a new batch of data, and this will generate me a new slice every five seconds, relentlessly, whether there's any data coming out of the source observable or not. So I then had to filter out all the ones that turned out to be empty because the device has fallen off my finger. And then, again, very similar code to what you saw before: subscribe to that and serialize that and send that up to my device, sorry, my Azure IoT Hub. So that is the whole application.

Reviewing the application: hot and cold observables

So that was a lot of code, let's just pull out and just review that in picture form. So lots and lots and lots of observables. So we've started with device discovery, so I have an observable source that represents newly discovered, well, previously discovered and newly discovered Bluetooth devices. Actually, this is worth mentioning, I should have mentioned earlier: with Rx there's two kinds of observable source. There are what are called hot sources and cold sources.

A cold source is one that gives you everything every time you subscribe to it. So the Observable.Range was an example of that. I can create an observable range of one to ten, and every time I subscribe to it, it will generate all the numbers just for that subscription. And that's actually kind of the default behavior. If you write something using Observable.Create, it will work the same way unless you take steps to make it not work that way.

Whereas a hot observable is more like a radio broadcast, conceptually. It's a thing that's just going on, and if you tune into it you will receive it, and if no one tunes into it, the data just vanishes. Now actually my device discovery is kind of a hybrid. It sort of starts off in cold mode, but just because of how the operating system Bluetooth stack works. When I tell Linux I want to start discovering Bluetooth devices, it turns out Linux just says, oh, here are the ones I already know about, so it's kind of acting like a cold source for me.

So I just go, oh, okay, I'll just enumerate all of those. I actually can't tell the difference in my code between ones that it already knew about and ones that it's just discovered now, other than by looking at the timestamps and going, okay, you just told me about 20 in the space of 2 milliseconds, and then there was a wait of ten minutes, and then you told me about another one. I'm going to guess that last one was a genuine new discovery, but there's no real difference from my code's point of view. So it's sort of this weird hybrid of starting up in cold mode and then transitioning to hot mode.

So the thing is a long running observable source of objects representing the devices that may or may not be available to my application, because the devices may drift out of range again. So then I'm using the Where... actually, this is not strictly true, is it? I'm actually using the First operator to do two things. I'm using the First operator to select which device I want and to then unsubscribe from the underlying source for me.

This is slightly out of date. I used to have a separate Where and First, till I realized that the First operator takes a callback and you can just do it all in one. Anyway, sorry about that, so these two boxes should be one single box. And that's going to spit out an object. This device is a different shape because this is not an observable, it's a thing that came out of an observable, which, by the way, by convention are drawn as round things in Rx diagrams.

Publish, RefCount and fanning out

This device happens to provide a property which is another observable source, representing the byte stream received from the virtual Bluetooth serial port that the device presents all this data through. I then plug that through the Timestamp operator that's provided by the Rx libraries. I then... again this is slightly out of date, I was using something else to project bytes to messages. Actually this should say custom operator that projects from bytes to messages. So I've got my thing that did the protocol decoding, looked at the bytes that came in, and worked out how to turn that into a sequence of messages, which is what we have here.

And then I use that Publish RefCount thing. And as I mentioned earlier, the purpose of the Publish thing is, Publish gives you a means of getting a thing called a connectable observable, which is a thing that implements IObservable as a wrapper on top of another observable, which doesn't actually subscribe to the underlying observable until you tell it, connect now. So it kind of lets you set up a downstream chain of processing, and then to kind of have a Connect method you can call that says, go now, to start everything off when you're good and ready.

And then wrapping that in RefCount gives me a way of bolting multiple subscriptions onto this in a way that results in actually just one single subscription to the underlying observable under here. So this is a way of avoiding spinning up multiple receive loops for my Bluetooth code. That's what I'm using the RefCount thing for: it lets me have multiple subscribers to a single source, but narrowing it down to a single subscription upstream of that. That's what that's for.

And then I use various Where operators to filter out the different message types I'm interested in. And there are other messages you get: there's a battery level event you get from time to time, there's a signal strength event, there's various things that will tell you. I only care about two in my application, but other applications might want to report more. And then I projected these through further things that kind of retrieved the data they wanted, and then subscribed to those to deliver data to the cloud, which is why you're able to see it on screen. So that is an example application using IObservable in action.

Why IObservable was the right abstraction

And I said I was going to argue that IObservable was the right abstraction here. Could I have used IEnumerable for any of these things? Well, IEnumerable, I think, is obviously the wrong abstraction for the device discovery, because you can go for minutes or even hours, potentially days, between discovering new devices. The Raspberry Pi sat on my desk that I'm using for this demo has been running for about five days non-stop now, and probably hasn't seen any new Bluetooth devices in that time.

So an IEnumerable, I mean, it would work, you could make it work, but you'd be blocking a thread for literally days until you had something interesting to report, so that's the wrong abstraction. Could I have used IAsyncEnumerable? Um, yes, I could, but again I don't think a foreach loop is really the best way to do this, because fundamentally this is sort of a service in the operating system, the Bluetooth discovery service, that sends out notifications from time to time, and so the push based mechanism is just a much more natural way of doing that.

It's kind of, you've got this thing sending out notifications. IObservable is a better fit for that than something that you foreach over. foreaching over an event just feels weird to me. And the same goes for the data coming in from the device at every stage. Well, the device is an autonomous actor that generates information according to its own schedule, so again IEnumerable would be entirely the wrong abstraction for this, really.

And it would be really quite hard to have implemented this in that way, because of the fan out thing. You know, I've ended up with multiple different things that subscribe to the same logical sequence of messages and do different things with it. It's not entirely obvious how you do that in a pull based approach. How do I arrange for the same live sequence of things being exposed through an IAsyncEnumerable to then be consumed by multiple different concurrently operating foreach loops?

I mean, you could do it, you know, there's no doubt you could certainly write it that way if you're prepared to do the work. I just don't think it's a natural way to represent it. It adds a bunch of complication that really doesn't help you. Whereas when I was writing this app, I felt that the observable based model was actually quite a good fit for what I wanted to do, because information flows through the system.

I want to filter it and annotate it with timestamps and massage it and process it in various ways as it flows through, and at every stage I want the next piece down to then react to what the previous stage has done, all the way down to pushing the data out to Azure IoT Hub, to make the data available to my Time Series Insights rendering. So I think observable was the right tool for that job, but I'll take questions if anyone thinks it wasn't.

Server-side Rx and Reaqtor

One last thing I quickly want to talk about before I go on. So I've just shown you kind of device based use of Rx, so this is like in the edge, if you like, which is a great application for it, because often you want to do a lot of filtering on the client side to make sure that what you actually send over the airwaves is efficient. So at endjin we work with utility companies quite a lot, and some of them have equipment that does have some sort of radio connection, but when they have equipment deployed in the field, it's often literally an actual field, and, yeah, the data rate's probably some 2G GSM type connection, where you really don't have much bandwidth.

So the ability to push logic out to the device is actually quite a useful thing, so definitely an important use case. But I wanted to talk quickly about the original use case, or one of the original use cases, for which Rx was invented, which is fully server side Rx. So Microsoft uses this. They've talked about this in public. They use it extensively in their Cortana digital assistant stuff, which is both exposed sort of as public branded stuff, but they also have other ways of using it in other technologies as well, so it's quite a widely used thing.

And this, for example, lets you do things like say, I want to find out when I should drive to the airport to catch this flight. And they do that by having long running observables in the cloud. They have an observable that represents all the traffic data, and then they have... it's not literally quite implemented like that, but logically that's what it is. You can subscribe to traffic information, then you can slap a where clause on that that says, I'd like to narrow it down to just traffic information within this locale.

And then you can also subscribe to flight status updates, and then those can get combined into another observable that's then working out at what point it should send you a notification to say, leave now. So it's all done with long running observables in the cloud. Now it's not the same implementation I've just shown you. They're not using System.Reactive, they're using a thing that the same team of people developed. At one point they were talking on Channel 9 about maybe making this the next version of Rx, but it's kind of diverged.

Now that Rx is its own open source project, this server-side implementation of it has sort of taken on its own life, and it's currently still a closed source thing. So it's not the next version of Rx, it's not technically Rx itself, it's more a thing that looks and feels exactly like Rx, but is a different bunch of code, because they support things like the ability to distribute computations across multiple servers in a cloud. They have the ability to take expressions and send those across the wire.

So you can write a client application that does a LINQ query, and that LINQ query can be turned into an expression tree that gets serialized and sent to a server farm, which may then tear it apart, turn it into a different actual physical implementation of the query, send that to potentially several different servers in a cluster, and then run those as long running queries. So that's the thing you can't do with the current shipping version of Rx.

Having said that, endjin has been lobbying Microsoft for some time now to get them to open source this, and we think we're in with a shot of that. So if anyone's interested in using this technology, which they have described as being... Reaqtor is the name this has been discussed under publicly. If anyone's interested in using Reaqtor, please let me know, because I would like to bolster the case for open sourcing it. The more people say they want to use it, the more likely that is to happen.

Anyway, so it's a concept that is supported, it's certainly possible to do. And even if you don't have access to, you know, that source code, which obviously no one outside Microsoft can use right now, there's nothing stopping you using some of the same principles to do kind of long computation on the server side. So it's definitely a possible thing to do. Obviously I don't have a demo I can show you of that, because Microsoft hasn't open sourced it yet, but maybe next year, who knows.

Conclusion

So, in conclusion, IObservable<T> represents one thing after another, and specifically it's a push model. It is designed to represent where the one thing after another is being produced by some potentially autonomous process on its own schedule, and the natural way to consume it is to be told when the next item is, so some sort of callback based model. And so it's quite a good fit for asynchronous sequences, although they don't have to be. As you saw in my very first demo, I just used a range of numbers, the simplest observable there is. That's actually synchronous, that just delivers the entire stream of numbers immediately and then shuts down again, but it is a good fit for async stuff.

So that's the model, and then we have the System.Reactive library, which gives us all of the actual implementations of those operators, a model for scheduling, and the ability to kind of bridge into and out of the world of Rx. And I think this is one of the fundamental abstractions in .NET, because sequences of things that happen, they're actually quite a common thing. So I've used this a lot over the last 10 years it has been available, and I would strongly recommend it if that's the sort of application you are writing.

So it remains for me to say thank you very much for listening, those of you who still are. Thank you very much for hosting me at your user group, and at this point I'm going to hand back to the host, and if anyone has any questions, feel free to enter those in the chat, or to ask on YouTube, whatever, and otherwise, thanks for listening.

About the presenter

Ian Griffiths

Technical Fellow I

Ian Griffiths

Ian has worked across an extraordinary breadth of computing - from embedded real-time systems and broadcast television to medical imaging and cloud-scale architectures. As Technical Fellow at endjin, he brings this deep cross-domain experience to bear on the hardest technical problems.

A 17-time Microsoft MVP in Developer Technologies, Ian is the author of O'Reilly's Programming C# 12.0 and one of the foremost authorities on the C# language and high-performance .NET development. He's a maintainer of Reactive Extensions for .NET, Reaqtor, and endjin's 50+ open source projects.

Ian has created Pluralsight courses on WPF fundamentals, WPF advanced topics, WPF v4, and the TPL, and has given over 20 talks at conferences worldwide. Technology brings him joy.