Ix.NET v7.0: .NET 10 and LINQ for IAsyncEnumerable<T>

We've just released a new version of the Interactive Extensions for .NET (Ix.NET). Now that .NET 10.0 offers built-in support for LINQ to IAsyncEnumerable<T>, it's time for Ix.NET's System.Linq.Async to step back.

Why you might need to upgrade

If you've been seeing errors of this kind since .NET 10 shipped:

error CS0121: The call is ambiguous between the following methods or properties: 'System.Linq.AsyncEnumerable.Select<TSource, TResult>(System.Collections.Generic.IAsyncEnumerable<TSource>, System.Func<TSource, TResult>)' and 'System.Linq.AsyncEnumerable.Select<TSource, TResult>(System.Collections.Generic.IAsyncEnumerable<TSource>, System.Func<TSource, TResult>)'

you may need to upgrade to v7 of Ix.NET's System.Linq.Async package. (In the long run you will want to stop using it entirely, and use the .NET runtime library System.Linq.AsyncEnumerable package instead, but if you've ended up seeing these errors because of an indirect dependency, you might not be able to remove the reference just yet, in which case you'll need to upgrade it instead.)

In most cases, that will solve the problem. It's possible you'll also need to add a reference to System.Interactive.Async v7 (or upgrade an existing reference to that version). There are also some more complex scenarios. This post explains what has changed and why.

.NET 10.0 and LINQ to IAsyncEnumerable<T>

The main reason for this new Ix.NET release is that .NET 10.0 now implements a feature that used to be part of Ix.NET: LINQ for IAsyncEnumerable<T>.

For years, if you wanted to use the standard LINQ operators with IAsyncEnumerable<T>, you had to use the System.Linq.Async library. Despite its name, this was not part of the .NET runtime class libraries, and was not maintained by Microsoft. That library was originally produced by the Rx.NET team as part of the Ix.NET libraries. This occurred after Microsoft stopped work on Rx.NET, so System.Linq.Async was always a community-maintained library. (See the History section for the reasons behind this.) But .NET 10.0 now supplies the functionality that System.Linq.Async was originally written to provide.

The .NET team did consider just taking over ownership of System.Linq.Async, but decided instead to reimplement LINQ for IAsyncEnumerable<T> from scratch. This was partly motivated by the fact that the old System.Linq.Async package predates some library design guidelines, and made some naming choices that do not align with current practice.

This new implementation lives in an assembly called System.Linq.AsyncEnumerable, which is built into .NET 10. It is also available for use on older verions of .NET (including .NET Framework) through the System.Linq.AsyncEnumerable NuGet package. This provides a complete implementation of LINQ for IAsyncEnumerable<T>.

What this means for developers

Anyone writing new code that targets .NET 10 can use the standard LINQ operators on any IAsyncEnumerable<T> without needing to add any NuGet packages. But where it gets a little more tricky is if either:

• you were already using the old System.Linq.Async and have upgraded to .NET 10.0
• you end up with a transitive dependency on the old System.Linq.Async

That second one will be quite common because System.Linq.Async is a widely used package. You might be using it without ever having asked for it.

(Note that it's also possible to hit problems without even upgrading to .NET 10.0. The problems that occur in these two scenarios aren't really to do with being on the .NET 10.0 runtime: they happen because System.Linq.Async v6 clashes with System.Linq.AsyncEnumerable. .NET 10.0 includes System.Linq.AsyncEnumerable 'in the box' so an upgrade to .NET 10.0 is likely to be the most common reason for encountering this clash. But it can also happen on .NET Framework or .NET 8.0 or .NET 9.0, because you can use System.Linq.AsyncEnumerable on those runtimes. The new System.Linq.AsyncEnumerable is built into .NET 10.0, but it's available for use via NuGet on those older platforms.)

The problem here is that you can end up with two implementations of LINQ for IAsyncEnumerable<T>: the old System.Linq.Async (Ix.NET) and the new System.Linq.AsyncEnumerable (.NET runtime libraries). When two implementations of every standard LINQ operator are available for IAsyncEnumerable<T>, the compiler emits error CS0121: The call is ambiguous messages any time you try to use them.

In most cases the fix is simple: upgrade to System.Linq.Async v7. (If you were using it directly, this just means upgrading your existing package reference to the latest version. If you have ended up with an indirect reference through some other package you'll need to add a new reference to the latest version of System.Linq.Async.) In some cases, for reasons described later, you might need to add a reference (or upgrade an existing reference) to System.Interactive.Async v7.

System.Linq.Async will be deprecated

V7 of System.Linq.Async provides a quick fix for the compilation errors that developers may encounter in .NET 10, but the longer term solution is for everyone to stop using System.Linq.Async. Its only purpose was to provide LINQ for IAsyncEnumerable<T>, and now that the .NET runtime libraries supply this through System.Linq.AsyncEnumerable, there is no longer any reason for Ix's System.Linq.Async to exist.

So Ix's System.Linq.Async is now a legacy component that exists purely for backwards compatibility reasons. If you're writing an application that has ended up depending on System.Linq.Async you might not be able to get rid of that dependency—you'll have to wait until the authors of the libraries that depend on it stop using it. But if you have only a direct dependency on System.Linq.Async, you should stop using it, and should switch to System.Linq.AsyncEnumerable instead. (You may also need to add a reference to System.Interactive.Async for reasons described later.)

We will be deprecating the System.Linq.Async package to encourage people to move off it.

However, if you do this, you may discover that the .NET runtime's System.Linq.AsyncEnumerable is not an exact replacement. There are two issues:

• Some methods have been renamed because naming conventions changed since Ix.NET first provided IAsyncEnumerable<T> LINQ support
• The new System.Linq.AsyncEnumerable has omitted some of the functionality that Ix supplied

An example of the first kind of issue occurs with operators that take callbacks, such as Where. When filtering an IAsyncEnumerable<T>, you might want to use a normal Func<T, bool> just like you would with IEnumerable<T>, but since you're in an asynchronous world, you might actually want to provide an async callback, and you might want that to support cancellation. To support that, Ix's System.Linq.Async offered not only Where, but also WhereAwait and WhereAwaitWithCancellation.

This same functionality exists in .NET's System.Linq.AsyncEnumerable, but there are two important changes:

• the async callback overloads have the same name as the normal ones (e.g., instead of WhereAwait and WhereAwaitWithCancellation, we now have just overloads of Where)
• the async callback overloads require the callback to accept a CancellationToken (which the callback is free to ignore)

If you're using these async callback operator forms today and you upgrade to v7 of System.Linq.Async, you will see warnings of this kind:

warning CS0618: 'AsyncEnumerable.WhereAwait<TSource>(IAsyncEnumerable<TSource>, Func<TSource, ValueTask<bool>>)' is obsolete: 'Use Where. IAsyncEnumerable LINQ is now in System.Linq.AsyncEnumerable, and the WhereAwait functionality now exists as overloads of Where. You will need to modify your callback to take an additional CancellationToken argument.'

We continue to provide these old methods, but we provide a deprecation warning to encourage you to move onto the new equivalents provided by the .NET runtime libraries.

Note: if you're using any of the old methods with Await in their name, you will need to do more than just using the new method name, because the new .NET runtime library implementations require your callback to take a CancellationToken. If you don't add this extra parameter to your callbacks, you will get errors of this form:

error CS4010: Cannot convert async lambda expression to delegate type 'Func<int, bool>'. An async lambda expression may return void, Task or Task<T>, none of which are convertible to 'Func<int, bool>'.

This is not a very helpful message because it doesn't explain what you need to do. (Note that the deprecation method does tell you what you need to do, but if you didn't read that all the way to the end, you will have missed the part that saves you from this error.) If you had this:

IAsyncEnumerable<int> evens = GenerateNumbersAsync(10)
    .WhereAwait(async x => x % 2 == 0);

and after reading about two thirds of the deprecation warning you changed it to this:

IAsyncEnumerable<int> evens = GenerateNumbersAsync(10)
    .Where(async x => x % 2 == 0); // CS4010 error on this line

you'll get that error above. You need to modify the lambda to accept an additional argument:

IAsyncEnumerable<int> evens = GenerateNumbersAsync(10)
    .Where(async (x, _) => x % 2 == 0);

The change here is that instead of the single x parameter, we now have a parameter list: (x, _). That underscore indicates that we don't actually want to use our second argument. (It's a discard.) That's OK, but we still have to accept the argument, because the new System.Linq.AsyncEnumerable (.NET runtime) implementation of Where does not support the single-argument callbacks that System.Linq.Async (Ix) did. This is not a change in fundamental capability: you're not obliged to do anything with that cancellation token. But it does mean you need to change more than just the name of the method you're calling.

There's a further trap for Select. If you had code like this:

xs.Select(async v => ...)

it's not enough to do this:

xs.Select(async (v, _) => ...)

because the compiler can't tell whether you mean the overload that accepts a Func<TElement, CancellationToken, ValueTask<Result>>, or the overload that takes a callback which receives an extra parameter indicating the index of the value, which is of the form Func<TElement, int, TResult>. (The basic issue here is that in its standard form, Select can accept either 1- or 2-argument projection callbacks. This means the additional cancellable forms can create ambiguity.) So you need to make it clear which overload you mean by specifying the argument types. For example if xs is an IAsyncEnumerable<int> you can write this:

xs.Select(async (int v, CancellationToken _) => ...)

Note that this ambiguity has nothing to do with the transition from Ix's System.Linq.Async to .NET's System.Linq.AsyncEnumerable. You can run into exactly this error if you create a brand new .NET 10 project and did not use Ix at all. (I don't know the history, but it's possible that the Rx team chose to use the unusual Await and WithCancellation naming conventions to avoid exactly this kind of ambiguity.)

Now you might have done all of this, and still find that if you attempt to remove your reference to the old System.Linq.Async NuGet package, your code no longer compiles. In which case, read on...

Relocated functionality

The new System.Linq.AsyncEnumerable (.NET runtime) library does not provide all of the functionality that System.Linq.Async (Ix) library did. This table describes the relevant extension methods for IAsyncEnumerable<T>:

Since our goal is for people to stop using System.Linq.Async, we can't just leave these methods in there. So we have moved them into System.Interactive.Async.

System.Linq.Async v7 includes a transitive reference to System.Interactive.Async, so it should 'just work'. But we are deprecating System.Linq.Async, and when a project stops using that, it might be necessary to add in a reference to System.Interactive.Async so that these non-standard methods or overloads remain available.

Note that System.Linq.Async also defined a ToEnumerable method that adapts any IAsyncEnumerable<T> to IEnumerable<T>. This is a 'sync over async' operation, and those are usually a bad idea, and we believe it was a mistake for System.Linq.Async ever to have offered this. We have elected not to provide a new version of that. If you need this, we suggest you rethink your design. And if after that you really think you still need it, well, Ix.NET is open source, so you can always find the original implementation, but our view is that you will be better off not using it.

There are also some extension methods from other types to IAsyncEnumerable<T> that System.Linq.AsyncEnumerable did not duplicate:

Again, these are now available in System.Interactive.Async.

There is one non-extension static method formerly defined by AsyncEnumerable:

We've moved this into System.Interactive.Async, and unfortunately this is a case where we've had to make a breaking change where we can't help developers out with an Obsolete message. It is necessary for the System.Linq.Async package's reference assemblies not to define a public AsyncEnumerable type, because if they did, it would cause compiler errors in code that attempted to use static members directly. For example, if you wrote AsyncEnumerable.Range(1, 10), then although this method is now available in .NET 10, if System.Linq.Async defined its own AsyncEnumerable it would cause this error:

error CS0433: The type 'AsyncEnumerable' exists in both 'System.Linq.Async, Version=7.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263' and 'System.Linq.AsyncEnumerable, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

If all the methods on AsyncEnumerable were extension methods this would matter less. (It would still be a potential problem because you are allowed to invoke extension methods using normal static method syntax.) C# doesn't care if two types with identical names both define extension methods: as long as the individual methods don't clash, there's no ambiguity problem. But if you ever refer to the defining class by name (i.e., you refer to the AsyncEnumerable class) then the existence of two definitions becomes a problem and you get that error.

So it is not possible for System.Linq.Async's public API to include an AsyncEnumerable type. For the most part this isn't a problem: for extension methods we can move them to a different type. (We call this AsyncEnumerableDeprecated, because the only reason System.Linq.Async's public API retains any of the methods that it used to define on AsyncEnumerable is to be able to provide [Obsolete] attributes telling you what to use instead, and for extension methods, those are equally effective even if we change the defining class name.) And most of the non-extension static methods we used to define are now available on the new System.Linq.AsyncEnumerable library's AsyncEnumerable.

But this one method, Create is an unfortunate exception. There's really nothing we can do other than remove if from the public face of System.Linq.Async, and define its replacement in AsyncEnumerableEx in System.Interactive.Async. We can retain binary compatibility because the runtime assemblies in System.Linq.Async continue to define AsyncEnumerable exactly as before. But code that was calling AsyncEnumerable.Create before will now just get an error reporting that this method does not exist, and the developer will have to guess that they now need to use AsyncEnumerableEx.Create. Our hope is that because this method hasn't been very useful since C# added support for IAsyncEnumerable<T> iterator methods (yield return etc.) that not many people will be using it. The built in language support does the same thing only better.

The methods described so far in this section are ones that the .NET Runtime team did not consider to be 'standard' operators. Interestingly, there are some that Ix.NET didn't consider to be 'standard' that the .NET Runtime team did. The following methods were defined in System.Interactive.Async v6 because at the time they didn't align with standard operators (or at least, standard overloads) available on other LINQ implementations:

The MaxByAsync and MinByAsync members are problematic because back when Ix.NET introduced these (and their non-async counterparts, MaxBy and MinBy, in System.Interactive), not only were they non-standard operators, they worked differently from the MaxBy and MinBy eventually introduced in .NET 6.0. Back in Ix.NET 6.0, System.Interactive had to hide the IEnumerable<T> versions of these methods and introduce new MaxByWithTies and MinByWithTies operators to continue to make the functionality available. Unfortunately, System.Interactive.Async was not updated in the same way at that time, which is a pity because the old methods could have been retained at that time with Obsolete attributes, giving people time to move onto the new names. Since that didn't happen, we have no choice but to get out of the way of the new MinByAsync and MaxByAsync that .NET supplies without warning. Anyone with code that was using the existing methods of these names in System.Interactive.Async is going to get confusing error messages when they upgrade to .NET 10.0, and we can't easily supply guidance. (In theory we could write an analyzer to detect this, but it's complex, and remember we have no budget at all for this work. It's all eating into time we'd rather be spending working on Rx.NET.)

So if you have code like this:

IList<Person> oldest = await people.MaxByAsync(p => p.Age);
IList<Person> youngest = await people.MinByAsync(p => p.Age);

that compiles on .NET 8.0, you'll find that it fails to compile on .NET 10.0 (or if you stay on .NET 8.0 but add a reference to System.Linq.AsyncEnumerable):

error CS0029: Cannot implicitly convert type 'Person' to 'System.Collections.Generic.IList<Person>'

Note that if you prefer to use var, you won't get an error with this code when upgrading:

var oldest = await people.MaxByAsync(p => p.Age);
var youngest = await people.MinByAsync(p => p.Age);

but the library upgrade will change the type of these two variables. So instead of the compiler error occurring on the line where things went wrong, you'll likely get an error later on in the code when you attempt to use the relevant variable. (Or worse, you won't get a compiler error, but the meaning of your code will subtly change without you intending it to. But since this changes the variables' types from IList<Person> to Person, you will most likely get an error later on in the code.)

You'll need to change it to the following:

IList<Person> oldest = await people.MaxByWithTiesAsync(p => p.Age);
IList<Person> youngest = await people.MinByWithTiesAsync(p => p.Age);

(The reason Ix.NET's returns a list here is that there might be more than one item that has the highest value. The .NET runtime's MinBy and MaxBy pick one arbitrary winner.)

Target frameworks

The System.Linq.Async, and System.Interactive.Async packages now have an additional net10.0 target framework. They don't technically need it, because the changes made necessary by .NET 10's addition of System.Linq.AsyncEnumerable are also required on older targets because the new System.Linq.AsyncEnumerable package can also be used on those older targets.

So this is purely a cosmetic move, intended to signal that these packages are .NET 10-aware.

Since v7 of Ix.NET is entirely about ensuring we work well with the new System.Linq.AsyncEnumerable package, nothing else has changed, and so the System.Interactive, and System.Interactive.Providers packages support the same TFMs as before, with nothing later than net6.0. They still work perfectly well on .NET 10.0.

We will update the TFMs so that no package has a .NET TFM lower than net8.0 in a future release. (Most likely we will produce a v8 fairly soon that makes explicit that we no longer support .NET 6 or 7.) We just didn't want to conflate that with the fixes required to coexist with the new System.Linq.AsyncEnumerable.

Method hiding: technical details

I've mentioned a few times that we have hidden some methods in System.Linq.Async, but I've not explained what that means.

We can't just remove methods from System.Linq.Async. If we did that we would break binary compatibility. Suppose you're using a library called UsesOldIx that was built against System.Linq.Async v6, and your application upgrades to v7 of System.Linq.Async. And suppose that the UsesOldIx library has not been updated. Your application will now be supplying UseOldIx with v7 of System.Linq.Async even though UseOldIx was built against v6. So UseOldIx doesn't know anything about the new System.Linq.AsyncEnumerable—it will expect all these LINQ to IAsyncEnumerable<T> methods still to reside in System.Linq.Async.

If we just removed methods completely, scenarios like this would cause the application to crash with a MissingMethodException. So we use a trick: we hide the method at build time, but continue to make it available at runtime.

We do this by supplying separate runtime and reference assemblies in the NuGet package. If you look inside the System.Linq.Async package you'll find that as well as the usual lib folder, there's also a ref folder. When both are present, the .NET build tools tell the compiler to look at the assemblies in the ref folder. This enables us to remove methods from the public API at build time—we omit them from the reference assembly—but to leave them present in the runtime assemblies (in the lib folder) so any code that was already built against an older version will still be able to access the hidden methods.

System.Interactive was already using this in v6 to deal with the Min/MaxBy issues. As described above it effectively had to rename MinBy to MinByWithTies and MaxBy to MaxByWithTies because .NET 6 had added new methods with these names that did different things. So these had to be hidden but not entirely removed. So this trick is not new—Ix.NET has already been using it for a while, and now we're using it in System.Linq.Async too.

History

You might be wondering why the LINQ implementation for IAsyncEnumerable<T> was a community supported project in the first place.

In fact the code that ultimately ended up in the Ix.NET System.Linq.Async package did originate from Microsoft. More specifically, it was an invention of the Rx.NET team, back when that was part of Microsoft. So the reason System.Linq.Async is a community supported project is that Rx.NET itself became a community-supported project.

(For a highly detailed account of this history and related events, you can request a copy of the 'A Little History of Reaqtor' ebook from https://reaqtive.net/)

So the real question is: why did the Rx.NET team end up implementing LINQ to IAsyncEnumerable<T>? And the answer is that IAsyncEnumerable<T> itself was also originally invented by the Rx.NET team.

Rx.NET team emerged from a group at Microsoft that formed back when cloud computing was just getting established, who were asked to investigate what the cloud would mean for ordinary developers, and how software development might need to change to be able to take advantage of cloud-native architectures. One of the main researchers in this group was Erik Mejier. He had also been a significant contributor to the design of LINQ. He had always envisaged LINQ as being more general than merely enabling database integration in C# and VB.NET, and Rx.NET was one realisation of his vision. When async/await were being developed, it would have been a natural step for him to consider what that might mean for LINQ, and so it is perhaps unsurprising that his Rx.NET team produced the original IAsyncEnumerable<T> interface definition.

If you download System.Interactive.Async v3.0.0, unzip the nupkg, and open up System.Interactive.Async.dll in ILDASM, you'll see that this library defines IAsyncEnumerable<T> and IAsyncEnumerator<T>. In fact, the Rx.NET team first published definitions of these interfaces back before NuGet existed, in 2010!

So IAsyncEnumerable<T> had been around for a decade by the time it was made an integral part of .NET, with the release of .NET Core 3.0. Rx.NET has always been very closely associated with LINQ, and so it also had a LINQ implementation for that whole time.

I don't know the history of why the .NET team chose not to provide LINQ for IAsyncEnumerable<T> back then. It's possible that the fact that there was no longer an Rx.NET team within Microsoft to advocate for this didn't help. Perhaps LINQ was out of favour within Microsoft at that time, and they underestimated the demand for this feature. And the fact that the (now open source) Rx.NET project already had an implementation (and the project's supporters, including some still working at Microsoft at the time, leapt into action to update that implementation to align with the fact that IAsyncEnumerable<T> had now moved into the runtime libraries) enabled them to believe that there was no need for the .NET runtime class library team to fill the gap.

There certainly was demand for this feature. The Rx.NET team packaged it in a few different ways over the years, but if we look just at the System.Linq.Async library the Rx.NET maintainers produced back in 2019, that has had a quarter of a billion downloads! And it is such an obviously useful feature that many people just assumed that the library was actually part of .NET. The package name certainly contributes to that perception—historically Rx.NET has always used System prefixes for its namespaces, because the Rx.NET team's vision was that Rx should be built right into .NET. (IObservable<T> did indeed make it into the class libraries in .NET Framework 4.0. And the versions of .NET that shipped in Windows Phone did actually include full Rx.NET libraries.)

But Microsoft has supplied no funding for the Rx.NET project for well over a decade. This made it impossible to keep up with people's expectations for what a high quality implementation of LINQ for IAsyncEnumerable<T> should be. When endjin took over maintenance of Rx.NET back in 2023, the company's owners very generously decided to pay for the time I spend working on it, but our motivation for this was that we believe in Rx.NET and want to keep it thriving. We ended up become responsible for the other projects in https://github.com/dotnet/reactive as a side effect, and we never wanted to be the guardians of LINQ to IAsyncEnumerable<T>.

Fortunately, Microsoft had by this time recognized that .NET developers expect LINQ for IAsyncEnumerable<T> to be available, and fully supported, and they offered to build it into the .NET runtime class libraries. And so, today, .NET 10.0 provides built-in support. We're happy that this is available to the .NET world, and that we can now focus our efforts on Rx.

Please try it out

This new 7.0 release of System.Linq.Async (and the corresponding System.Interactive.Async) is available on NuGet today. If you're using Linq to IAsyncEnumerable<T>, please try upgrading (even if you're not yet on .NET 10). If you have any problems, please file issues at https://github.com/dotnet/reactive/issues. Meanwhile, we hope you enjoy this new version of the Interactive Extensions for .NET.

More Rx content

See my recent .NET conf talk about Rx.NET for more information on this work, and our other Rx.NET activities.