What's up with Half.MaxValue in .NET?
.NET Deep Dives
In this episode of .NET Deep Dives, Ian Griffiths explores the peculiar behaviour of the half-precision floating point type.
By using the simplicity of a C# program that prints an incorrect maximum value for this type, the video explains floating point representation intricacies, especially with 16-bit values. It delves into the concept of precision and range, illustrating the vast differences between single, double, and half-precision floats. The discussion clarifies .NET's formatting behaviour, which rounds half-precision values to avoid spurious precision in display, leading to seemingly wrong output.
Endjin are proud to be a .NET Foundation Corporate Sponsor, as we are maintainers of Reactive Extensions for .NET (AKA ReactiveX AKA Rx.NET) which is one of the most well established and widely used open source .NET projects.
Transcript
.NET recently led me down a rabbit hole. If you run this very simple one line C# program, it prints out a value that is quite simply wrong. It says 65,500, but that's not right. The maximum value that this type, called half, can represent is 65,504. What's going on? How can such a simple program print out the wrong value?
So this is not a bug. It's meant to work this way, and for good reasons. I'm going to explain why, because what I found interesting is that this makes plain some issues with floating point numbers that are usually much better hidden, but which are always lurking.
For most of my life as a developer, floating point numbers have come in two sizes. There were 32 bit floats, Also known as Single Precision, or you could use Double Precision values, which took up twice as much space, but had significantly better range and precision. But then, back in 2020, .NET 5.0 added a new floating point type called Half. As you might guess, this takes up half as much space as a Single Precision value.
These are 16 bit values. One of the defining features of Single and Double Precision floats is that they have enormous range. Even a single precision float can represent values as high as 340 undecillion, a number so large that I had to go and look its name up in Wikipedia. It's more or less impossible for the human brain to comprehend numbers at these scales.
Wikipedia says that the observable universe is about 93 billion light years wide. To get a rough idea of the range of scale of single precision floating point, consider the relative size of one atom to the size of the observable universe. That's actually not quite as big a difference as between the largest single precision floating point number and the number one, but it's close. You need to go down to about one thirtieth of the size of an atom. That's the difference in scale between the number one and float.MaxValue. But that undersells it, because floating point values can also get incredibly tiny. The difference between the number one and the smallest single precision floating point value is even bigger.
It's a factor of roughly a quattuordecillion. So going from the size of the observable universe to one thirtieth of an atom didn't even get us half the way. A float can shrink down by the same factor again, and then to a billionth of that. From the largest to the smallest value, single precision floating point spans 78 decimal orders of magnitude. And the observable universe just isn't big enough to put that range of scale into context. And that's just single precision. Double precision has a range of 632 decimal orders of magnitude. In short, these floating point formats have incomprehensible range. So it comes as something of a surprise that half precision floating point values top out at 65,504.
That's not the number of orders of magnitude, that's just the number. The largest value that half can represent is quite a lot lower than the number of seconds in a day. And on the small end, it can go all the way down to 1 divided by about 17 million. Not only is the range unimpressive, half is not very precise either. The maximum value is 65,504, but the next lowest value it's able to represent is 65,472. Want to represent some value in between, like 65,492? You can't. Now, this is normal for floating point types. They can only represent certain values, so any value that you'd like to represent that happens not to align with one of those gets sort of funnelled down to the nearest available value.
With single and double precision numbers, this usually only becomes obvious at relatively large or small scales, but half really puts this in your face. It's roughly like having three digits of decimal precision. So the half type is a blunt instrument. It almost looks like a joke. So why does it exist? The obvious advantage is in the name. It only takes half as much space as the next size up. If you're dealing with billions of values, that can matter. And the less obvious advantage is that it's possible to create hardware that can perform half precision calculations more quickly than would be possible with more precise formats.
So, half precision floating point values are useful in applications that don't need particularly large or small values, and that can tolerate limited precision. And that may need to perform calculations across very large sets of values. This is exactly what a lot of GPU applications are like. This limited range and precision is often good enough for graphics, and it also turns out to work well enough for the kinds of artificial intelligence applications that often get run on GPUs.
That's why this data type has become popular. Many graphics cards have direct hardware support for it. They have a limited amount of memory, So being able to fit twice as many values can be really useful, particularly with AI models, which often have a lot of parameters. So that's why Half was introduced to .NET.
But what about the title of this talk? Why am I saying something is odd with Half.MaxValue? Well, look at this program. It just displays Half.MaxValue to the console, and the value it prints out is 65,500. That's not the value I said earlier. I said it was 65,504. And Wikipedia agrees with me. What's even more strange is if I just convert the value to a single precision floating point first and print that out, that does show the correct value of 65,504.
So what's going on? Well, first of all, rather than just taking Wikipedia's word for it, let's work out what the maximum value really should be. The half data type looks like this. One bit determines whether the number is positive or negative. For max value, that's zero, indicating a positive number. Next is the exponent, which is basically a number indicating the scale.
Is this a big or a small number? There are 5 bits for the exponents, but the highest possible value, all 1s, is reserved for special values like infinity and not a number. So for max value, this actually ends up being all 1s except for the final digit. In binary, that's 11110, which, in decimal, is 30. The remaining 10 bits represent a fraction.
You take the value of this 10 bit number and divide it by 1024. For half dot max value this will be all ones, which is a value of 1023, so the fraction is 1023, 1024ths. To work out the scale of the whole thing, we add 1 to this fraction, and then multiply that by the scale. The scale is calculated by subtracting 15 from the exponent, and then raising 2 to the power of the result.
Why? Because that's how the IEEE 754 specification says you do it. So, let's work that out. The exponent is 30, but we subtract 15, which gives us 15. So we raise 2 to the power of 15, which is 32,768. That's our scale. And we already worked out the fraction. So, we multiply 1 plus the fraction by the scale. So it's 1 + 1023/1024 x 32,768 and if you perform that calculation, the answer is indeed 65,504. That is definitely the right answer. So, why does writing half max value to the console display a different value, 65, 500? The answer is that when you ask .NET to display a floating point type as decimal text, then unless you tell it otherwise, it tries to avoid displaying a result with spurious precision.
Remember, earlier I showed how floating point formats can represent only certain values. If we pick any number on the number line that's within the range that half can represent, the chances are that that number won't be one that half can represent precisely. Whole ranges of real numbers end up with the same representation.
If the result of a calculation was anywhere between 65,456 and 65,488, the closest value half can represent is going to be 65,472. So, if you've got a half value of 65,472, you can't be certain of what the precise result of the calculation was. It might have been that number, but it could equally have been any number within that range.
So although we've got a half whose value is definitely 65,472, we know that it could be off by as much as 16 in either direction. Normally, if we have only an approximate idea of what some value is, we round it up or down. If I say that there are 280,000 people living in my city, you'll understand that this figure is, at best, accurate only to the nearest 10,000. But if I told you that in 2021 there were 277,200 people living in my city, you could reasonably assume that I believe the figure was accurate to the nearest 100. The number of significant digits implies the level of precision on offer. Now .NET knows that the half data type can't represent any value between 65,504 and 65,472.
If we have performed a calculation for which the correct answer is somewhere in the upper half of that range, such as 65,495, the closest half can get to the result is 65,504. To display the digits, 65,504 spuriously suggests that we know the answer to the nearest whole number. This would fail to convey the fact that the precise result may well have been 65,502, or 65,498.
So to make it clear that we just don't have that kind of precision, .NET displays the final two digits as zero. This rounding only happens when converting the numeric value to a decimal string. If you first convert the value to some higher precision format, it does get the correct value. And then when we try to display that value, the text formatting sees that we have a high precision format here, so it sees no reason to round to a smaller number of significant digits.
We can also override the default rounding with a format specifier. And this code also prints out the exact value. So what have we learned? Half precision floating point values aren't fundamentally different from their higher precision cousins. But because they have such a small range and such limited precision, the anomalies that afflict all floating point types can be much more obvious. And when we add in the .NET framework's default behaviour of conveying limited precision by rounding, we end up seeing the "wrong value" for Half.MaxValue.