To null or not to null, that is the question …

[DaveSkender]

I’m considering changing IQuote and Quote in the library to have nullable OHLCV property values.

Here’s my logic:

• currently, missing quote property values are automatically converted to 0 values, since it is the default for non-nullable decimal
• 0 values will throw off indicator calculations, producing real results that are inaccurate or bad data
• you could argue that bad data in means bad data out and we just live with that scenario
• instead, with null as the default, the results are incalculable and also return null for periods of missing quote values
• null is easier to handle whereas a bad calculation with 0 values is not logically identifiable in the results … so it’s better, right?

On the other hand, there’s a do-not-use nullable trend right now in C#. When did null become evil?

My view is that null is a useful way to represent incalculable or intentionally missing data; and it’s only bad when not properly avoiding NullReferenceException scenarios.

What’s your opinion on the topic? Help me learn!

[DaveSkender]

From a user:

There has been vast amounts written on why null was the billion dollar mistake, but to get very specific to C# coding style, Anders Hejlsberg was very clear in an interview why he is on a war path to eradicate null out from the future of C#: "50% of the bugs that people run into today, coding with C# in our platform, are probably null reference exceptions."

[DaveSkender]

I totally agree with the null problem, especially in C#. With that said, I think it comes down to intentional design and handling -- if you're not considering and handling it, it is a problem. The big question is, without null, how are we supposed to represent incalculable data?

Microsoft's answer to this "oops, I didn't consider null" problem is to increase awareness with default ON nullable diagnostics in .NET 6 -- which is a good thing. With these new diagnostics, I've been able to find all sorts of "hidden" problems with my own use of nullable data types; but I haven't yet converted to the full "never use null" camp.

For handling, there's some good advice here: Introducing Nullable Reference Types in C#

[mihakralj]

In C# numerical analysis, there are three special constants that are not used often enough: Double.PositiveInfinity , Double.Epsilon and Double.NaN.

All three of them can be used in math calculations, do not raise exceptions and can be used/handled very gracefully for situations like div/0, missing data and, asymptotically small numbers.

On the other hand. nulls are pure evil and should be nuked from the orbit. :-)

[DaveSkender]

They can be, for sure. I always thought of NaN as a value returned when you did something funny, like trying to do math on a string or use an imaginary number. How does it behave differently (e.g. less evil) than null if you were to simply replace null values with NaN (assuming everything is double).

[NavjotSinghMinhas]

Any default object created with no value will be null; If we go on the path of setting null values. It will be difficult to know if it happened because of the data supplied or if some internal calculation resulted in a null. Therefore, I think using Double.NaN and null will be better to segregate things, instead of just using null everywhere.

[DaveSkender]

I'm actually coming to this conclusion myself, so I agree. It's difficult to balance some of these things, but I think I'll be returning double.NaN in cases where we hit naturally occurring div/0 scenarios. The library performs faster when I can avoid propagating null in internal calculations, but still think it makes sense to leave null as outputs where there's simply no data to calculate, such as we see in warmup periods.

[mihakralj]

One more case in point for NaN and against null:

• if element1=null and elememt2=null, then element1 == element2
• if element1=NaN and element2=NaN, by design element1 != element2

from dotnet documentation:
The Equality operator considers two NaN values to be unequal to one another. In general, Double operators cannot be used to compare Double.NaN with other Double values, although comparison methods (such as Equals and CompareTo) can.

For the use case of time series used in technical analysis, I prefer this behavior. If I am comparing current and previous value in the series close(n) and close(n-1), if they are null, they will look the same. If they are NaN, they will not.

[DaveSkender]

Good point. I’m fixing some of these in #799

[mihakralj]

The looming issue will be that NaN is of type Double while most of math here is done with Decimal types...

There is no Decimal.NaN constant.

[DaveSkender]

That is a constraint on some, but most of the indicators calculate and return double, actually. Most of the underlay oscillator types are double and decimal is just used where we’re returning price units on overlays. I’m slowly looking at converting some of those as well, but need to be careful in spots where more precision is needed.

[DaveSkender]

Based in feedback from users after experimenting with use of NaN in version 2.0.0, we’ve limited the use of it to internal processes only in v2.0.3 to reduce user friction. The takeaway here is that most apps and systems that use our library handle null more gracefully than NaN, especially when translating result values into 3rd party visualization tools.