New feature: Kalman filter with missing (NaN) values

[matyasfarkas]

Hi Thore, I saw it on your to-do list.

I always wished to be contributor to your wonderful project, thus I decided to make it happen, not the least it comes handy in my own work.

The change is super simple and works. In the Kalman iteration just replace:

ℒ.axpby!(1, data_in_deviations[:, t], -1, z)

with:

z = replace(ℒ.axpby!(1, data_in_deviations[:, t], -1, z),NaN=>0) # v = data_in_deviations[:, t] - z

Using julia's missing gave me type errors and headache with Turing. Changing to NaN -s opened a shortcut to just call a replace whenever a missing value it is encountered in the KF iteration, due to missing data, then the prediction error is set to zero, leading to no update. Let me know what you think!

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.58%. Comparing base (414d765) to head (c3023b3).
Report is 35 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (414d765) and HEAD (c3023b3). Click for more details.

HEAD has 22 uploads less than BASE

Flag	BASE (414d765)	HEAD (c3023b3)
	23	1

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #94       +/-   ##
===========================================
- Coverage   85.23%   61.58%   -23.66%     
===========================================
  Files           8        8               
  Lines        5492     6653     +1161     
===========================================
- Hits         4681     4097      -584     
- Misses        811     2556     +1745     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[thorek1]

Hi Matyas,

Thanks for pushing this issue forward.

I see the following todo's before merging this:

• Construct diverse test cases (n time series, with missing values in m<=n time series. Catch potential filter breakdowns for different degrees of missingness. Missing values at beginning, end, or at random.). See current Kalman related test cases.
• benchmark/profile solution for speed (categorical replace might be slower than if NaN... else. Not sure about overall overhead for that operation... benchmarking will reveal it)

Best

Thore