problem with WatchDog or RebootDevice(int exeConstraintTimeout)

[lsd-techno]

Was looking how to implement hardware level watchdog to the system.

Most suitable function what I found is 'nanoFramework.Runtime.Native.RebootDevice(int exeConstraintTimeout)'
Was trying to reinitialize it before timeout expires.

Where 'exeConstraintTimeout' should be timeout in milliseconds, but this method fires 'OnRebootEvent' right after execution with any number 50K to 50M that suppose to be 50 seconds in milliseconds or in nanoseconds... just in the case of typo.
When 'OnRebootEvent' completes firmware goes directly to reboot.

Does anybody use any hardware watchdog?

[josesimoes]

If I'm no t mistaken, hardware watchdog it's enable as default for all platforms...

As for the call to RebootDevice the parameter is not supposed to be a delay, rather a timeout for any pending event handler to complete before the timeout expires.

[lsd-techno]

Thanks, for RebootDevice it is clear now.

Then it will be good option if we can control hardware watchdog from managed code.
In the case if anything goes wrong and managed code can not manage to ignite watchdog counter reset.

That would be good to have something like:

bool nanoFramework.Runtime.Native.WD.Status(get)
uint nanoFramework.Runtime.Native.WD.Counter(get;set); //optional
bool nanoFramework.Runtime.Native.WD.Arm(uint Counts)
bool nanoFramework.Runtime.Native.WD.DisArm()
bool nanoFramework.Runtime.Native.WD.Feed()

[josesimoes]

Hardware watchdog it's already being managed, started and feed. That ensures that the watchdog will do it's thing, if needed.
It's not exposed to managed code on purpose. Like other things typical from embedded systems world.
C# developers don't need to worry with the watchdog, just like one doesn't have to when coding a C# app for desktop.

[lsd-techno]

I have application that doing some action on timer or other events.
Then (it happened 2 times) timer event disappear... device was connected to uart (not debugger) to collect logging.
I didn't see any exception in the log related to timer.
So I was not able to find what is the reason for timer disappear. Then I start to search what maybe helpful to solve this issue.

I can make another timer and event that will check if main timer is alive. But what if both of them will disappear.

Then I start to investigate if there is any memory leakage or something... but didn't find anything with GC.Run(false) - it always shows >68KB free

So first idea that came it would be good to have hardware watchdog under control.

PS:
actually my current timer that fires every second is checking that other modules are up and running and sometimes ignites event to make http requests.

[alberk8]

I was also going to ask about having access to WDT in managed code. There are many times where the underlying CLR or RTOS is still running but the app hangs due to some other issues, a managed WDT would be good in this case.

[lsd-techno]

@josesimoes @alberk8
I did some deeper research on hardware watchdog. It is integrated into CLR too much to handle it in managed code.
If rely on CLR stability in handling HWDT (we still can ping device on network interface when app "hangs", for the moment not sure who responds to the ping) then some kind of software emulation e.g. special timer that feed CLR with it's activity would be good option.

For the moment I found that my scheduler timer was calling function that was calling HttpClient.
Then sometimes HttpClient got exception socket error (which I suspect related to some wifi or other network issues).
But effect was not immediate, timer continue execution few more times after exception happened then suddenly its output to debug console and functionality that it suppose to run disappear. HttpListener disappear for some reason too even though thread that handles it suppose to restart it and doing that successful, left me with only option to ping it's IP address.
So I think that exception handled inside HttpClient that push me to dig deeper :)

I've add try/catch to wrap that suspected place... let's see if it will catch any error and let timer function survive.

[josesimoes]

I think you're trying to "fix" an issue elsewhere by adding more control over it and putting the burden of responsibility where it doesn't belong.

The execution engine it's responsible for keep threads flowing and there will be hardly a way to get it locked if it isn't for a call to native code that hangs or a bug on itself. The first it's already covered by the HWD.

In the situation that you're describing it seems that this is a bug in the http library. Which is complex and has been improved with time. Throwing an HWD there won't fix it, at most will remedy it and highly likely that it help to find the root cause.

The sensation of control of having a HWD available that you can feed it's a false one. You don't have it in a desktop C# app, so this begs the question : why would you need it here?
Actually it may be even worst because you'll give control (and responsibility) to the developer on something that he/she only partially controls (there's a LOT) going on at native code that accounts for potential lock, more than with C# code. So that would be, at best, confusing and unfair.

Plush there is also the viability, there are platforms on which HWD once started can't be stopped without a cpu reset. That would make it impossible to honor APIs like Start and Stop.

[lsd-techno]

Yes, as see from my exploring I found that we no need to take direct or indirect control on HWD from managed code, CLR is responsible for that.

But some kind of alternative that will let CLR to decide that application not doing what it suppose to do (if it is activated) then make decision to restart application or reboot hardware or throw some exception.

As an alternative such virtual WD for managed code may be done with nuget package (If another assembly not affected by main user assembly hanging) then it will be good solution for debug and good entry point for beginners to switch from native HAL implementation to managed C#.

I will try to play with nugets to understand how to manage them.

PS

After wrap around call to function that use HttpClient (even though that function have it's own error handling) with try/catch I didn't have any hang cases yet. But didn't have any catch cases too. Will keep it for testing longer time.

Some times I get guru meditation on web requests. Usually but not always that happens when I don't send any webrequests to device for a while then first request to device generates error then reboots itself.

In my case reboot is better than hang :). So I didn't investigate HttpListener that much.

GC runs with false option every minute - just to see what is going on with free RAM.

[alberk8]

This error looks like the native code try to read an address that does not exists anymore and the RTOS panics and reboot. This is a lot easier to debug then hang without any message.

[josesimoes]

@lsd-techno calling GC.Run puts pressure on the GC and eats a lot pf CPU cycles. Calling it very often can be counter productive...
I wouldn't be surprised if you remove the call to GC.Run that the guru meditation error would go away.

[lsd-techno]

@josesimoes
Every minute it is like normal rate especially for debug stage and some kind of stress test. I have WebRequest that also executes every minute but different time (some kind of state machine). So in that way I can test stability and find if there is any bugs.

I suspect guru meditation was coming from HttpListener thread (it is running on its own now and my "scheduler" timer not checking if it is alive).

I've add software flush emulation for serial ports too - simple read out data from serial port in parts to save RAM until there is data if I see there is more data than it is suppose to be. And try to check every place where I could find anything uncertain.

So flush buffers for serial ports might be good idea to implement too.

For today test I din't have any guru meditation yet... even though deploy insrease few 1~2 KB and free RAM in GC shows less 1 KB.

[josesimoes]

Fell free to call it at the rate you feel like, the API it's there to be used.
Was just trying to point out that GC it's perfectly capable of stand alone operation without any intervention from the developer.
If it's needed it runs, if not, it will stay quietly waiting.

[josesimoes]

Regarding the serial port flush: are you referring to the C# SerialPort class or to the native UART(s)?

[lsd-techno]

Yes, GC is not for regular use - with that I was trying to get any idea from where to start to look for the problem.
As a novice I need to poke everywhere :)

Regarding serial port: I'm using System.IO.Ports.SerialPort class to control ESP32 UARTs in my test project.
Native UARTs didn't try to use directly yet.

It is working in interrupt mode. Some moment I found that sometimes because of different hardware issues I may receive "noise" from UARTs when ESP32 device reboots or when UART targets reboots.

So I want to flush received data that I don't want to process.
First I try to close then open UART but currently I can not open it after close. It gives me exceptions - will check it later.
Then I try to check if there is any flush exist. Didn't find. Then implement readout what all in the buffer, print to debug report,

            int sens_len = 17;
            if (serialDevice.BytesToRead > sens_len)
            {
                Thread.Sleep(30);
                Debug.WriteLine($"Flush {serialDevice.PortName} {serialDevice.BytesToRead} bytes...");
                byte[] trash = new byte[sens_len + 10];
                while (serialDevice.BytesToRead > sens_len)
                {
                    serialDevice.Read(trash, 0, sens_len);
                    Debug.WriteLine($"trash: {BitConverter.ToString(trash)}");
                }
                serialDevice.ReadExisting();
                Debug.WriteLine($"Flush {serialDevice.PortName}... ok");
            }

last serialDevice.ReadExisting(); could do same thing, but I notice that received data sometimes more than 200 bytes, to not let system reserve any extra buffer in RAM I add partial readout before that.

[lsd-techno]

With last application changes and latest firmware 1.8.0.313 and latest nuget updates firmware become much more stable.
and no exceptions (at least no one exception for unknown reason) anymore.
Only once I got exception when HttpListener die when wifi network disappear.
No guru meditation too.