Page loaded but waitload/waitidle reach generates an error => No XHR in outputs

[alban-stourbe-wmx]

Current Behavior:

I'm currently crawling a headless site in order to detect as many XHR requests as possible.
While debugging, I noticed that some pages are well loaded in the browser and are stable. However, katana doesn't see them that way. The code waits for the timeout to be reached before returning the error.

The error is linked with this code:

The context exceeds because the timeout is reached, rendering the DOM inaccessible and causing the error.

However, in the browser, the page seems to be fully loaded.

According to the code, the page should have been closed and the reply processed, but this is not the case. The code is waiting for the timeout.

The problem is that XHR and other requests detected before the timeout are not written to the output because of the error.

Expected Behavior:

I'd like to understand why the page is considered unstable and yet appears to be so when viewed in the browser.

Steps To Reproduce:

1. Run Katana in headless mode
2. Choose web page endpoint with XHR and other requests starting from the page
3. Open browser for debugging
4. Check that a page is fully loaded but does not close in the browser, meaning that it is about to reach the defined timeout.

Anything else:

It could also be a performance problem with my computer. However, I'm on a MAC OS:

• 2.6 GHz Intel Core i7 6 cores
• 16 Go 2667 MHz DDR4
  And I also had the same problem on another PC.

Have a nice day !

[Fly-Playgroud]

katana can try use rod's Page.waitStable or waitDOMStable method instead of idle