Window Resize Handling

[ludos1978]

Hi, i have been trying to figure out why screen recording stops when windows are resized. I figured out that, while it restarts the thread when capturing is interrupted by window resizing, it will allways abort. The reason is that it requires the stored window size to be the same as the image received from the desktop. The window is the one first defined when starting capturing, which will cause the thread to be aborted every loop. On OSX this is caused by:

line 30 in src/ios/CGFrameProcessor.cpp

        if (width != window.Size.x || height != window.Size.y) {
            CGImageRelease(imageRef);
            return DUPL_RETURN_ERROR_EXPECTED; // this happens when the window sizes change.
        }

commenting that lines seem to fix it and not cause any problems, but i dont know about the other plattforms.

---

Now the question is how should it be handled? Send a callback trough the API to give the possibility to react to the thread being restarted. Or should it try to keep recording the same window. In that case do we need to inform the user about the image size change, or do we assume he gets the image and can handle it on his own.

[smasherprog]

Your saying it does not resume capturing?

The reason the library stops and restarts is mostly because its easier to do this.
There are buffers and assumptions that the library makes internally about the state of what needs to be captured. So, if the buffersize, window changes, monitor size changes, then the library will return the error your are referencing and restart itself automatically.

[smasherprog]

Also, removing that line of code will cause your stomp all over memory if the new area to capture is larger than the previous one since the buffers are pre allocated in the init function and not in the process functions

[smasherprog]

screen_capture_lite/src/ScreenCapture.cpp

Line 76 in 49873d5

void start()

This is the main loop. As long as expected errors keep recurring, the library will restart itself automatically.

[ludos1978]

Your saying it does not resume capturing?

Technically it does resume, but restarts immediately again. So for the end-user it does not resume capturing.

Because the variable named window (and it's parameter size) is not be changed before restarting. It will keep restarting every loop, because window.size.x and window.size.y will rarely ever be the same value again.

[smasherprog]

Okay so the library is working.
The issue your pointing out is that while resizing a window, the library does not continuously capture the resizing as fast as it should because it restarts itself during the resizing.
This can be fixed, but code will have to be changed to accommodate this.

[ludos1978]

I dont think it will resume capturing the window at any point in time. At least it did not at any point in time for me.
The window variable is not the one internally available, but the one given by the user when window-capturing starts. Which has a fixed size defined. Correct me if i read the code wrong, but thats what my experience until now tells me.

[smasherprog]

When an expected error occurs the library will rebuild itself calling init, getting a new list of windows/monitors to capture.

Just following the code from what I linked above.
The start function calls ThreadManager init which calls the callbacks
getThingsToWatch (Badly Named, but is the callback to get a list of monitors or windows to watch)
Then a thread is spawned for each of the things to watch
When the thread is spawned, buffers are allocated and each thread goes into a loop that will exit when an expected error occurs
When ANY thread has an expected error, they all stop and exit.
ScreenCaptureManager will join(wait) on all threads
After that the process restarts itself from what I listed above.

[smasherprog]

I think you will need to debug a little more unfortunately
as the library is supposed to rebuild itself when a resize event occurs(Not the best way it should be handled but the easiest to support this event)

So add some debug statements around the code base to see where the code is failing to restart. You would be the first person to report an error like this so its not likely this is whats happening

[smasherprog]

These are these are the areas where these events are handled

screen_capture_lite/src/ScreenCapture.cpp

Line 85 in 49873d5

if (Thread_Data_->CommonData_.ExpectedErrorEvent) {

AND

screen_capture_lite/include/internal/ThreadManager.h

Line 112 in 49873d5

if (ret != DUPL_RETURN_SUCCESS) {

So if its not resuming this will be easy to spot

[ludos1978]

1. window variable is defined when starting the thread
2. window is resized while capturing
3. if (width != window.Size.x || height != window.Size.y) will fail and abort the thread
4. thread is restarted
5. if (width != window.Size.x || height != window.Size.y) will fail and abort the thread
6. thread is restarted
7. ... (will repeat forever)

i dont really know how to explain it better then this.

[ludos1978]

just to be sure: this is on ios / osx implementation, i dont know about the windows or linux implementation.

[smasherprog]

Ok, so your saying when the thread aborts, is the callback to get windows called again?

screen_capture_lite/Example/Screen_Capture_Example.cpp

Line 132 in 49873d5

auto mons = SL::Screen_Capture::GetMonitors();

So im gonna layout what I think is the the callstack

screen_capture_lite/src/ios/CGFrameProcessor.cpp

Line 29 in 49873d5

if (width != window.Size.x || height != window.Size.y) {

Is called and DUPL_RETURN_ERROR_EXPECTED is returned

The function above is this one

screen_capture_lite/include/internal/ThreadManager.h

Line 134 in 49873d5

template <class T, class F> bool TryCaptureWindow(const F &data, Window &wnd)

This function has the loop for the window capturing and it keeps going
while (!data->CommonData_.TerminateThreadsEvent) {

On this line

screen_capture_lite/include/internal/ThreadManager.h

Line 152 in 49873d5

if (ret == DUPL_RETURN_ERROR_EXPECTED) {

data->CommonData_.ExpectedErrorEvent = true;
Then a few lines down there is a

return true statement.

Does this seem correct so far?

So, the thread is created here for that function above

screen_capture_lite/src/ThreadManager.cpp

Line 40 in 49873d5

m_ThreadHandles[i] = std::thread(&SL::Screen_Capture::RunCaptureWindow, data, windows[i]);

The line above is called from the MainScreenCaptureManager

screen_capture_lite/src/ScreenCapture.cpp

Line 83 in 49873d5

while (!Thread_Data_->CommonData_.TerminateThreadsEvent) {

As you can see, when
Thread_Data_->CommonData_.ExpectedErrorEvent is true
Thread_Data_->CommonData_.TerminateThreadsEvent = true;
Which causes everything to stop and restart itself.

So, the window variable you referring to will be thrown out because this line will recreated everything

screen_capture_lite/src/ScreenCapture.cpp

Line 93 in 49873d5

ThreadMgr.Init(Thread_Data_);

I think we need more information. The line you are referring to will recreate everything

If you can verify that when the thread restarts, that the callback to get the windows to watch is called, that would be helpful because it means that code is working fine and its something else.

Can you put a stop on the window size not matching to see how much it is off by? Is it off by 1, or is it 0. I think that's helpful too,

I wana help, but from everything i am seeing, its likely something other than the library.

A way to figure this out would be to run the example file and see how that behaves to eliminate any other factors.

[smasherprog]

What I think we will find is that you have a buffer overrun somewhere that is overwriting the memory.

[smasherprog]

Another possibility is that the size of the image coming out of

screen_capture_lite/src/ios/CGFrameProcessor.cpp

Line 21 in 49873d5

auto imageRef = CGWindowListCreateImage(CGRectNull, kCGWindowListOptionIncludingWindow, static_cast<uint32_t>(window.Handle),

has changed in some release of ios. Maybe it not includes padding or something else?
The get windows function here
https://github.com/smasherprog/screen_capture_lite/blob/master/src/ios/GetWindows.cpp
Might have some strangeness going on too.
Or some scaling issue might be occurring. Unfortunately, I dont use ios and developing on it is cumbersome so it takes me a long time to around to debugging ios.

[ludos1978]

I can Agree to everything you wrote above except 1 thing i assume is wrong / different in my case:

So, the window variable you referring to will be thrown out because this line will recreated everything

ThreadMgr.Init(Thread_Data_);

It will use the old Window variable given by the user at the first/initial creation of the thread. It will not use the updated window size. It might actually also be that because using a c interface the window variable content is a copy of the original window variable and not a pointer to the internal variable.

[ludos1978]

After long consideration of my problem and your description how it should work i am pretty sure about the cause of the problem.

in c++ you return a list of windows to the user, which afaik is actually a pointer to the variable. The user returns the list of windows he wants to capture, which usually would be the original variable you sent the user in the first place. Using the C-Interface or from C# i cant return the same variable, so it's a copy of the variables content. I actually wondered why you would use the whole window variable to define which windows to capture. Thats the reason my window variable is static and does not get udpated with the new window size values.

[smasherprog]

So this is happening when serializing from c to c#?
I am going to be working on that more today. Its alot more work than I thought it would be

[smasherprog]

Since the handle is a size_t, you can use IntPtr for the c# representation.

[ludos1978]

I could be mistaken, but i think there is no way of accessing the contents of a intptr without copying the contents from the intptr using marshal. also i wouldnt know how to manipulate and return a pointer at all from c#.

In my opinion it's a very bad idea to assume that the user returns the same pointer, as the library returned from a function. Especially when it's interfaced with other languages that might not even know about pointers. I'd rather just use a api that takes window/screen id's (if that is fix and consitent over all plattforms) and possibly a rect for a specific part of a screen, to define which parts to record.

[smasherprog]

I just merged a large PR into master which includes part of the working c# implementation. So, that should help out with examples on how to move forward.