initial sync: missing/duplicate entries depending on paging

[mdraheim]

version: 7.1.0

Depending on the page size setting, the initial sync can produce duplicate (silently ignored) or missing (fatal) entries.

This happened on one of three crowd instances. The adapter failed on group membership because of the group to add as member not existing. However, the group does exist in crowd.

Curl testing showed that paged results have duplicate groups, ie the same group in different chunks, and the same amount of missing groups, so that overall entry count is correct but the data is bad.

Can you add some kind of sanity check to catch this problem and, at least, log a warning hinting at paging? And add this as a possible problem to the docs? The workaround fix here was to increase page size until the errors no longer showed.

I guess the real problem is crowd's handling of paging and probably cannot be fixed in the short term