-
Notifications
You must be signed in to change notification settings - Fork 365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallelism to improve grouping / sorting / filtering performance #534
Comments
I was going through the source and this is a snippet from the FilteredViewTransaction code: if (block == NULL) return;
if ((options & NSEnumerationReverse) == 0)
{
// Forward enumeration
std::vector<int64_t>::iterator iterator = vector->begin();
std::vector<int64_t>::iterator end = vector->end();
NSUInteger index = 0;
BOOL stop = NO;
while (iterator != end)
{
int64_t rowid = *iterator;
block(rowid, index, &stop);
if (stop) break;
iterator++;
index++;
}
}
else
{
// Reverse enumeration
std::vector<int64_t>::reverse_iterator iterator = vector->rbegin();
std::vector<int64_t>::reverse_iterator end = vector->rend();
NSUInteger index = vector->size() - 1;
BOOL stop = NO;
while (iterator != end)
{
int64_t rowid = *iterator;
block(rowid, index, &stop);
if (stop) break;
iterator++;
index--;
}
} I'm not sure I fully understand how vectors work in this case. We can convert this to use |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
One of my bottle-necks here (in the range of 100.000 elements in a collection) is applying the grouping, sorting, and in particular filtering blocks. As far as I can see we're only using one thread here, I wonder whether it would be possible to distribute that workload.
Applying a filtering block to one key, element, and/or metadata has no effects whatsoever on other filter blocks, hence I would think they could run in parallel.
Is this something to pursue further or not an issue at all for you guys?
The text was updated successfully, but these errors were encountered: