StreamDM-100 #101
Open
StreamDM-100 #101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/* * Cache class keeps a hashmap where it * stores the number of the bucket that holds * the major coreset and the coreset itself * r is the merge threshold in this case * we use r = 2 since we implemented an 2-way * coreset tree */
/** * Implements the CachedKM++ algorithm for data streams. CachedKM++ computes a * small (weighted) sample of the stream by using <i>coresets</i> from cache * to reduce the number of coresets needed for a merge step, and then uses * it as an input to a k-means++ algorithm. It uses a data structure called * <tt>BucketManager</tt> to handle the coresets. * * <p>It uses the following options: * <ul> * <li> Number of microclusters (<b>-m</b>) * <li> Initial buffer size (<b>-b</b>) * <li> Size of coresets (<b>-s</b>) * <li> Learning window (<b>-w</b>) * </ul> */
prevent sum from NaN when funcost is zero. splitCoresetTree needed more cases when leaf has no elements
Stream dm 100
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bug report #100 - part of the solution
1. remove duplicates of the output coreset
2. fixed the break statement
still needs some fixes in StreamKM
Summary of the changes
BucketManager's getCoreset returned a lot of duplicate Examples in the resulted coreset. Almost over 50% of the examples was duplicates. With these changes this percentage is around 10% but not zero.
in splitCoresetTreeLeaf sometimes the one child had all the points and the other was empty. Now it rechooses a center and splits the points again.
in splitCoresetTree changed the way of choosing the which node to split. Similar to moa's treeCoreset
in BucketManager the break statement in getCoreset does not work properly. Now there is a flag that stops the process when a full bucket is found.
Tests
need to print the resulted coreset in StreamKM be sure getCoreset is invoked at least once