Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added bloom filter support. Improve concurrency and fix some issues. #93

Closed
wants to merge 43 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
0b884e7
Removed some lock that are not required or very bad.
pcmind Jan 17, 2017
03e9367
Enable concurrent writes to DB folowing original implementation
pcmind Oct 26, 2017
38c1d3a
Enable concurrent writes to DB folowing original implementation
pcmind Oct 26, 2017
85fe5c4
Enable concurrent writes to DB following original implementation
pcmind Oct 26, 2017
f02c211
Add LRU read cache and cleanup table implementation
pcmind Nov 27, 2017
82c61ca
travis start
pcmind Nov 28, 2017
761d29c
Merge branch 'master' of https://github.com/pcmind/tldb
pcmind Nov 28, 2017
015b450
Fix black lines and remove proguard plugin
pcmind Nov 28, 2017
d050f22
Add configurable usage of Memory Mapped Files for writes
pcmind Nov 28, 2017
2351122
Fix non thread safe access to current version
pcmind Nov 28, 2017
489e47e
Fix non thread safe access to current version. Set to verbose unit test.
pcmind Nov 28, 2017
9f530cb
[Improve] Add filter policy with bloom filter implementation.
pcmind Dec 1, 2017
252540b
Merge branch 'master' of https://github.com/pcmind/leveldb
pcmind Dec 1, 2017
1d21845
Fix incorrect condition wait and release mutex on file write
pcmind Nov 5, 2017
35a5b76
Add more unit tests
pcmind Dec 5, 2017
deb4fd4
Abstract file access from it's implementation to improve code quality…
pcmind Dec 16, 2017
dd53f3f
Add multi thread benchmark support. Add Benchmark histogram output im…
pcmind Dec 17, 2017
558fa22
Enable to set bloom filter and block cache site for benchmarks
pcmind Dec 17, 2017
dd6c944
Remove unneeded data conversion to string, this boost performance of …
pcmind Dec 22, 2017
afa845e
Merge remote-tracking branch 'origin/master'
pcmind Dec 23, 2017
700b681
Fix manual range compaction and implement for full DB compaction usin…
pcmind Dec 27, 2017
4521c68
Merge branch 'master' of https://github.com/dain/leveldb
pcmind Dec 27, 2017
a3d22e6
Fix issue due to changes in pom and parent pom
pcmind Dec 27, 2017
199b1b1
Fix issue due to changes in pom and parent pom
pcmind Dec 27, 2017
5fcec6a
Merge branch 'master' of https://github.com/pcmind/leveldb
pcmind Dec 27, 2017
0b5ee7b
Fix SnapshotSeekingIterator and compaction issues with deleted keys
pcmind Dec 29, 2017
0141055
Reduce some un required instance creation and check only once per lev…
pcmind Dec 29, 2017
3cbb94c
Fix concurrency issue in compaction that could lead to mark a ManualC…
pcmind Dec 30, 2017
2fd0297
Use own implementation of findFile.
pcmind Dec 30, 2017
47e34e1
Implement snapshot tracking.
pcmind Jan 1, 2018
5eaf51e
Detect overlapping ranges in level 0 files. Fix issue #94.
pcmind Jan 3, 2018
25800f1
Merge branches 'fixoverlappingfiles' and 'fixsnapshots' into dev
pcmind Jan 3, 2018
5a61f30
Add git attributes
fulgas Jan 4, 2018
9fde742
correct eol lf
fulgas Jan 4, 2018
206ba93
Reduce code duplication of in LevelX classes.
pcmind Jan 4, 2018
90e9d8a
Check if immutable compaction is required before requesting mutex loc…
pcmind Jan 4, 2018
416a561
IMPROVE: add new line
fulgas Jan 5, 2018
0934b94
Merge pull request #1 from fulgas/master
pcmind Jan 5, 2018
767548e
Move IO access, out of mutex lock, in openCompactionOutputFile
pcmind Jan 5, 2018
be8125a
Merge remote-tracking branch 'remotes/origin/master' into develop
pcmind Jan 8, 2018
3c25bbf
add support for compaction stats
fulgas Jan 8, 2018
09e0c6c
Merge pull request #2 from fulgas/compactionstats
pcmind Jan 9, 2018
f2c0083
Fix stats time unit conversion
pcmind Jan 10, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
* text=auto
*.java text eol=lf
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ Recommended Package imports:

```java
import org.iq80.leveldb.*;
import static org.iq80.leveldb.impl.Iq80DBFactory.*;
import java.io.*;
```

Expand Down
11 changes: 9 additions & 2 deletions leveldb-api/src/main/java/org/iq80/leveldb/DB.java
Original file line number Diff line number Diff line change
Expand Up @@ -88,10 +88,17 @@ void suspendCompactions()
void resumeCompactions();

/**
* Force a compaction of the specified key range.
* Compact the underlying storage for the key range [begin, end].
* In particular, deleted and overwritten versions are discarded,
* and the data is rearranged to reduce the cost of operations
* needed to access the data. This operation should typically only
* be invoked by users who understand the underlying implementation.
* <p>
* Call to {@code db.compactRange(null, null);} will compact the
* entire database.
*
* @param begin if null then compaction start from the first key
* @param end if null then compaction ends at the last key
* @param end if null then compaction ends at the last key
*/
void compactRange(byte[] begin, byte[] end)
throws DBException;
Expand Down
3 changes: 3 additions & 0 deletions leveldb-api/src/main/java/org/iq80/leveldb/DBIterator.java
Original file line number Diff line number Diff line change
Expand Up @@ -62,4 +62,7 @@ public interface DBIterator
* Repositions the iterator so it is at the end of of the Database.
*/
void seekToLast();

@Override
void close();
}
63 changes: 63 additions & 0 deletions leveldb-api/src/main/java/org/iq80/leveldb/Options.java
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,30 @@ public class Options
private DBComparator comparator;
private Logger logger;
private long cacheSize;
private boolean allowMmapReads = true;
private boolean allowMmapWrites = true;
private XFilterPolicy filterPolicy;

public static Options fromOptions(Options options)
{
final Options options1 = new Options();
options1.createIfMissing = options.createIfMissing;
options1.errorIfExists = options.errorIfExists;
options1.writeBufferSize = options.writeBufferSize;
options1.maxOpenFiles = options.maxOpenFiles;
options1.blockRestartInterval = options.blockRestartInterval;
options1.blockSize = options.blockSize;
options1.compressionType = options.compressionType;
options1.verifyChecksums = options.verifyChecksums;
options1.paranoidChecks = options.paranoidChecks;
options1.comparator = options.comparator;
options1.logger = options.logger;
options1.cacheSize = options.cacheSize;
options1.allowMmapReads = options.allowMmapReads;
options1.allowMmapWrites = options.allowMmapWrites;
options1.filterPolicy = options.filterPolicy;
return options1;
}

static void checkArgNotNull(Object value, String name)
{
Expand Down Expand Up @@ -173,4 +197,43 @@ public Options paranoidChecks(boolean paranoidChecks)
this.paranoidChecks = paranoidChecks;
return this;
}

public Options allowMmapReads(boolean allowMmapReads)
{
this.allowMmapReads = allowMmapReads;
return this;
}

public boolean allowMmapReads()
{
return allowMmapReads;
}

public Options allowMmapWrites(boolean allowMmapWrites)
{
this.allowMmapWrites = allowMmapWrites;
return this;
}

public boolean allowMmapWrites()
{
return allowMmapWrites;
}

/**
* Set table filter policy
*
* @param filterPolicy new filter policy
* @return self
*/
public Options filterPolicy(XFilterPolicy filterPolicy)
{
this.filterPolicy = filterPolicy;
return this;
}

public XFilterPolicy filterPolicy()
{
return filterPolicy;
}
}
19 changes: 19 additions & 0 deletions leveldb-api/src/main/java/org/iq80/leveldb/WriteOptions.java
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,25 @@ public class WriteOptions
private boolean sync;
private boolean snapshot;

/**
* If true, the write will be flushed from the operating system
* buffer cache (by calling WritableFile::Sync()) before the write
* is considered complete. If this flag is true, writes will be
* slower.
* <p>
* If this flag is false, and the machine crashes, some recent
* writes may be lost. Note that if it is just the process that
* crashes (i.e., the machine does not reboot), no writes will be
* lost even if sync==false.
* <p>
* In other words, a DB write with sync==false has similar
* crash semantics as the "write()" system call. A DB write
* with sync==true has similar crash semantics to a "write()"
* system call followed by "fsync()".
* <p>
* In java Implementation if process crash
* Default: false
**/
public boolean sync()
{
return sync;
Expand Down
36 changes: 36 additions & 0 deletions leveldb-api/src/main/java/org/iq80/leveldb/XFilterPolicy.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
* Copyright (C) 2011 the original author or authors.
* See the notice.md file distributed with this work for additional
* information regarding copyright ownership.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.iq80.leveldb;

/**
* A database can be configured with a custom FilterPolicy object.
* This object is responsible for creating a small filter from a set
* of keys. These filters are stored in leveldb and are consulted
* automatically by leveldb to decide whether or not to read some
* information from disk. In many cases, a filter can cut down the
* number of disk seeks form a handful to a single disk seek per
* DB::Get() call.
* <p>
* Most people will want to use the builtin bloom filter support (see
* NewBloomFilterPolicy() below).
*
* @author Honore Vasconcelos
*/
public interface XFilterPolicy
{
}
1 change: 1 addition & 0 deletions leveldb-benchmark/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

<properties>
<air.main.basedir>${project.parent.basedir}</air.main.basedir>
<air.check.fail-checkstyle>false</air.check.fail-checkstyle>
</properties>

<dependencies>
Expand Down
Loading