Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* better benchmark * added generic sender/receiver testing * fixed the generator * fixed n benchmark * add NPSI testing of sender & receiver * adjust godoc to say \n for the FromReader versions * change load factor * debugged parallel npsi * documentation fixes * new version of util.Exhaust * use value struct * loadfactor 2 seems to be fine * fix typo * improve proto naming schemes * add data flow chart * fix flow chart * add ot readme * fix typo in OT readme, initial attempt in defining baseOT struct and interfaces * init files * embed BaseOt struct in NaorPinkas struct and Simplest struct * broken Send and Receive function, will use crypto/elliptic package instead of curve25519 * implemented simplest OT, channel shouldn't work yet, and need test * added tests for ot methods. * use io.ReadWriter instead of channel for Send and Receive * debug Read Write * first version of simplest completed, passed tests * close msgBus when sender encounter error * refactor * configure test to test on baseCount and on longer messages * rm random file * add ristretto implementation of OT, Simplest OT * improve tests * Add points.go as a thin layer of api for wrapping the points struct and elliptic api implements NaorPinkas baseOt with both elliptic and ristretto huge refactor of ot.go and ot_ristretto.go huge refactor of tests all tests passing. * refactor block cipher to its own file add XOR cipher: H(key, index) ^ plaintext add cipherMode in OT structs * xorcipher supports variable length messages, but it recycles the hashed keys * add benchmark for AES cipher and XOR cipher * fix typo in err messages * add time log in testing * mv ot, cuckoo to internal * IKNP OT extension (#20) * N choose 1 KKRT OT extension (#21) * fix Transpose3D test * KKRT OPRF (#22) * Define OPRF interface * Implement KKRT OPRF * Improve README and comments * KKRT PSI (#24) * parallelize kkrt ot receiver * new improved kkrt OPRF algorithm done by Justin * lower back the number of tests * improved OPRF for kkrtPSI * clean up * remove comments * improve cuckoo * lower back oprf test number * fix cuckoo test * improve oprf * cleanup * change back to use value instead of pointers for cuckoo (#27) * change back to use value instead of pointers for cuckoo * lower test numbers * buffer sender hashIds channel * use buffered channels * remove unnecessary waitgroup in stage 3 * convert cuckoo outputs to channels * static array of channels * use empty struct for map values * inplace bits operations * bump receiver size in tests * improve oprf kkrt by removing 3D transpose of matrices * fix bug in util.ExtractBytesToBits which ignores the last byte use kkrt oprf instead of the more complicated improvedKKRT * instantiate aes block and reuse * reset timer for inplaceXor benchmark * use gob to send precomputed hash maps * add dummy cuckoo to avoid allocation for sender * sender sends hashed encodings right away, receiver hash and index local encodings with corresponding ID and intersect each received hashed encodings * added mechanism to process a batch of identifiers per go routine and launch only runtime.GOMAXPROCS(0) number of gorouintes instead of a single goroutine processing a single identifier * inplace operation for PseudorandomCode * reduce PseudorandomCode allocation by 1 * remove outdated crypto encryption methods update base OT tests to reflect real use case scenario meaning number of messages is fixed to 512, and vary number of bytes per messages (this should be the same as 1.4 * number of messages) use Naor-Pinkas base OT instead of Simplest Update elliptic curve point deriveKey function, call point.x.Bytes() instead of points.Marshal() Add timing information in KKRT OPRF for better performance analysis * inplace xor id's last byte with hash idx, remove go routines from cuckoo hashtable and stage1 * add PrgWithSeed ImprovedKKRT seems to fail due to crypto/rand * reuse oprfInputs in stage3 to avoid reiterate on cuckooHashtable buckets remove expose cuckooHashTable api * remove blake2 encrypt decrypt * remove constant XORBlake2 * use AES-CTR Drbg * undo a previous commit that broke ImprovedKKRT with the incoming BitVec implementation, we should no longer need the extract 1 byte to 8 bytes, each byte containing 1 bit of information anymore * implement proper pseudorandom generators update all ot/oprf/ot-extension that uses prg remove the hacky PseudorandomGeneratorWithBlake3 which just uses Blake3.Read() * update new cuckoo benchmark in readme which has 2 times speed up in insert * cleanup * rename crypto to cipher * buffer reader/writer in stage3 for send and receive hashes * buffer map * KKRT BitVect (#36) * Untested blockwise transpose * Start testing of 512x512 transpose * Fix 512x512 transpose * Add ability to pad blocks and combine functions for tall and wide matrices * Unravel a 2D matrix into 512x512 padded bit blocks * URaveling and unraveling methods rough but not fully working * Refactor Unraveling methods * Fix raveling and unraveling methods. * Benchmark transpose * Crude first working concurrent transpose implementation * Don't export unnecessary functions * Move random matrix generation to bits.go * Refactor and improve performance on transpose * Add variables for testing * Slice and matrix conversions for Byte to Uint64 along with XOR and AND * Update naming of conversion functions * Try to apply BitVect-based transpose to KKRT * Revert "Try to apply BitVect-based transpose to KKRT" This reverts commit 37bdd60. * lower test case for transpose to pass pipeline * Remove padding support to BitVect * SBitVect little endian implementation start * Wrapper function around convert and transpose and convert tests * Debugging baseOt * use the correct choice bit * kkrt runs but fails * read bits in little endian way * happy path working, but we really need to figure out what is going on with the secret choice bits * test with sending only T = 0, d = 1 matrix * Convert transpose to work Little Endian * Begin partially unrolling of uint64 portion of transpose * KKRT with BitVect transpose * Improved KKRT uses BitVect transpose * Fix improvedKKRT oprf fix tests bitvec version of KKRT working * Address comments on PR #36 * Address comments from PR #36 * Set k constant to 512 bits in OPRF Co-authored-by: Justin <[email protected]> * Modify concurrent transpose to use a number of goroutines equal to number of cores * remove unnecessary OTs remove ExtractByteToBits Convert all remaining OT/OT-extension to use densely packed bytes * move points.go from internal/ot to internal/crypto * use uncompressed marshal and unmarshal for points since it's faster fix all OT/Ot-extension to use only padTil512 fix all tests * remove findK since no longer needed * remove old proto * Apply suggestions from code review Co-authored-by: Xavier Capaldi <[email protected]> * Apply suggestions from code review Co-authored-by: Xavier Capaldi <[email protected]> * preallocate aesBlock * simply allocation in bpsi * KKRT concurrent bit operations (#39) * Cleanup XOR operation and write concurrent version * Add concurrent in place AND operations and clean up * avoid append after cuckoo hashing * Clean up KKRT code (#40) * clean up cuckoo * add clean util/bit * remove pad function since it's not needed elsewhere * Cleanup docstring * add comments to prg remove unused functions in util/bits * changed PseudorandomCode to use all bytes of input, instead of always the first 15 bytes * add comments for points * combine baseOT test and OT test * hide New point interface * unexpose points in baseOT * cleanup ot * Cleanup bit and bitvect * change how oprf keys are stored and used * change K to pointK * remove err in ristretto * move deriveKeyRistretto to crypto/points * move ristretto point related functions to crypto/points * refactor points and ristretto points * minor fix up Co-authored-by: Xavier Capaldi <[email protected]> * copy to a preallocated slice instead of appending * Revert "avoid append after cuckoo hashing" This reverts commit 17749f6. * hide oprfinput details * change batch size to be correlated to number of cores each batch contains 42 * 768 bits of inputs * revert back to constant * update README Correct typo in README and update/simplify diagram * update diagram in readme * Add printing of memory allocations in KKRT test * Improve memory and time reporting * Output memory information to stderr * Move memory logging * Print memory info in MiB rather than MB * Remove append while creating OPRF input * Remove tmp slice for PseudorandomCode * Cuckoo hash index now stored as first element in front of item * Remove padding and encryption of dummy pseudorandom code * Reuse temporary slice in BitVect transpose * Concurrent unsafe casting between byte matrix and uint64 matrix * Clean up casting so default is byte to uint64 cast * Revert densely encoded cuckoo which is still buggy * Revert densely encoded cuckoo which is still buggy * Cleanup unnecessary utility functions for conversion between bytes and uint64s * Fix PseudorandomCode so it works with 16 byte input * add timing of cuckoo insert * Reduce memory usage and improve performance, focusing on stage 2 (#43) * Stop appending hash index to value * Test hashing functions (SHA256, FNV1a) * Store identifiers and bucket lookup indices in cuckoo struct * Change signature of OPRF to receive Cuckoo directly * Cleanup * Pass maps instead of arrays * Fix PseudorandomCode so it works with 16 byte input * Append hash index again right before inputt to PseudorandomCode * Includes the hash indices in the Cuckoo struct * Various optimizations to improve performance and reduce memory in stage 2 * Start cleanup * minor clean up in oprf.KKRT and oprf.ImprovedKKRT fix oprf tests * fix PSI tests * remove print statements and close err channel * Use unsafe casting to improve performance of bitwise operations * Clean up bit operations and remove AndByte * Add error checking * Apply suggestions from code review Co-authored-by: Justin Li <[email protected]> * Corrections from PR review * Small fixes for PR Co-authored-by: Justin <[email protected]> * Address suggestions from review in cipher.go * Add CipherMode type as suggested in review (#44) * Add CipherMode type as suggested in review * Add undefined default value for CipherMode type * KKRT Cleanup 1 (#45) * Incremental cleanup 1 * Cleanup bit utilities * Kkrt cleanup 2 (#46) * Incremental cleanup 1 * Cleanup bit utilities * Delete unused function * KKRT updated benchmarks (#47) * Update heatmaps for previous benchmarks * Resize plots * Update plots * Fix plot * Update plots * Add memory and GC plots * Update plots again * Update plots x3 * Add last data * Split detailed KKRT benchmark into its own file * Grammar and reduce height of scatter plots * Add benchmark for varying system threads * Add Bosko's description of thread results * remove unused OT, and clarify readme (#48) * rm ot, and clarify readme * remove KKRT oprf as well * reflect new changes to the OPRF readme * fix golangci-lint errors * Apply suggestions from code review Co-authored-by: Xavier Capaldi <[email protected]> Co-authored-by: Xavier Capaldi <[email protected]> * Add comment that only tested on AMD64 as well as improved concurrent bit op functions * Fix collision issues for items shorter than 64 bytes and issue with bit operation * Remove unnecessary copy after unsafe cast * test PSI with different length inputs (#49) * test different size * test the right protocols * test 8 bytes input as well * test prints the actual number of bytes being matched (including the 2 bytes from prefix) * KKRT Proper Pseudorandom Encode and Precompute Hash in Stage 1 (#50) * No longer send number of OPRFs since it can easily be calculated locally by sender * Remove allocation for number of OPRFs * Precompute pseudorandom ids in stage 1 of sender * Merge upstream changes * Cleanup and write docstrings * Fix typo * Make more concise * Update pdocstring * Update docstring again * Fix benchmark * Use GOMAXPROCS(0) rather than NumCPU() * Split transpose into different functions for wide and tall * Change pseudorandom code to handle 8 byte output from hash function * Fully convert to use xxhash * Bug fix * Use OneOfOne implementation of xxhash * Update docstrings * benchamrk other hash functions * add xxh3 as a new hasher, but it's not as fast as highwayhash for hashing to uint64 * add second murmur3 golang package * Perform OPRF encode in-place * Test unsafe casting for output of Murmur * Use TWMB Murmur3 for hashing in PseudorandomCode * OPRF encode in-place * Fix typo * Use hash index as both seeds and update docstring * Remove duplicate * Cleanup * Remove useless bit tests * Remove all unused hash functions * Removed unused functions in PRG * Cleanup * Tidy go mod * Cleanup and add tests and benchmarks for bit utils * Panic on error in encode and hash Co-authored-by: Justin <[email protected]> * Move unused testing functions into bitvect test file * Update benchmarks * KKRT Purge (#51) * Naor-Pinkas only base OT * XOR cipher with Blake3 is only cipher mode * Remove unnecessary function * Use P256 as only elliptic curve * Remove other curve (P256) * Remove ristretto points * Address suggestions in code review #1 * Address review comments #2 * Address review comments #3 * Corrent endianness of PseudorandomCode * Update Readme to specify that it is only compatible with x86-64 * Address review comments #5 * Adjust tests * Remove Encrypt and Decrypt functions and instead just use XorCipherWithBlake3 * refactor internal/crypto/point.go and its tests * define Equal for points * Add static tests for PseudorandomCode and Encryption/Decryption * Add encryption followed by decryption test * add kkrt psi description in main REAME * Replace all usage of math/rand with crypto/rand * Clean up tests * use logger in kkrt (#53) * Add log to PSI stages (#42) * add logging * /s/Finish/Finished/g * remove testing logs Co-authored-by: Xavier Capaldi <[email protected]> * add log to indicate verbosity will default to 0 with values outside of [0, 2] * add exitOnErr function * remove logr package, embed logger in sender/receiver struct * do not embed logger in sender and receiver update README * fix newline * reflect changes to README Co-authored-by: Xavier Capaldi <[email protected]> * add logging to kkrt report memory stats with log.V(2) * format memory * fetch logger with FromContextOrDiscard * rm comments * Update README.md * Update examples/receiver/main.go * Update pkg/bpsi/sender.go Co-authored-by: Xavier Capaldi <[email protected]> * KKRT Cuckoo Update (#54) * Check errors and use a CuckooHasher instead of a DummyCuckoo * Track item index in Cuckoo and avoid Cuckoo channels * Apply suggestions from code review Co-authored-by: Justin Li <[email protected]> * Address comments from review Co-authored-by: Justin Li <[email protected]> * fix pipeline * Fix cuckoo benchmarks * Remove colon from log * Make logging of memory and gc calls consistent for benchmark parsing * KKRT Metro Hash (#56) * Test Metro Hash * Test another Metro Hash implementation * Test City Hash from Google * Swap Highway Hash for Metro Hash * Update docstring * Specify that 'm' stands for 'million' in the benchmarks * address stylistic comments * New metro hash behavior * use go-metro hasher * remove oprf interface, and rename improvedKKRT to simply OPRF * address OT comments * move declaration of overwritable variables inside loops in NaorPinkas OT * annotate all NaorPinkas errors * add IsBitSet and BitExtract helper methods * share format function calls between examples/sender and examples/receiver * remove error from getBlake3Hash since blake3 does not return errors * Remove references to improved KKRT * Rename struct to indicate it holds the encoded input * Remove testing of unsafeslice conversions * Transpose determines number of workers internally * Number workers internal to wide transpose function * Divide blocks among workers in tranpose * Pass BitVect blocks via pointers to reuse The unraveling and reraveling functions should now operate on BitVect pointers. BitVects can be instantiated once per worker in the transpose method and then reused for all blocks. This improves performance. * address comments on oprf * remove error handling in go routine, and panic instead * move SampleRandomOTMessages to oprf * simplify testing with net.Pipe() fix ot bugs remove error from instantiating a new OT or a new OPRF * Pass OPRF encoded inputs with hasher via struct Rather than passing each encoded input into a buffered channel and the hasher via it's own channel, we combine them into a new struct. The struct contains the full slice of OPRF encoded inputs along with the hasher. Rather than using an error channel, we panic as the only possible error is due to programmer error in generating the seed for hashing or AES encoding. * Simplify concurrency in encodeAndHash Remove complex job structure and replicate goroutine model used in the rest of the library for transpose and bit operations. * KKRT Property-based testing for bit operations (#58) * Use property-based tests on bit operations For all bit operations, test that the fast or concurrent versions return identical results to the naive implementation. For the fundamental operations (AND and XOR), test their properties as well. XOR: - Commutative A ^ B = B ^ A - Associative A ^ (B ^ C) = (A ^ B) ^ C - Identity A ^ 0 = A - Self-inverse A ^ A = 0 AND: - Annulment A & 0 = 0 - Commutative A & B = B & A - Associative A & (B & C) = (A & B) & C - Identity A & 1 = A - Idempotent A & A = A * Remove single-threaded transpose benchmark * KKRT panic on bit errors (#59) * Panic for non-equal length input to bit ops In the context of our use, the input to bit operations should always have equal length. Instead of returning an error, we will panic. * NewCuckooHasher panics instead of returning error * Avoid panic by initialize AES outside goroutine * Panic when try to get item at index greater than number of items * Simplify EncodesRead so it can be used in loop * Use const in tests * Update comments in OPRF * sequential key generation and encode on sender side, and key generation and decoding on receiver side * Consolidate Pad and padBitMap into a single function * Range over inputs rather than channel * give more descriptive names to oprf sender * Begin amortization in stage 3 * Remove unnecessary error checking in KKRTPSI * Use ErrGroup to handle err in ParallelEncodeAndHash * Remove comment * Amortize by batching encode and hash * Update comments and small details * /s/Keys/Key/g * /s/msgLen/msgLens/g /s/baseMsgLen/baseMsgLens/g /s/sk/secretKey in oprf * panic on wrong input for cuckoo.GetBucket /s/oprfEncoding/oprfEncodings/g * /s/EncodesRead/EncodingsRead/g /s/EncodesWrite/EncodingsWrite/g * add PadBitMap * cosmetic changes in kkrt sender * remove polymorphic New in hasher * return Hasher * instantiate AESBlock outside of goroutine to avoid panic * receiver deduplicate intersected items * reworked sender batch encodeAndHash utility * Clean up encode and hashing in stage 3 sender * Pass inputToOprfEncode by pointer * Small grammar * move for loop in goroutine * Address comments on PR * fix merge artifacts * Remove comment and add two benchmark plots * Clean up comments * Updated benchmarks * update comments * Fix axis labels * Golint Co-authored-by: Dominic Gregoire <[email protected]> Co-authored-by: Xavier Capaldi <[email protected]> Co-authored-by: Xavier Capaldi <[email protected]>
- Loading branch information
4 people
authored
Dec 7, 2021
1 parent
545c3de
commit b236d33
