Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Test #6

Open
wants to merge 54 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
8087bf5
Merge client and replica
gabrieltron Jul 22, 2020
fb1db19
Create n threads to send requests
gabrieltron Jul 22, 2020
bad6ba6
Read requests outside threaded loop
gabrieltron Jul 22, 2020
033fbe5
Allow one workload file for each sender thread.
gabrieltron Jul 23, 2020
a3bf70d
Remove client threads
gabrieltron Jul 24, 2020
f3dd30e
Simplify requests format
gabrieltron Jul 25, 2020
6bd01f9
Remove barriers for ending.
gabrieltron Jul 26, 2020
d89b83f
Make each partition count its own requests.
gabrieltron Jul 26, 2020
17fd1e7
Remove necessity to pass port as arg.
gabrieltron Jul 26, 2020
a016e59
Remove port from run function
gabrieltron Jul 26, 2020
fccca0f
Make n partitons be a config option
gabrieltron Jul 26, 2020
cf8c8d1
Print latency with chronological info
gabrieltron Jul 27, 2020
4ca1a69
Make throughput print a counter, fix latency measurement counter
gabrieltron Jul 27, 2020
52258ad
Make scheduler control workload graph and partition syncs
gabrieltron Jul 28, 2020
6124de4
Fix update graph loop duration
gabrieltron Jul 28, 2020
e057b06
Ignore evclient and evpaxos on build
gabrieltron Jul 28, 2020
c5df4e0
Print makespan after execution
gabrieltron Jul 28, 2020
bd4f7ab
Merge branch 'master' into simplified-replica
gabrieltron Jul 29, 2020
d23a026
Stop recording latency
gabrieltron Jul 29, 2020
4e6adc5
Add round-robin as a repartition policy
gabrieltron Jul 29, 2020
0633c7c
Remove concurrent unordered set on graph
gabrieltron Jul 30, 2020
3c17e9b
Update workload graph in a different tread
gabrieltron Jul 30, 2020
facf039
Add O3 flag to repartitioning
gabrieltron Jul 30, 2020
836a5c1
Send sync request to graph update thread
gabrieltron Jul 31, 2020
14648a1
Don't update graph in roud-robin
gabrieltron Jul 31, 2020
497f520
Take end timestamp after join throughput
gabrieltron Jul 31, 2020
3db6e3c
Remove tbb from partition's includes
gabrieltron Jul 31, 2020
cdc79c6
Increse args size to 4096
gabrieltron Jul 31, 2020
eefb6f2
Remove args from write, put template in storage write
gabrieltron Jul 31, 2020
526ef89
Reduce size of client message args
gabrieltron Aug 1, 2020
fecde06
Stop creating reply struct
gabrieltron Aug 1, 2020
0456b02
Fix storage's template_value initialization
gabrieltron Aug 1, 2020
3551f2e
Compress data before store and retrieve it
gabrieltron Aug 1, 2020
43b85e5
Use zlib without boost
gabrieltron Aug 1, 2020
3d35cc2
Download zlib with conan
gabrieltron Aug 1, 2020
513a0fd
Sync partitions after repartitioning
gabrieltron Aug 4, 2020
ede9dff
Register moment of repartition
gabrieltron Aug 4, 2020
17ba701
Correct fennel cost function
gabrieltron Aug 7, 2020
7dc4d3b
Make REFENNEL use updated partition size
gabrieltron Aug 7, 2020
bbdca3c
Add max imbalance to fennel and refennel
gabrieltron Aug 8, 2020
7c723da
Make graph thread update keys
gabrieltron Aug 12, 2020
5525d15
Correct max partition size formula on fennel and refennel
gabrieltron Aug 12, 2020
bc21598
Make ReFENNEL use the scheduler's mapping
gabrieltron Aug 13, 2020
8a52572
Insert data with the proper weight in ReFENNEL
gabrieltron Aug 13, 2020
018f16a
Do not increase variable weight on inserts
gabrieltron Aug 13, 2020
9200ccf
Remove unused variable
gabrieltron Aug 13, 2020
22432c7
Remove vertex from partition before finding a new one on REFENNEL
gabrieltron Aug 13, 2020
aa11291
Reduce size of array used for compression and decompression
gabrieltron Aug 14, 2020
3a3ee4d
Ignore exception from decompression
gabrieltron Aug 14, 2020
347cc03
Pass proper partition pointer to graph thread when adding key
gabrieltron Aug 14, 2020
8d98711
Remove unused nada structur
gabrieltron Aug 18, 2020
dd486de
Execute FENNEL if it's the first time executing ReFENNEL
gabrieltron Aug 24, 2020
59be4e9
Make ReFENNEL updata partitions after first repartition
gabrieltron Aug 25, 2020
8d691b2
Make refennel reexcute under fennel at all repartitions
gabrieltron Feb 12, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ include(cmake/conan.cmake)

conan(
PACKAGES
zlib/1.2.11
toml11/3.4.0
libevent/2.1.11
tbb/2020.1
Expand Down
23 changes: 2 additions & 21 deletions src/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,30 +1,13 @@
add_subdirectory(compresser)
add_subdirectory(request)
add_subdirectory(evclient)
#add_subdirectory(evclient)
add_subdirectory(types)
add_subdirectory(storage)
add_subdirectory(scheduler)
add_subdirectory(graph)

add_executable(client)
add_executable(replica)

target_sources(
client
PRIVATE
client.cpp
)

target_link_libraries(
client
PRIVATE
CONAN_PKG::tbb
CONAN_PKG::toml11
request
evclient
evpaxos
types
)

target_sources(
replica
PRIVATE
Expand All @@ -36,8 +19,6 @@ target_link_libraries(
PRIVATE
CONAN_PKG::toml11
request
evclient
evpaxos
types
scheduler
)
234 changes: 0 additions & 234 deletions src/client.cpp

This file was deleted.

15 changes: 15 additions & 0 deletions src/compresser/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
add_library(compresser)

target_sources(
compresser
PUBLIC
compresser.h
PRIVATE
compresser.cpp
)

target_link_libraries(
compresser
PUBLIC
CONAN_PKG::zlib
)
93 changes: 93 additions & 0 deletions src/compresser/compresser.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
// Copyright 2007 Timo Bingmann <[email protected]>
// Distributed under the Boost Software License, Version 1.0.
// (See http://www.boost.org/LICENSE_1_0.txt)
//
// Original link http://panthema.net/2007/0328-ZLibString.html

#include "compresser.h"


/** Compress a STL string using zlib with given compression level and return
* the binary data. */
std::string compress(const std::string& str,
int compressionlevel/* = Z_BEST_COMPRESSION*/)
{
z_stream zs; // z_stream is zlib's control structure
memset(&zs, 0, sizeof(zs));

if (deflateInit(&zs, compressionlevel) != Z_OK)
throw(std::runtime_error("deflateInit failed while compressing."));

zs.next_in = (Bytef*)str.data();
zs.avail_in = str.size(); // set the z_stream's input

int ret;
char outbuffer[4096];
std::string outstring;

// retrieve the compressed bytes blockwise
do {
zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
zs.avail_out = sizeof(outbuffer);

ret = deflate(&zs, Z_FINISH);

if (outstring.size() < zs.total_out) {
// append the block to the output string
outstring.append(outbuffer,
zs.total_out - outstring.size());
}
} while (ret == Z_OK);

deflateEnd(&zs);

if (ret != Z_STREAM_END) { // an error occurred that was not EOF
std::ostringstream oss;
oss << "Exception during zlib compression: (" << ret << ") " << zs.msg;
throw(std::runtime_error(oss.str()));
}

return outstring;
}

/** Decompress an STL string using zlib and return the original data. */
std::string decompress(const std::string& str)
{
z_stream zs; // z_stream is zlib's control structure
memset(&zs, 0, sizeof(zs));

if (inflateInit(&zs) != Z_OK)
throw(std::runtime_error("inflateInit failed while decompressing."));

zs.next_in = (Bytef*)str.data();
zs.avail_in = str.size();

int ret;
char outbuffer[4096];
std::string outstring;

// get the decompressed bytes blockwise using repeated calls to inflate
do {
zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
zs.avail_out = sizeof(outbuffer);

ret = inflate(&zs, 0);

if (outstring.size() < zs.total_out) {
outstring.append(outbuffer,
zs.total_out - outstring.size());
}

} while (ret == Z_OK);

inflateEnd(&zs);

if (ret != Z_STREAM_END) { // an error occurred that was not EOF
std::ostringstream oss;
oss << "Exception during zlib decompression: (" << ret << ") "
<< zs.msg;
throw(std::runtime_error(oss.str()));
}

return outstring;
}
Loading