Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make process resumable and enhance IP classifier #54

Merged
merged 56 commits into from
Aug 21, 2024
Merged
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
ed19f6e
refactor for simplicity and resumability
Aug 20, 2024
f5076e7
refactor for simplicity and resumability
Aug 20, 2024
6a4853b
refactor for simplicity and resumability
Aug 20, 2024
fb08027
refactor for simplicity and resumability
Aug 20, 2024
caf030c
fixes; adapting tests
Aug 20, 2024
b7a83f5
fix
Aug 20, 2024
95e4bda
fix
Aug 20, 2024
fb48dbb
fix
Aug 20, 2024
93bfd3f
fix
Aug 20, 2024
661e1a7
fix
Aug 20, 2024
dc8c6d9
fix
Aug 20, 2024
38ec7e6
fix
Aug 20, 2024
2e863b6
fix
Aug 20, 2024
60b6ae5
adjusting tests
Aug 21, 2024
46f3375
debug reduction tests
Aug 21, 2024
3bfba48
get binning mostly there
Aug 21, 2024
442b5e3
fix
Aug 21, 2024
1f095e6
debug binning
Aug 21, 2024
ae3b352
debug binning
Aug 21, 2024
e6432be
update mapping; debug year span
Aug 21, 2024
d3b4c80
debug skip protocol
Aug 21, 2024
8ad2c88
improve default folder creation
Aug 21, 2024
cb5e7ff
improve default folder creation
Aug 21, 2024
08c1786
improve default folder creation
Aug 21, 2024
e6c68fd
improve default folder creation
Aug 21, 2024
4518145
add CLI for binning
Aug 21, 2024
4fcbe88
update argument name; enhance readme
Aug 21, 2024
344b09b
add binning tracking
Aug 21, 2024
d5ed6da
reformat
Aug 21, 2024
bb16fd4
reformat
Aug 21, 2024
ae5b057
reformat
Aug 21, 2024
0000b2a
adjust name
Aug 21, 2024
4bc5924
add progress bar to binning
Aug 21, 2024
c442696
add a file buffer to binning
Aug 21, 2024
d2830da
add CIDR regions
Aug 21, 2024
297e4ff
make argument simpler
Aug 21, 2024
8dbd722
update readme
Aug 21, 2024
b6f5f97
add extra progress bar for binning
Aug 21, 2024
5925438
skip ipv6 gcp
Aug 21, 2024
5af93c5
add services to cache
Aug 21, 2024
a43572a
add extra test case and debug
Aug 21, 2024
89c0bd5
add helper function for in cidr
Aug 21, 2024
b3a9bcc
add helper function for in cidr
Aug 21, 2024
041abb9
add extra cache
Aug 21, 2024
40cb4e9
add extra cache
Aug 21, 2024
89022a4
add extra cache
Aug 21, 2024
1604016
add extra cache
Aug 21, 2024
74df297
try this again
Aug 21, 2024
85ff066
fix skip condition in reducer
Aug 21, 2024
1694ac4
remove other helper
Aug 21, 2024
e170cd2
remove other helper
Aug 21, 2024
121a974
debugging non-skip
Aug 21, 2024
9730cab
debugging non-skip
Aug 21, 2024
c6f712d
debugging non-skip
Aug 21, 2024
9ed0a5b
debugging non-skip
Aug 21, 2024
75579e9
Merge branch 'main' into make_resumable
CodyCBakerPhD Aug 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix
CodyCBakerPhD committed Aug 20, 2024
commit 95e4bda0d3dc417b394c600b6b16eb5f4957acd2
10 changes: 10 additions & 0 deletions src/dandi_s3_log_parser/_command_line_interface.py
Original file line number Diff line number Diff line change
@@ -44,6 +44,13 @@
type=click.IntRange(min=1), # Bare minimum of 1 MB
default=1_000, # 1 GB recommended
)
@click.option(
"--excluded_years",
help="A comma-separated list of years to exclude from parsing.",
required=False,
type=str,
default=None,
)
@click.option(
"--excluded_ips",
help="A comma-separated list of IP addresses to exclude from parsing.",
@@ -56,8 +63,10 @@ def _reduce_all_dandi_raw_s3_logs_cli(
reduced_s3_logs_folder_path: str,
maximum_number_of_workers: int,
maximum_buffer_size_in_mb: int,
excluded_years: str | None,
excluded_ips: str | None,
) -> None:
split_excluded_years = excluded_years.split(",") if excluded_years is not None else list()
split_excluded_ips = excluded_ips.split(",") if excluded_ips is not None else list()
handled_excluded_ips = collections.defaultdict(bool) if len(split_excluded_ips) != 0 else None
for excluded_ip in split_excluded_ips:
@@ -69,6 +78,7 @@ def _reduce_all_dandi_raw_s3_logs_cli(
reduced_s3_logs_folder_path=reduced_s3_logs_folder_path,
maximum_number_of_workers=maximum_number_of_workers,
maximum_buffer_size_in_bytes=maximum_buffer_size_in_bytes,
excluded_years=split_excluded_years,
excluded_ips=handled_excluded_ips,
)

2 changes: 1 addition & 1 deletion src/dandi_s3_log_parser/_dandi_s3_log_file_reducer.py
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@ def reduce_all_dandi_raw_s3_logs(

Assumes the following folder structure...

|- <base_raw_s3_log_folder_path>
|- <raw_s3_log_folder_path>
|-- 2019 (year)
|--- 01 (month)
|---- 01.log (day)