Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds new handle to object stores that support S3 API #107

Draft
wants to merge 83 commits into
base: develop
Choose a base branch
from
Draft

Adds new handle to object stores that support S3 API #107

wants to merge 83 commits into from

Conversation

mcakircali
Copy link
Contributor

Adds S3 support via AWS SDK C++

mcakircali and others added 30 commits January 31, 2024 17:07
@mcakircali
feat(S3): initial commit for AWS S3 support
c70f2ce
@mcakircali
refactor(S3): removed UPtr and SPtr
c182574
@mcakircali
feat(S3)!: adds Config, Session, Client, Context
e92e14e
@mcakircali
feat(S3): add create,delete,list for bucket and put,list object
5fbf932
@mcakircali
feat(S3): adds delete,list object
4aaec56
@mcakircali
fix(S3): throw on unkown S3 client type
eda4bfd
@mcakircali
feat(S3): adds credentials to S3Config
3fb4e4d
@mcakircali
refactor(S3): throw instead of return boolean
408d9c2
@mcakircali
chore(S3): remove advanced attribute from AWS_S3 cmake option
028a4ee
@mcakircali
style(S3): fix todo style comment
7423b5c
@mcakircali
refactor(S3): namedspace -> unnamed namespace
c3e93a2
@mcakircali
refactor(S3): cleanup S3Session
1aab48f
@mcakircali
fix(S3): add missing pragma once
d775bff
@mcakircali
refactor(S3): cleanup S3Handle
620fd18
@mcakircali
refactor(S3): more cleanup for S3Handle, S3Name
de556ff
@mcakircali
fix(S3): aws error parsing
ed4ed4b
@mcakircali
feat(S3): rework credentials
7eaa9be
@simondsmart
Add some URI infrastructure, and tweak some tests
545f8c6
@mcakircali
refactor(S3): rework S3Config
e8bcf92
@mcakircali
feat(S3): start S3Name and S3URIManager
c84f046
@mcakircali
Merge remote-tracking branch 'refs/remotes/origin/feature/backend-s3'…
a7bdb15 
… into feature/backend-s3
@mcakircali
fix(S3): authority must be true
563679e
@mcakircali
feat(Endpoint): construct from a URI
2003a5c
@mcakircali
fix(S3): client missing error msg
ff342d1
@mcakircali
feat(S3): S3Name: parse bucket/object, exists, size
4fbcea1
@mcakircali
feat(S3): add bucket/object exists, object size
929c0f2
@mcakircali
refactor(S3): session cleanup
611742a
@mcakircali
feat(S3): adds open and size to S3Handle
894bbae
@mcakircali
fix(S3): adapt s3handle test
6a95ecb
@mcakircali
refactor(S3): cleanup and copyright fix
140d75d
@mcakircali
fix(S3): removed unnecessary stuff from S3Handle
6d01bf8
@mcakircali
fix(S3): write throws if object exists
34b8f2c
@mcakircali
feat(S3): added test for no bucket s3handle
64c15a0
@mcakircali
fix(S3): commented expensive code in openForWrite
2ffe322
@mcakircali
refactor(S3): removed eckit namespaces
e17e438
@mcakircali
refactor(S3)!: S3BucketName
ff3342e
@mcakircali
refactor(S3)!: S3ClientAWS throws
13a3d01
@mcakircali
refactor(S3)!: S3Exception
b4ae6a1
@mcakircali
feat(S3)!: added S3ObjectName
0c84e60
@mcakircali
fix(S3): memory leak S3Name
48797f6
@mcakircali
feat(S3): added S3Name accessors
f938234
@mcakircali
refactor(S3): various log debug
7a47f99
@mcakircali
fix(S3)!: S3Handle bug read/write return
227db9f
@mcakircali
feat(S3): added S3Handle tests, S3Bucket/Object accessors and change …
c07f52c 
…to ListObjects V2
@mcakircali
fix(S3): S3Client and IMDS bugs
9cb443b
@mcakircali
feat(S3): tests 1000 object writes
b4c8f7f
@mcakircali
feat(S3): write performance tests
b9e30d2
@mcakircali
feat(S3): improved perf 20% .. reuse S3Client
a9f0cd1
@mcakircali
fix(S3): removed S3Context
17fd771
@mcakircali
feat(S3): move uri to base
b481597
@FussyDuck
Copy link

FussyDuck commented Feb 21, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 63.04%. Comparing base (1e188aa) to head (7d76c42).
Report is 17 commits behind head on develop.

Files Patch % Lines
src/eckit/net/Endpoint.cc 0.00% 3 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop     #107      +/-   ##
===========================================
- Coverage    63.05%   63.04%   -0.02%     
===========================================
  Files          792      792              
  Lines        45418    45421       +3     
===========================================
- Hits         28638    28634       -4     
- Misses       16780    16787       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mcakircali mcakircali changed the title Feature/backend s3 Adds new handle to object stores that support S3 API Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simondsmart simondsmart Awaiting requested review from simondsmart

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mcakircali @FussyDuck @codecov-commenter @simondsmart @nicolau-manubens