Initial code of the caching repository #17

eileen-kuehn · 2020-12-09T18:05:31Z

This PR adds the initial code from @tfesenbecker.
Current state of documentation can be found in a hidden build of readthedocs.

Closes #11, closes #15, closes #19, closes #22.
Also closes #8.

…tfile source

codecov · 2021-05-27T14:28:41Z

Codecov Report

❗ No coverage uploaded for pull request base (master@423f9ae). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #17   +/-   ##
=========================================
  Coverage          ?   79.24%           
=========================================
  Files             ?       33           
  Lines             ?     1797           
  Branches          ?      195           
=========================================
  Hits              ?     1424           
  Misses            ?      316           
  Partials          ?       57

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 423f9ae...cc9f45f. Read the comment docs.

* [CLI] Implement CLI-extension for caching * [Project] Configuring CLI simulate script installer * [CLI] Enable multiple storage_files * [CLI] Simplify by getting rid of subcommands Co-authored-by: Eileen Kuehn <[email protected]> Co-authored-by: Max Fischer <[email protected]>

eileen-kuehn · 2021-06-09T20:33:36Z

@maxfischer2781, this is slowly getting really huge... I will mark it as Ready for review so that we can work through the first lines 😸

maxfischer2781

Honestly, I didn't check every line. But it looks generally good to me, tests are passing (locally, travis seems to be dead) and there's some good work in it.

So, let's go for it – if we've missed anything, we can fix it later on.

eileen-kuehn added 30 commits May 9, 2020 16:41

changed assignment of connections a bit

13e9e94

removed creation of connection module from test as it is not required

e1deb6c

converted storage and file sizes to bytes

e5a54c7

corrected access to numberofaccesses

ccdebfc

changed signature of StoredFile and adapted in IO operations

19c2368

improved storage

0cf879a

minimum required usim version set to 0.4.3

560d551

renamed remove_from_storage and add_to_storage to remove and add

4955813

made free_space a property of storage

bb0a99b

removed method find_file from storage

bd15741

added todo

26e5b52

ignored B006 for flake8

06639e4

if file is available on storage, transfer now receives correct size

0a885e6

fixed position of noqa

1f782dc

made determine_inputfile_source private to connection

62579c2

renamed transfer_inputfiles to transfer_files

5282e1b

removed queue from file lookup in storage and improved determine inpu…

4a0cdc3

…tfile source

improved stream file in connection

535ac0c

introduced HitrateStorage that transfers data based on a cache hitrate

c3a6b22

fixed position of noqa

b73b21f

removed cachehitrate from connection

3b4bd41

connection now sets reference to remote_connection for storage

8162167

storage objects are now created based on specified cache hit rate

18238e4

adapted usage of caching for jobs

2df4769

introduced calculation efficiency to job

24a0240

introduced calculation efficiency for jobs to cli

f984017

added more type hints for job

3b011eb

removed initialisation of connection

9cc625a

moved caching related monitoring to extra file

08d2907

each simulation run now can be identified

e9e6e07

eileen-kuehn added 15 commits May 6, 2021 16:49

hitratebased requested file now subclasses requested file

052984d

hitrate 0 if not defined

b0e8dc4

changed order in which a file is stored on cache

7638ab5

adapted typehints for filesizes and time information

67c8d6d

fixed implementation of delete_oldest

c7e1025

adapted implementation of delete_oldest_few_used

77e3f56

files are now required for storage elements

211a8d9

fixed typehints

14b2812

calculation_efficiency not optional anymore

dc996d3

improved calculation of probabilities

82a3afc

drone is optional when scheduling

32634ac

fixed type

a3f7e0f

improved creation of pools

36233bc

fixed loop error

5eb03a4

made flake happy

3927f67

eileen-kuehn marked this pull request as ready for review June 9, 2021 20:33

eileen-kuehn requested a review from maxfischer2781 June 9, 2021 20:33

eileen-kuehn mentioned this pull request Jun 21, 2021

Chosing the caching scenario via CLI #8

Closed

eileen-kuehn added 2 commits June 21, 2021 18:53

checking all caches for file transfers, closes #11

b54c99c

added static tests for gh actions

0aa3e63

maxfischer2781 previously approved these changes Jun 21, 2021

View reviewed changes

moved unit tests to gh actions

cc9f45f

eileen-kuehn dismissed maxfischer2781’s stale review via cc9f45f June 21, 2021 17:52

eileen-kuehn requested a review from maxfischer2781 June 21, 2021 17:54

maxfischer2781 approved these changes Jun 22, 2021

View reviewed changes

eileen-kuehn merged commit 8f4f614 into master Jun 22, 2021

eileen-kuehn deleted the add/initial-code branch June 22, 2021 07:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial code of the caching repository #17

Initial code of the caching repository #17

eileen-kuehn commented Dec 9, 2020 •

edited

Loading

codecov bot commented May 27, 2021 •

edited

Loading

eileen-kuehn commented Jun 9, 2021

maxfischer2781 left a comment

Initial code of the caching repository #17

Initial code of the caching repository #17

Conversation

eileen-kuehn commented Dec 9, 2020 • edited Loading

codecov bot commented May 27, 2021 • edited Loading

Codecov Report

eileen-kuehn commented Jun 9, 2021

maxfischer2781 left a comment

Choose a reason for hiding this comment

eileen-kuehn commented Dec 9, 2020 •

edited

Loading

codecov bot commented May 27, 2021 •

edited

Loading