Skip to content

Commit

Permalink
Update 2022-management-scientific-dataset-hierarchical-storage-reinfo…
Browse files Browse the repository at this point in the history
…rcement-learning.md
  • Loading branch information
olas committed Nov 14, 2023
1 parent ae745b4 commit 882e8de
Showing 1 changed file with 5 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ bibtex_type = "article"
author="Zhang T, Gupta A, Rodríguez MAF, Spjuth O, Hellander A and Toor S."
title="Management of Scientific Datasets in Hierarchical Storage Using Reinforcement Learning"
journal="Expert Systems With Applications"
year="2023"
year="2024"
date="2023-09-05T00:00:00+02:00"
volume="Accepted"
number=""
volume="237"
number="121443"
preprint = false
pages=""
abstract=""
doi=""
abstract="In many areas of data-driven science, large datasets are generated where the individual data objects are images, matrices, or otherwise have a clear structure. However, these objects can be information-sparse, and a challenge is to efficiently find and work with the most interesting data as early as possible in an analysis pipeline. We have recently proposed a new model for big data management where the internal structure and information of the data are associated with each data object (as opposed to simple metadata). There is then an opportunity for comprehensive data management solutions to account for data-specific internal structure as well as access patterns. In this article, we explore this idea together with our recently proposed hierarchical storage management framework that uses reinforcement learning (RL) for autonomous and dynamic data placement in different tiers in a storage hierarchy. Our case-study is based on four scientific datasets: Protein translocation microscopy images, Airfoil angle of attack meshes, 1000 Genomes sequences, and Phenotypic screening images. The presented results highlight that our framework is optimal and can quickly adapt to new data access requirements. It overall reduces the data processing time, and the proposed autonomous data placement is superior compared to any static or semi-static data placement policies."
doi="10.1016/j.eswa.2023.121443"
url_html=""
+++

0 comments on commit 882e8de

Please sign in to comment.