From abe8643c2513a9ea56f3d382cff7232914c775d3 Mon Sep 17 00:00:00 2001 From: olas Date: Wed, 10 Jan 2024 09:20:21 +0000 Subject: [PATCH] deploy: 46a7b9fb2e3e2c2e6389be4c1e6671162874557d --- about/index.html | 2 +- blog/10th-profiling-symposium-2020/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- blog/alex-intropres-2017/index.html | 2 +- blog/amelie-joins-team/index.html | 2 +- blog/andreina-new-postdoc/index.html | 2 +- .../index.html | 2 +- blog/bbq2017/index.html | 2 +- blog/bbq2018/index.html | 2 +- blog/bigdata-ssf-grant-2017/index.html | 2 +- blog/blodomloppet2017/index.html | 2 +- blog/cfp-jcheminf-2017/index.html | 2 +- blog/charme-bigdata-2017/index.html | 2 +- blog/christa-mobility-granted/index.html | 2 +- blog/cim2018/index.html | 2 +- blog/cim2021/index.html | 2 +- blog/cloud-beer-2017/index.html | 2 +- blog/copa2017/index.html | 2 +- blog/copa2018-niharika/index.html | 2 +- blog/cp-design-microplates/index.html | 2 +- blog/data-engineer-position-2020/index.html | 2 +- blog/david-joins-team/index.html | 2 +- blog/denbi-conf-2017/index.html | 2 +- blog/dissertation-samuel/index.html | 2 +- blog/ecpc-escience-cancer-workflow/index.html | 2 +- blog/elrig-2022/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- blog/haste-azvisit-2017/index.html | 2 +- .../index.html | 2 +- blog/iccws2017/index.html | 2 +- blog/index.html | 2 +- blog/interview-spjuth-faculty/index.html | 2 +- blog/jon-devfest-siberia/index.html | 2 +- blog/jordi-joins-team/index.html | 2 +- blog/khinsen-pres-2018/index.html | 2 +- blog/laeeq-phd-defence/index.html | 2 +- blog/ldsv2018/index.html | 2 +- blog/marco-phd-defence/index.html | 2 +- blog/marco-spark-summit/index.html | 2 +- blog/neic-glenna-final-meeting/index.html | 2 +- blog/ola-biomedit-video/index.html | 2 +- blog/ola-spjuth-professor/index.html | 2 +- blog/olas-pres-dis2019/index.html | 2 +- blog/olas-pres-slaseurope2019/index.html | 2 +- blog/open-positions-cbcs-node/index.html | 2 +- .../index.html | 2 +- blog/orn-workshop-2017/index.html | 2 +- blog/papers-accepted-copa2019/index.html | 2 +- blog/phdposition2019-cellbio/index.html | 2 +- blog/phenomenal-glenna2/index.html | 2 +- .../index.html | 2 +- blog/phil-starting-2017/index.html | 2 +- .../index.html | 2 +- blog/postdoc-haste-2017/index.html | 2 +- blog/postdoc-pos-ai-decisions-2020/index.html | 2 +- blog/postdoc-pos-ai-haste-2020/index.html | 2 +- .../index.html | 2 +- blog/presenting-at-copa2019/index.html | 2 +- .../index.html | 2 +- blog/recruitment-phd-haste/index.html | 2 +- blog/recruitment-phd-lecturer-2019/index.html | 2 +- blog/retreat2019/index.html | 2 +- blog/roland-pres-ptgs/index.html | 2 +- blog/saml-gostockholm2018/index.html | 2 +- blog/samuel-phd-defence/index.html | 2 +- blog/scilife-covid19-funded/index.html | 2 +- .../index.html | 2 +- blog/spark-vs/index.html | 2 +- .../index.html | 2 +- blog/tryggve2kickoff/index.html | 2 +- .../index.html | 2 +- blog/workshop-halle-2017/index.html | 2 +- categories/index.html | 2 +- education/index.html | 2 +- home/index.html | 2 +- index.html | 16 +-- index.xml | 9 ++ infrastructure/index.html | 2 +- people/abir/index.html | 2 +- people/akshai/index.html | 2 +- people/alex/index.html | 2 +- people/amelie/index.html | 2 +- people/anders/index.html | 2 +- people/andreina/index.html | 2 +- people/anton/index.html | 2 +- people/arvid/index.html | 2 +- people/ash/index.html | 2 +- people/axel/index.html | 2 +- people/ayildirim/index.html | 2 +- people/benjamin/index.html | 2 +- people/christa/index.html | 2 +- people/dahlo/index.html | 2 +- people/dalia/index.html | 2 +- people/dan/index.html | 2 +- people/daniel/index.html | 2 +- people/david/index.html | 2 +- people/davidd/index.html | 2 +- people/ebba/index.html | 2 +- people/erik_p/index.html | 2 +- people/ernst/index.html | 2 +- people/gokce1/index.html | 2 +- people/index.html | 2 +- people/jon/index.html | 2 +- people/jonalv/index.html | 2 +- people/jonne/index.html | 2 +- people/jordi/index.html | 2 +- people/juaninda/index.html | 2 +- people/laeeq/index.html | 2 +- people/laura/index.html | 2 +- people/malin/index.html | 2 +- people/marco/index.html | 2 +- people/maris/index.html | 2 +- people/martin/index.html | 2 +- people/matteo/index.html | 2 +- people/morgan/index.html | 2 +- people/niharika/index.html | 2 +- people/nima/index.html | 2 +- people/olas/index.html | 2 +- people/oliver/index.html | 2 +- people/ovidiu/index.html | 2 +- people/paschalis/index.html | 2 +- people/patrik/index.html | 2 +- people/phil/index.html | 2 +- people/polina/index.html | 2 +- people/rikard/index.html | 2 +- people/saml/index.html | 2 +- people/sebastian/index.html | 2 +- people/star/index.html | 2 +- people/steph/index.html | 2 +- people/tanya/index.html | 2 +- people/ulf/index.html | 2 +- people/valyo/index.html | 2 +- people/victor/index.html | 2 +- people/wes/index.html | 2 +- people/ximeng/index.html | 2 +- people/xinyi/index.html | 2 +- poster/2008-bioclipse2/index.html | 2 +- poster/2013-ecpc/index.html | 2 +- poster/2015-cml/index.html | 2 +- poster/2015-cp-spark/index.html | 2 +- poster/2015-mlindd/index.html | 2 +- poster/2015-sciluigi/index.html | 2 +- poster/2015-screening-spark/index.html | 2 +- poster/2016-bridgedb/index.html | 2 +- poster/2016-easymapreduce/index.html | 2 +- poster/2016-scipipe/index.html | 2 +- poster/2018-infrastructure/index.html | 2 +- poster/2018-mare/index.html | 2 +- poster/2018-metabolic-predictions/index.html | 2 +- poster/2018-transfer-cnn/index.html | 2 +- poster/2019-cloud/index.html | 2 +- poster/2019-dynamic-model/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- poster/index.html | 2 +- presentation/2015-agile-ml-ebi/index.html | 2 +- presentation/2015-ecpc/index.html | 2 +- .../2015-sensitive-data-ebi/index.html | 2 +- presentation/2016-bigdata-medicine/index.html | 2 +- .../2016-cont-modeling-icpb/index.html | 2 +- presentation/2016-opentox/index.html | 2 +- presentation/2016-smwcon-rdfio/index.html | 2 +- .../2017-big-data-training-school/index.html | 2 +- presentation/2017-cloudbeer/index.html | 2 +- .../2017-copa-cross-venn-abers/index.html | 2 +- presentation/2017-denbi/index.html | 2 +- .../2017-pachyderm-big-data/index.html | 2 +- presentation/2017-sparksummit/index.html | 2 +- presentation/2018-copa/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../2018-vendor-agnostic-spark/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- presentation/index.html | 2 +- .../ai-confidence-drugdiscovery/index.html | 2 +- project/autonomous-phenomics/index.html | 2 +- project/haste/index.html | 2 +- project/index.html | 2 +- .../index.html | 2 +- project/phenptypic-drug-discovery/index.html | 2 +- project/rdfio/index.html | 2 +- .../2006-the-lcb-data-warehouse/index.html | 2 +- publication/2007-bioclipse/index.html | 2 +- publication/2008-c1c2/index.html | 2 +- publication/2008-pcm-hiv-protease/index.html | 2 +- publication/2009-bioclipse2/index.html | 2 +- publication/2009-xmpp-iodata/index.html | 2 +- publication/2010-escience-bayes/index.html | 2 +- publication/2010-metaprint2d/index.html | 2 +- .../index.html | 2 +- publication/2010-qsar-ml/index.html | 2 +- publication/2011-bioclipse-ds/index.html | 2 +- publication/2011-bioclipse-opentox/index.html | 2 +- publication/2011-brunn/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2011-hiv-drc/index.html | 2 +- .../index.html | 2 +- publication/2011-rdf-cheminf-pcm/index.html | 2 +- publication/2011-sail/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2012-bioclipse-opendd/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../2013-inchi-cdk-bioclipse/index.html | 2 +- .../index.html | 2 +- publication/2013-mapreduce-vs/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2013-quantmap/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2014-dils-sail/index.html | 2 +- publication/2014-htseq-hadoop/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../2015-benchmarking-data-set/index.html | 2 +- publication/2015-bioimg/index.html | 2 +- publication/2015-cp-interpret/index.html | 2 +- .../index.html | 2 +- publication/2015-modeling-cloud/index.html | 2 +- .../index.html | 2 +- publication/2015-spark-cp/index.html | 2 +- .../index.html | 2 +- publication/2015-wf-bioinfo/index.html | 2 +- .../index.html | 2 +- publication/2016-large-scale-svm/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2016-sciluigi/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../2017-docking-of-macrocycles/index.html | 2 +- .../2017-escience-cancer-screening/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2017-rdfio/index.html | 2 +- .../2017-silver-resistance-genes/index.html | 2 +- .../index.html | 2 +- .../2017-virtual-screening-spark/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2018-galaxy-kubernetes/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2018-ptp/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2019-scipipe/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2021-scconnect/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 118 ++++++++++++++++++ .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2023-bf-moa-plos/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/2023-nam-edc-parc/index.html | 2 +- .../index.html | 2 +- .../index.html | 2 +- publication/index.html | 36 +++++- publication/index.xml | 9 ++ rightcol-front/index.html | 2 +- sitemap.xml | 5 + tags/index.html | 2 +- tool/aros/index.html | 2 +- tool/bioclipse/index.html | 2 +- tool/easymapreduce/index.html | 2 +- tool/flowbase/index.html | 2 +- tool/index.html | 2 +- tool/kubenow/index.html | 2 +- tool/metaprint2d/index.html | 2 +- tool/modelingweb/index.html | 2 +- tool/ndcp/index.html | 2 +- tool/rdfio/index.html | 2 +- tool/sciluigi/index.html | 2 +- tool/scipipe/index.html | 2 +- tool/sparkcp/index.html | 2 +- tool/sparknow/index.html | 2 +- tool/xmetdb/index.html | 2 +- 359 files changed, 537 insertions(+), 362 deletions(-) create mode 100644 publication/2022-management-scientific-dataset-hierarchical-storage-reinforcement-learning/index.html diff --git a/about/index.html b/about/index.html index 4dd623ab2..d45118750 100644 --- a/about/index.html +++ b/about/index.html @@ -7,7 +7,7 @@ - + + + + + + + + + + +
+
+ +
+
+ +
+
+ +

Management of Scientific Datasets in Hierarchical Storage Using Reinforcement Learning

+

← Back to publications

+ +

Published: 2024-01-01

+ +

Formatted citation

+

+ Zhang T, Gupta A, Rodríguez MAF, Spjuth O, Hellander A and Toor S.. + Management of Scientific Datasets in Hierarchical Storage Using Reinforcement Learning. +
Expert Systems With Applications. + 237, 121443 (2024). + DOI: 10.1016/j.eswa.2023.121443 + +

+

Abstract

+

In many areas of data-driven science, large datasets are generated where the individual data objects are images, matrices, or otherwise have a clear structure. However, these objects can be information-sparse, and a challenge is to efficiently find and work with the most interesting data as early as possible in an analysis pipeline. We have recently proposed a new model for big data management where the internal structure and information of the data are associated with each data object (as opposed to simple metadata). There is then an opportunity for comprehensive data management solutions to account for data-specific internal structure as well as access patterns. In this article, we explore this idea together with our recently proposed hierarchical storage management framework that uses reinforcement learning (RL) for autonomous and dynamic data placement in different tiers in a storage hierarchy. Our case-study is based on four scientific datasets: Protein translocation microscopy images, Airfoil angle of attack meshes, 1000 Genomes sequences, and Phenotypic screening images. The presented results highlight that our framework is optimal and can quickly adapt to new data access requirements. It overall reduces the data processing time, and the proposed autonomous data placement is superior compared to any static or semi-static data placement policies.

+ +
+
+ + + + + + + + + diff --git a/publication/2022-merging-bioactivity-predictions-cell-morphology-chemical-fingerprint/index.html b/publication/2022-merging-bioactivity-predictions-cell-morphology-chemical-fingerprint/index.html index f67f0b00e..b294e1995 100644 --- a/publication/2022-merging-bioactivity-predictions-cell-morphology-chemical-fingerprint/index.html +++ b/publication/2022-merging-bioactivity-predictions-cell-morphology-chemical-fingerprint/index.html @@ -7,7 +7,7 @@ - +