Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use fsspec #53

Merged
merged 2 commits into from
Sep 24, 2024
Merged

use fsspec #53

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b3c3ab7
use fsspec
ttdu Sep 24, 2024
7c69869
Merge branch 'main' into new-cloud-access
ttdu Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 21 additions & 33 deletions content/notebooks/data_discovery_and_access/data_discovery_and_access.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,17 +138,24 @@
"metadata": {},
"outputs": [],
"source": [
"obs = Observations.query_criteria(obs_collection='HST',\n",
" filters='F160W',\n",
" instrument_name='WFC3/IR',\n",
" proposal_id=['11360'],\n",
" dataRights='PUBLIC',\n",
" )\n",
"# query MAST for matching observations\n",
"obs = Observations.query_criteria(obs_collection='JWST',\n",
" filters='F444W',\n",
" instrument_name='NIRCAM/IMAGE',\n",
" proposal_id=['1783'],\n",
" dataRights='PUBLIC')\n",
"# get the list of products (files)\n",
"products = Observations.get_product_list(obs)\n",
"\n",
"# filter the products\n",
"filtered = Observations.filter_products(products,\n",
" calib_level=[3], productType=['SCIENCE'], dataproduct_type=['image'], project=['CALWF3'])\n",
"print('Filtered data products:\\n', filtered, '\\n')"
" productSubGroupDescription='RATE')\n",
"print('Filtered data products:\\n', filtered, '\\n')\n",
"\n",
"# filter for just one product\n",
"single = Observations.filter_products(filtered,\n",
" obsID='87766440')\n",
"print('Single data product:\\n', single, '\\n')"
]
},
{
Expand Down Expand Up @@ -189,23 +196,7 @@
"metadata": {},
"source": [
"### Streaming files directly into memory\n",
"Here, we will use `s3fs` to directly access the data stored in the AWS S3 servers. Typically to access data from AWS, authentication or log-in credentials need to be passed into `S3FileSystem`. This is primarily used to access private S3 servers. However to access publicly available data, `s3fs` can be used in \"anonymous\" mode by setting `anon=True`. As the data on MAST is publicly available, we will use the anonymous mode here."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fs = s3fs.S3FileSystem(anon=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because the URI points to a FITS file, we can use `astropy` to access the information in the file."
"Here, we will use `fsspec` to directly access the data stored in the AWS S3 servers. Because the URI points to a FITS file, we can use `fits.open` to access the information in the file."
]
},
{
Expand All @@ -214,14 +205,11 @@
"metadata": {},
"outputs": [],
"source": [
"# Open the file in AWS: 'F' is the S3 file\n",
"import numpy as np\n",
"with fs.open(uri, 'rb') as f:\n",
" # Now actually read in the FITS file \n",
" with fits.open(f, 'readonly') as HDUlist:\n",
" HDUlist.info()\n",
" sci = HDUlist[1].data\n",
"print(type(sci))"
"with fits.open(uri, 'readonly', fsspec_kwargs={\"anon\":True}) as HDUlist:\n",
" HDUlist.info()\n",
" sci = HDUlist[1].data\n",
" \n",
"type(sci)"
]
},
{
Expand Down
Loading