Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve subprocess CRDS calls for easier debugging #140

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Improve subprocess CRDS calls for easier debugging #140

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d97ab85
Improve subprocess CRDS calls for easier debugging
mcara Jun 23, 2020
ba41308
Shell=False
mcara Jun 23, 2020
c1e88de
shell=False by default
mcara Jun 23, 2020
f5c4a70
use crds API
mcara Jun 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions notebooks/DrizzlePac/Initialization/Initialization.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,12 @@
"metadata": {},
"outputs": [],
"source": [
"subprocess.check_output('crds bestrefs --files ib2j02n5q_flc.fits --sync-references=1 --update-bestrefs', shell=True, stderr=subprocess.DEVNULL)"
"stdout = subprocess.check_output(\n",
" 'crds bestrefs --files ib2j02n5q_flc.fits --sync-references=1 --update-bestrefs',\n",
" shell=True,\n",
" stderr=subprocess.STDOUT\n",
")\n",
"print(stdout.decode())"
]
},
{
Expand Down Expand Up @@ -329,7 +334,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.6"
"version": "3.7.6"
}
},
"nbformat": 4,
Expand Down
12 changes: 9 additions & 3 deletions notebooks/DrizzlePac/drizzle_wfpc2/drizzle_wfpc2.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,12 @@
"os.environ['CRDS_SERVER_URL'] = 'https://hst-crds.stsci.edu'\n",
"os.environ['CRDS_PATH'] = os.path.abspath(os.path.join('.', 'reference_files'))\n",
"\n",
"subprocess.check_output('crds bestrefs --files ua0605*_c0m.fits --sync-references=1 --update-bestrefs', shell=True, stderr=subprocess.DEVNULL)\n",
"stdout = subprocess.check_output(\n",
" 'crds bestrefs --files ua0605*_c0m.fits --sync-references=1 --update-bestrefs',\n",
" shell=True,\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to do this without shell=True?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't CRDS written in Python? Can't we call its Python API directly?

Copy link
Member Author

@mcara mcara Jun 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC: CRDS experts @eslavich @jaytmiller

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pllim May I ask: what is the issue with shell=True?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.python.org/3/library/subprocess.html#security-considerations

Copy link
Member Author

@mcara mcara Jun 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of academic curiosity, is there benefit in sorting the glob results first in this usage?

I do not see why would it matter here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, let's say if the bestref is usually named in a way that would appear last in listing, and the function do a simple linear search, then sorted(..., reverse=True) would make the search faster? Anyway, I have no idea how it works, so I am just thinking aloud. And of course, it is probably out of topic as far as this PR is concerned.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order shouldn't matter -- the glob result is a list of datasets, and we have to make the best references determination for each of them.

Incidentally, I don't think shell=True is a security concern in this case. We wouldn't want to use it in web server code (for example) that interpolates user input into a command, but here the user is already in an environment where they can execute arbitrary shell commands so it's no more unsafe than a notebook in general.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not as much as a safety in this case, as to promote best practices. What if someone copy-paste what is here into server code? 😉

And given that it is possible to call Python API directly, there is really no reason to have shell=True at all in the first place.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

p.s. Yeah, just ignore my comment about sorting. Sorry for the noise!

" stderr=subprocess.STDOUT\n",
")\n",
"print(stdout.decode())\n",
"\n",
"os.environ['uref'] = os.path.abspath(os.path.join('.', 'reference_files', 'references', 'hst', 'wfpc2')) + os.path.sep"
]
Expand Down Expand Up @@ -424,7 +429,8 @@
"# About this Notebook\n",
"\n",
" Author: M. Cara, STScI Data Analysis Tools Branch\n",
" Updated: December 14, 2018"
" Updated: December 14, 2018\n",
" Updated: June 23, 2018"
]
}
],
Expand All @@ -444,7 +450,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.6"
"version": "3.7.6"
}
},
"nbformat": 4,
Expand Down
10 changes: 7 additions & 3 deletions ...ks/DrizzlePac/using_updated_astrometry_solutions/using_updated_astrometry_solutions.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,12 @@
"source": [
"for row in filtered_products:\n",
" if row['productSubGroupDescription'] == 'FLC':\n",
" subprocess.check_output('crds bestrefs --files {}_flc.fits --sync-references=1 --update-bestrefs'.format(row['obs_id']), \\\n",
" shell=True, stderr=subprocess.DEVNULL)\n"
" stdout = subprocess.check_output(\n",
" 'crds bestrefs --files {}_flc.fits --sync-references=1 --update-bestrefs'.format(row['obs_id']),\n",
" shell=True,\n",
" stderr=subprocess.STDOUT\n",
" )\n",
" print(stdout.decode())"
]
},
{
Expand Down Expand Up @@ -764,7 +768,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.0"
"version": "3.7.6"
}
},
"nbformat": 4,
Expand Down