-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Private extractor support #991
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested this with the corresponding pyclowder branch. Private extractor registers, only visible to the user.
Does the pyclowder branch need to be merged before this one? |
Sorry can you resolve conflicts, thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There might be an issue. I added some extractors with this branch running. I enabled them. When I go to the dataset page and click on the Analysis tab I get a blank page and errors in backend and frontend. Guessing it's not able to serialize the something in the new extractor json.
INFO: 127.0.0.1:65327 - "GET /api/v2/listeners?skip=0&limit=5&heartbeat_interval=0&category=&label=&alive_only=false&process=dataset&all=665df0aa6cdde48fda0fde9e&dataset_id=false HTTP/1.1" 422 Unprocessable Entity
Uncaught Error: Objects are not valid as a React child (found: object with keys {loc, msg, type}). If you meant to render a collection of children, use an array instead.
React 16
unstable_runWithPriority scheduler.development.js:468
React 4
Redux 4
s (index):3
middleware Redux
immutableStateInvariantMiddleware Immutable
dispatch Redux
fetchListenerCategories listeners.js:147
promise callback*then CancelablePromise.ts:81
fetchListenerCategories listeners.js:146
middleware Redux
immutableStateInvariantMiddleware Immutable
dispatch (index):6
listAvailableCategories Listeners.tsx:72
Listeners Listeners.tsx:101
React 5
unstable_runWithPriority scheduler.development.js:468
React 4
unstable_runWithPriority scheduler.development.js:468
React 17
tsx index.tsx:14
Webpack 3
react-dom.development.js:13231
The above error occurred in thecomponent:
p
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
Typography@http://localhost:3000/main.bundle.js:70947:87
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
DialogContentText@http://localhost:3000/main.bundle.js:49627:83
div
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
DialogContent@http://localhost:3000/main.bundle.js:49498:82
div
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
Paper@http://localhost:3000/main.bundle.js:60706:83
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
div
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
Transition@http://localhost:3000/main.bundle.js:238854:30
Fade@http://localhost:3000/main.bundle.js:50560:77
FocusTrap@http://localhost:3000/main.bundle.js:37082:7
div
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
Portal@http://localhost:3000/main.bundle.js:37815:7
Modal@http://localhost:3000/main.bundle.js:58736:82
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
Dialog@http://localhost:3000/main.bundle.js:48993:83
ActionModal@http://localhost:3000/main.bundle.js:124340:20
ErrorModal@http://localhost:3000/main.bundle.js:124383:19
main
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
div
./node_modules/@emotion/react/dist/emotion-element-43c6fea0.browser.esm.js/withEmotionCache/<@http://localhost:3000/main.bundle.js:13186:66
Box@http://localhost:3000/main.bundle.js:78283:72
PersistentDrawerLeft@http://localhost:3000/main.bundle.js:119552:74
Dataset@http://localhost:3000/main.bundle.js:121627:80
PrivateRoute@http://localhost:3000/main.bundle.js:151156:18
RenderedRoute@http://localhost:3000/main.bundle.js:237340:7
Routes@http://localhost:3000/main.bundle.js:238029:7
Router@http://localhost:3000/main.bundle.js:237968:7
BrowserRouter@http://localhost:3000/main.bundle.js:235921:7
AppRoutes
Provider@http://localhost:3000/main.bundle.js:233513:15
RtlProvider@http://localhost:3000/main.bundle.js:77228:9
ThemeProvider@http://localhost:3000/main.bundle.js:74294:7
ThemeProvider@http://localhost:3000/main.bundle.js:77517:7
ThemeProvider@http://localhost:3000/main.bundle.js:72567:9
Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://reactjs.org/link/error-boundaries to learn more about error boundaries. localhost:3000:12714:25
I think the issue above is also reflected in the failing pytest:
|
This should be resolved. It was due to the ListenerAuthorization dependency injection I added using |
@max-zilla
|
@max-zilla I see that error when I am on the page and am not in superAdmin mode. And no extractors are visible at all until I reload the page with superAdmin. I was thinking that they should show up, and we just need superAdmin mode to enable/disable? |
dataset_id was being sent in by the component as 'true' when it should be 'null' and then the 'true' should have gone after
I think I fixed the error that was causing no extractors to be visible. The component was sending in 'true' as a dataset_id as a default value. It should be 'null' and then the next one for 'all' should be 'true.' Committed this. |
With the commit I made, I think things work as intended for the admin user, but I don't think it is working for other users. I registered the wordcount extractor NOT as a private one, and even if it's running and enabled, it still only shows up for my admin user, and not for any others. |
I found what was going wrong here. When extractors are picked up by the heartbeat, the owner is "" but the script checks if they are None. I added that and now extractors that are not private are showing up like they should. I did create an issue and a branch, but since this is such a tiny fix I just put it here. To test, kill the heartbeat_listener docker container, and run the script in pycharm. Access will be None for public extractors registered. |
thanks for the fix, @tcnichol ! |
… query was not matching a user since it was an `equality` query on an `array` and not a `in` query.
…ue and it broke the query. Not sure what happened.
Adds optional support for an owner in the extractor heartbeat.
To test this, run an extractor with the following ENV VARs:
When the extractor starts, the email will be assigned as owner. That user should be able to see the extractor in lists as normal, but other users cannot. There are endpoints to grant other permissions:
...where 123 is extractor ID, and 456 is group ID. Only owner has permission to do this currently.
BASIC RULES.
If you are looking at a file or dataset Analysis tab, you will see extractor listed...
If you are looking at Extraction History, same rules but dataset rule is ignored. NOTE: if the user has submitted a job to an extractor, they should be able to see it here... need to work out permissions on that.
Changed the logic slightly for the process filter so if the extractor has process set to {} for file or dataset (i.e. no rule specified) it will be included in those lists.