-
Notifications
You must be signed in to change notification settings - Fork 445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to override endpoint for pyspark [BUG] #709
Comments
+1 |
Sure thing. I will do the quick test on the suggested fix on my end. |
Hi Brian, Some updates:Line 241 and 244 are calling _override_session_settings() to override session related setting(pls. see the method definition below). The endpoint info are saved in credential element like kernel_python_credentials[1].
The configure magic seems to be designed to override the spark session parameters only. But combined the overriding code from Konrad(see below), the session can be created for the overridden endpoint. The refresh_configuration method should be added before line 242 and 244 to handle two cases depending on if an existing session has been created or not.
So we can make this quick fix(simply add refresh_configuration) to enable the endpoint overriding. Any thoughts? [1] https://github.com/jupyter-incubator/sparkmagic/blob/master/sparkmagic/example_config.json#L2 |
**Description** Introduce an override type parameter for configure magic to support changing the configuration at runtime. To avoid breaking the default type is session_configs to ensure no behavior change. Configure type could be kernel_python_credentials, kernel_scala_credentials, authenticators etc. Also, after overriding the configuration, refresh_configuration is called to reflect the configuration changes. **Motivation** This change is to address the request: jupyter-incubator#709 **Testing Done** Added unit test cases and also manually test the endpoint overriding in a notebook using following magic. %%configure -f -t kernel_python_credentials {"username": "billy","base64_password": "d2VsY29tZTEyMw==", "url": "http://abc:8998", "auth": "Basic_Access"} %%configure -f -t kernel_python_credentials {"username": "", "password": "", "url": "http://abc:8998", "auth": "None"}
Describe the bug
I'm trying to override config for a livy endpoint in a pyspark notebook which doesn't work.
To Reproduce
Expected behavior
I'd expect that I can override credentials like above as the technique works for (at least some) other settings, e.g. the following works as expected:
If there is a reason to not allow such overriding there should ideally be some error or warning to explain.
Versions:
Additional context
I'm now aware that as a workaround I can start an IPython notebook, followed by
%load_ext sparkmagic.magics
and%spark add -l python -u http://livy.example.com
and add%%spark
to every cell.The text was updated successfully, but these errors were encountered: