[WIP] Colab Env Check & Install #9
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…(if any) specific to allocation
Create colab-env-check.sh
Thinking this is more runable, having issues executing !nvidia-smi
Switch to pynvml from !nvidia-smi
…/libcuml_1571339163826/work/cpp/src_prims/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory)
reformatted; use pynvml (preinstalled in base Colab) - remove other scripts or format like so if kept
Colab Environment Check (function & script)
Feature utils branch
make inot script
make into script, formatting
Thank you, Adam.
testing elimination of temp fix colab install
sync with master
Checking for GPU type:
Woo! You got the right kind of GPU!
Traceback (most recent call last):
File "colab_env.py", line 46, in <module>
subprocess.Popen(['blazingsql-orchestrator', '9100', '8889', '127.0.0.1', '8890'],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'blazingsql-orchestrator': 'blazingsql-orchestrator'
Trying to complete the setting paths part of temp p-100 colab install within bsql-colab.sh
takes too long
does not work as python script, forever pends connection 8889 ``` ********************************************* Your Google Colab instance is BlazingSQL ready! ********************************************* --2019-11-13 02:19:45-- https://github.com/gumdropsteve/bsql-demos/raw/feature/utils/utils/set_paths.py Resolving github.com (github.com)... 140.82.118.4 Connecting to github.com (github.com)|140.82.118.4|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://raw.githubusercontent.com/gumdropsteve/bsql-demos/feature/utils/utils/set_paths.py [following] --2019-11-13 02:19:46-- https://raw.githubusercontent.com/gumdropsteve/bsql-demos/feature/utils/utils/set_paths.py Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 571 [text/plain] Saving to: ‘set_paths.py’ set_paths.py 100%[===================>] 571 --.-KB/s in 0s 2019-11-13 02:19:46 (116 MB/s) - ‘set_paths.py’ saved [571/571] Setting RAPIDS paths SLF4J: The requested version 1.6 by your slf4j binding is not compatible with [1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10] SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details. /usr/local/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning: Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.so. For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup') warnings.warn(errors.NumbaWarning(msg)) /usr/local/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning: Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice/. For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup') warnings.warn(errors.NumbaWarning(msg)) Usage: blazingsql-engine <RAL_ID> <GPU_ID> <ORCHESTRATOR_HTTP_COMMUNICATION_[IP|HOSTNAME]> <ORCHESTRATOR_HTTP_COMMUNICATION_PORT> <RAL_HTTP_COMMUNICATION_[IP|HOSTNAME]> <RAL_HTTP_COMMUNICATION_PORT> <RAL_TCP_PROTOCOL_PORT> <OPTIONAL_NETWORK_INTERFACE_NAME_ETH0_DEFAULT> RAL Engine starting Using the network interface: eth0 RAL ID: 1 GPU ID: 0 Orchestrator HTTP communication host: 127.0.0.1 Orchestrator HTTP communication port: 9100 RAL HTTP communication host: 172.28.0.2 RAL HTTP communication port: 9001 RAL protocol port: 8891 Socket Name: 172.28.0.2 Log Name: RAL.1.log RAL TCP protocol port: 8891 java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:126) at com.blazingdb.calcite.application.TCPClient.send(TCPClient.java:38) at com.blazingdb.calcite.application.TCPClient.send(TCPClient.java:55) at com.blazingdb.calcite.application.TCPService.synchronizeSchema(TCPService.java:63) at com.blazingdb.calcite.application.TCPService.<init>(TCPService.java:53) at com.blazingdb.calcite.application.CalciteApplication.main(CalciteApplication.java:204) schema synchronization with Orchestrator failed Waiting for messages in TCP port: 8890 ```
Does not work
Shouldn't be here, bad fork
Merge pull request #8 from gumdropsteve/gumdropsteve-patch-1
Doing this in a separate cell works more logically in notebook.
More clear success output.
gumdropsteve
changed the title
|
@roaramburu ready for review, thanks
|
gumdropsteve
changed the title
gumdropsteve
changed the title
plus working bsql colab solution
remove extra
* Create blazingsql.sh * Update blazingsql.sh
* Update bsql-colab.sh * Update bsql-colab.sh * Delete bsql-colab.sh
|
Pending: BlazingDB/blazingsql#182 Error reproduced: https://gist.github.com/gumdropsteve/ccd783eb9d53fcf74b8de4852fec4d91 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add
env_check.pyto check the Colab environment and let users know their situation andbsql-colab.shto install BlazingSQL in Colab. + remove extra files.