Skip to content

Commit

Permalink
- Databricks Kernels are now moved to the top when opening a notebook…
Browse files Browse the repository at this point in the history 
… from the local sync folder or via `dbws:/`

- added `Insert` icons for Secrets and SecretScopes to easily add the code snippet to the current editor
- fixed an issue with `%run`
  • Loading branch information
Gerhard Brueckl committed Oct 28, 2022
1 parent 8362f4e commit b7f1ae9
Show file tree
Hide file tree
Showing 10 changed files with 178 additions and 165 deletions.
59 changes: 8 additions & 51 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,15 @@ The extensions can be downloaded from the official Visual Studio Code extension

# Release Notes

**v1.2.1:**
**v1.2.3:**
- Databricks Kernels are now moved to the top when opening a notebook from the local sync folder or via `dbws:/`
- added `Insert` icons for Secrets and SecretScopes to easily add the code snippet to the current editor
- fixed an issue with `%run`

**v1.2.2:**
- added `Restart Kernel` to Databricks notebooks
- added details specs of cluster to Notebook Kernel selector
- fixed an issue where the extension did not load when no default connection was specified
- fixed an issue with `%run` and absolute paths ([#93](/../../issues/93))
- fixed an issue with `Files in Repos` ([#101](/../../issues/101))
- fixed an issues with CLI Connection Manager ([#99](/../../issues/99))
Expand Down Expand Up @@ -104,55 +110,6 @@ The extensions can be downloaded from the official Visual Studio Code extension
- improved logging and output of messages
- use native VSCode icons instead of custom ones

**v0.9.4:**
- added support for VS Code setting `http.proxyStrictSSL` to optionally allow invalid certificates from the proxy. Can only be used in combination with `http.proxySupport = "on"`.
- use VS Code native secret management instead of `KeyTar`
- update Dev Dependencies to VS Code 1.66
- fix minor issue with SQL Browser refresh
- support for environment variables in `localSyncFolder`

**v0.9.3:**
- added `Copy Path` feature for Workspace and DBFS browser
- added Folders for Repositories as in the Databricks Web UI

**v0.9.2:**
- Make use of Databricks [Jobs API v2.1](https://docs.databricks.com/dev-tools/api/latest/jobs.html)

**v0.9.1:**
- fix issue when opening `.ipynb` files/notebooks

**v0.9.0:**
- Added support for [Repos API](https://docs.databricks.com/dev-tools/api/latest/repos.html)
- added new Repos tab
- switching branches
- deleteing a repo

**v0.8.9:**
- add support to list regular files from Git repositories. Up-/Download is not yet supported by the underlying API

**v0.8.8:**
- fix issues with Databricks running on GCP

**v0.8.7:**
- fix issue with repositories in workspace

**v0.8.6:**
- added support for Databricks CLI profiles
- use Databricks CLI profiles to manage connections
- support `DATABRICKS_CONFIG_FILE` for custom config file locations
- further improvements to the SQL Data browser
- Show Definition
- Datatypes for columns including complex columns (array, struct, map)
- Table and column comments
- improved tooltips and desciption
- improved tooltip for Connections
- fixed an issue with upload of whole workspace folders
- fixed an issue with Azure KeyVault secret scopes

**v0.8.0:**
- add SQL Data browser as in the Databricks UI
- fixed an issue with Secrets - you can now add/delete secrets again

# Outlook upcoming/planned features
- Support for Drag&Drop of local files directly into the workspace
- Fix remaining issues with Notebook Kernel and code cells
Expand Down Expand Up @@ -288,7 +245,7 @@ The cluster manager also distinguishes between regular user-created clusters and

Using Databricks Notebook Kernels you can execute local code againt a running Databricks cluster. Simply open a `.ipynb` notebook and select the Databricks kernel of your choice. A new kernel will be added automatically for each Databricks cluster that is currently running. In case you need to restart the kernel, you can do so by right-clicking the underlying cluster in the [Cluster Manager](#cluster-manager) and select `Restart Databricks Kernel`.
The execution of the first cell may take a bit longer as the remote execution context has to be set up before any commands can be executed. The notebook kernel supports the magics `%sql`, `%python`, `%r`, `%scala`, `%md` and also `%pip`. Depending on the output of the cell, the results are rendered accordingly - e.g. as text, as table or as image. Due to the limitations of the underlying APIs, the output is limited to one output type so you should avoid mixing `print(...)`, `display(df)` and `df.plot` as only one of them will be show.
If you need richer output especially for tables, you can install additional Notebook Renderer Extensions. We recommend [vscode-data-table](https://github.com/RandomFractals/vscode-data-table). Unfortunately the author decided to unpublish the extension from the offical VSCode market place but it can still be downloaded the `.vsix` from the [Github Releases](https://github.com/RandomFractals/vscode-data-table/releases)
For better visualization of tabluar results this extension includes a dependency to the extension [vscode-data-table](https://github.com/RandomFractals/vscode-data-table) which offers much more features than the standard visualizations.

# File System Integration
![File System Integration](/images/FileSystemIntegration.jpg?raw=true "File System Integration")
Expand Down
15 changes: 14 additions & 1 deletion package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"name": "databricks-vscode",
"displayName": "Databricks VSCode",
"description": "Extension to manage your Databricks workspace and ease local development",
"version": "1.2.1",
"version": "1.2.3",
"publisher": "paiqo",
"icon": "resources/databricks_extension.png",
"author": {
Expand Down Expand Up @@ -59,6 +59,9 @@
"onFileSystem:dbfs",
"onFileSystem:dbws"
],
"extensionDependencies": [
"RandomFractalsInc.vscode-data-table"
],
"main": "./dist/node/extension.js",
"browser_NOT_WORKING_YET": "./dist/web/extension.js",
"contributes": {
Expand Down Expand Up @@ -669,6 +672,11 @@
"title": "Delete",
"icon": "$(remove-close)"
},
{
"command": "databricksSecret.insertCode",
"title": "Insert",
"icon": "$(code)"
},
{
"command": "databricksSQL.refresh",
"title": "Refresh",
Expand Down Expand Up @@ -926,6 +934,11 @@
"when": "view == databricksSecrets && viewItem == SECRET_DATABRICKS",
"group": "inline@99"
},
{
"command": "databricksSecret.insertCode",
"when": "view == databricksSecrets && viewItem =~ /^SECRET_.*/",
"group": "inline@90"
},
{
"command": "databricksSQL.refresh",
"when": "view == databricksSQL && viewItem == DATABASE",
Expand Down
1 change: 1 addition & 0 deletions src/extension.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ export async function activate(context: vscode.ExtensionContext) {
vscode.commands.registerCommand('databricksSecretScope.addSecret', (scope: DatabricksSecretScope) => scope.addSecret());
vscode.commands.registerCommand('databricksSecret.update', (secret: DatabricksSecret) => secret.update());
vscode.commands.registerCommand('databricksSecret.delete', (secret: DatabricksSecret) => secret.delete());
vscode.commands.registerCommand('databricksSecret.insertCode', (scope: DatabricksSecret) => scope.insertCode(scope));

// register DatabricksSQLTreeProvider
let databricksSQLTreeProvider = new DatabricksSQLTreeProvider();
Expand Down
131 changes: 73 additions & 58 deletions src/vscode/notebook/DatabricksKernel.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,12 +60,20 @@ export class DatabricksKernel implements vscode.NotebookController {
this._controller.supportsExecutionOrder = this.supportsExecutionOrder;
this._controller.executeHandler = this.executeHandler.bind(this);
this._controller.dispose = this.disposeController.bind(this);
//this._controller.updateNotebookAffinity = this.updateNotebookAffinity.bind(this);
this._controller.onDidChangeSelectedNotebooks(this._onDidChangeSelectedNotebooks, this, []);

vscode.workspace.onDidOpenNotebookDocument((event) => this._onDidOpenNotebookDocument(event));

ThisExtension.PushDisposable(this);
}

async _onDidOpenNotebookDocument(notebook: vscode.NotebookDocument) {
// set this controller as recommended Kernel for notebooks opened via dbws:/ file system or from or local sync folder
if (notebook.uri.scheme == "dbws" || notebook.uri.toString().startsWith(ThisExtension.ActiveConnection.localSyncFolder.toString())) {
this._controller.updateNotebookAffinity(notebook, vscode.NotebookControllerAffinity.Preferred);
}
}

// #region Cluster-properties
static getId(clusterId: string, kernelType: KernelType) {
return this.baseId + clusterId + "-" + kernelType;
}
Expand All @@ -89,17 +97,14 @@ export class DatabricksKernel implements vscode.NotebookController {
details += "Head: " + this._cluster.driver_node_type_id;

if (this._cluster.num_workers != undefined) {
if(this._cluster.num_workers > 0)
{
details += " - " + this._cluster.num_workers + " Worker(s): " + this._cluster.node_type_id;
if (this._cluster.num_workers > 0) {
details += " - " + this._cluster.num_workers + " Worker(s): " + this._cluster.node_type_id;
}
else
{
else {
details += " (SingleNode)";
}
}
else if(this._cluster.autoscale)
{
else if (this._cluster.autoscale) {
details += " - " + this._cluster.executors.length + " (" + this._cluster.autoscale.min_workers + "-" + this._cluster.autoscale.max_workers + ")";
details += " Worker(s): " + this._cluster.node_type_id;
}
Expand All @@ -111,11 +116,11 @@ export class DatabricksKernel implements vscode.NotebookController {
return this._language;
}

private get ClusterRuntime(): {major: number, minor: number} {
private get ClusterRuntime(): { major: number, minor: number } {
let tokens: string[] = this._cluster.spark_version.split("-")[0].split(".")
return { major: +tokens[0], minor: +tokens[1]}
return { major: +tokens[0], minor: +tokens[1] }
}

// #endregion

async disposeController(): Promise<void> {
for (let context of this._executionContexts.entries()) {
Expand Down Expand Up @@ -183,12 +188,11 @@ export class DatabricksKernel implements vscode.NotebookController {

vscode.window.showErrorMessage("Could not setup Files in Repo for '" + notebookUri + "' on cluster '" + this.ClusterID + "'\nPlease check logs for details!");
}
else
{
else {
ThisExtension.log(`NotebookKernel: Successfully set up FilesInRepo by running '${commandTexts[2]}' and '${commandTexts[3]}' on driver!`);
ThisExtension.setStatusBar("FilesInRepo initialized!");
}

}
}

Expand All @@ -198,18 +202,7 @@ export class DatabricksKernel implements vscode.NotebookController {
}
interruptHandler?: (notebook: vscode.NotebookDocument) => void | Thenable<void>;
readonly onDidChangeSelectedNotebooks: vscode.Event<{ readonly notebook: vscode.NotebookDocument; readonly selected: boolean; }>;
private async _onDidChangeSelectedNotebooks(event: { notebook: vscode.NotebookDocument; selected: boolean }) {
// TODO: this could should help to make the Databricks Kernel more prominent for notebooks loaded via Databricks extension
if (event.notebook.uri.scheme == "dbws" || event.notebook.uri.toString().startsWith(ThisExtension.ActiveConnection.localSyncFolder.toString())) {
this._controller.updateNotebookAffinity(event.notebook, vscode.NotebookControllerAffinity.Preferred);
}
}
updateNotebookAffinity(notebook: vscode.NotebookDocument, affinity: vscode.NotebookControllerAffinity): void {

//let x = "updateNotebookAffinity";

//ThisExtension.log(x);
}
updateNotebookAffinity(notebook: vscode.NotebookDocument, affinity: vscode.NotebookControllerAffinity): void { }



Expand Down Expand Up @@ -297,6 +290,7 @@ export class DatabricksKernel implements vscode.NotebookController {
switch (cell.notebook.uri.scheme) {
case "dbws":
runUri = vscode.Uri.file(runFile);
runUri = runUri.with({ scheme: "dbws" })
break;
case "file":
runUri = await FSHelper.joinPath(ThisExtension.ActiveConnection.localSyncFolder, ThisExtension.ActiveConnection.localSyncSubfolders.Workspace, runFile);
Expand All @@ -307,40 +301,61 @@ export class DatabricksKernel implements vscode.NotebookController {
}

ThisExtension.log("Executing %run for '" + runUri + "' ...");
switch (runUri.scheme) {
case "dbws":
// we cannot use vscode.workspace.fs here as we need the SOURCE file and not the ipynb file
commandText = Buffer.from(await DatabricksApiService.downloadWorkspaceItem(runUri.path, "SOURCE")).toString();
break;
case "file":
// %run is not aware of filetypes/extensions/langauges so we need to check which ones actually exist locally
let allFiles = await vscode.workspace.fs.readDirectory(FSHelper.parent(runUri));
let relevantFiles = allFiles.filter(x => x[0].startsWith(FSHelper.basename(runUri)));

if (relevantFiles.length == 1) {
runUri = vscode.Uri.joinPath(FSHelper.parent(runUri), relevantFiles[0][0]);
commandText = Buffer.from(await vscode.workspace.fs.readFile(runUri)).toString();

if (FSHelper.extension(runUri) == ".ipynb") {
let notebookJSON = JSON.parse(commandText);
let cells = notebookJSON["cells"];
cells = cells.filter(x => x["cell_type"] == "code"); // only take code cells

let commands: string[] = [];
for (let cell of cells) {
commands.push(cell["source"].join("\n"));
}
try {

switch (runUri.scheme) {
case "dbws":
if (!await FSHelper.pathExists(runUri)) {
throw vscode.FileSystemError.FileNotFound(runUri);
}
// we cannot use vscode.workspace.fs here as we need the SOURCE file and not the ipynb file
commandText = Buffer.from(await DatabricksApiService.downloadWorkspaceItem(runUri.path, "SOURCE")).toString();
break;

case "file":
// %run is not aware of filetypes/extensions/langauges so we need to check which ones actually exist locally
let allFiles = await vscode.workspace.fs.readDirectory(FSHelper.parent(runUri));
let relevantFiles = allFiles.filter(x => x[0].startsWith(FSHelper.basename(runUri)));

if (relevantFiles.length == 1) {
runUri = vscode.Uri.joinPath(FSHelper.parent(runUri), relevantFiles[0][0]);
commandText = Buffer.from(await vscode.workspace.fs.readFile(runUri)).toString();

commandText = commands.join("\n");
if (FSHelper.extension(runUri) == ".ipynb") {
let notebookJSON = JSON.parse(commandText);
let cells = notebookJSON["cells"];
cells = cells.filter(x => x["cell_type"] == "code"); // only take code cells

let commands: string[] = [];
for (let cell of cells) {
commands.push(cell["source"].join("\n"));
}

commandText = commands.join("\n");
}
}
else {
throw vscode.FileSystemError.FileNotFound(runUri);
}
}
else {
throw vscode.FileSystemError.FileNotFound(runUri);
}
break;
default:
throw new Error("%run is not supported for FileSystem scheme '" + runUri.scheme + "'!");
break;
default:
throw new Error("%run is not supported for FileSystem scheme '" + runUri.scheme + "'!");
}
}
catch (error) {
execution.appendOutput(new vscode.NotebookCellOutput([
vscode.NotebookCellOutputItem.text(error.name + ": " + error.message, 'text/html')
]));

execution.appendOutput(new vscode.NotebookCellOutput([
vscode.NotebookCellOutputItem.error(error),
]));
execution.end(false, Date.now());
return;
}



let outputRun: vscode.NotebookCellOutput = new vscode.NotebookCellOutput([
vscode.NotebookCellOutputItem.text(commandText, 'text/plain')
])
Expand Down
Loading

0 comments on commit b7f1ae9

Please sign in to comment.