-
Notifications
You must be signed in to change notification settings - Fork 1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones. The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend. Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented. In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs. In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files. The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
- Loading branch information
Showing
14 changed files
with
2,333 additions
and
96 deletions.
There are no files selected for viewing
1,859 changes: 1,859 additions & 0 deletions
1,859
client/galaxy/scripts/components/RuleCollectionBuilder.vue
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
176 changes: 176 additions & 0 deletions
176
client/galaxy/scripts/mvc/upload/collection/rules-input-view.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,176 @@ | ||
| import _l from "utils/localization"; | ||
| import Ui from "mvc/ui/ui-misc"; | ||
| import Select from "mvc/ui/ui-select"; | ||
| import UploadUtils from "mvc/upload/upload-utils"; | ||
| import axios from "axios"; | ||
|
|
||
| export default Backbone.View.extend({ | ||
| initialize: function(app) { | ||
| this.app = app; | ||
| this.options = app.options; | ||
| this.setElement(this._template()); | ||
| this.btnBuild = new Ui.Button({ | ||
| id: "btn-build", | ||
| title: _l("Build"), | ||
| onclick: () => { | ||
| this._eventBuild(); | ||
| } | ||
| }); | ||
| _.each([this.btnBuild], button => { | ||
| this.$(".upload-buttons").prepend(button.$el); | ||
| }); | ||
| const dataTypeOptions = [ | ||
| { id: "datasets", text: "Datasets" }, | ||
| { id: "collections", text: "Collection(s)" }, | ||
| ]; | ||
| this.dataType = "datasets"; | ||
| this.dataTypeView = new Select.View({ | ||
| css: "upload-footer-selection", | ||
| container: this.$(".rule-data-type"), | ||
| data: dataTypeOptions, | ||
| value: this.dataType, | ||
| onchange: value => { | ||
| this.dataType = value; | ||
| // this._renderSelectedType(); | ||
| } | ||
| }); | ||
|
|
||
| const selectionTypeOptions = [ | ||
| { id: "paste", text: "Pasted Table" }, | ||
| { id: "dataset", text: "History Dataset" }, | ||
| { id: "ftp", text: "FTP Directory" } | ||
| ]; | ||
| this.selectionType = "paste"; | ||
| this.selectionTypeView = new Select.View({ | ||
| css: "upload-footer-selection", | ||
| container: this.$(".rule-select-type"), | ||
| data: selectionTypeOptions, | ||
| value: this.selectionType, | ||
| onchange: value => { | ||
| this.selectionType = value; | ||
| this._renderSelectedType(); | ||
| } | ||
| }); | ||
| this.selectedDatasetId = null; | ||
|
|
||
| this._renderSelectedType(); | ||
| }, | ||
|
|
||
| _renderSelectedType: function() { | ||
| const selectionType = this.selectionType; | ||
| if (selectionType == "dataset") { | ||
| if (!this.datasetSelectorView) { | ||
| this.selectedDatasetId = null; | ||
| const history = parent.Galaxy && parent.Galaxy.currHistoryPanel && parent.Galaxy.currHistoryPanel.model; | ||
| const historyContentModels = history.contents.models; | ||
| const options = []; | ||
| for (let historyContentModel of historyContentModels) { | ||
| const attr = historyContentModel.attributes; | ||
| if (attr.history_content_type !== "dataset") { | ||
| continue; | ||
| } | ||
| options.push({ id: attr.id, text: `${attr.hid}: ${_.escape(attr.name)}` }); | ||
| } | ||
| this.datasetSelectorView = new Select.View({ | ||
| container: this.$(".dataset-selector"), | ||
| data: options, | ||
| placeholder: _l("Select a dataset"), | ||
| onchange: val => { | ||
| this._onDataset(val); | ||
| } | ||
| }); | ||
| } else { | ||
| this.datasetSelectorView.value(null); | ||
| } | ||
| } else if (selectionType == "ftp") { | ||
| UploadUtils.getRemoteFiles(ftp_files => { | ||
| this._setPreview(ftp_files.map(file => file["path"]).join("\n")); | ||
| }); | ||
| } | ||
| this._updateScreen(); | ||
| }, | ||
|
|
||
| _onDataset: function(selectedDatasetId) { | ||
| this.selectedDatasetId = selectedDatasetId; | ||
| if (!selectedDatasetId) { | ||
| this._setPreview(""); | ||
| return; | ||
| } | ||
| axios | ||
| .get( | ||
| `${Galaxy.root}api/histories/${Galaxy.currHistoryPanel.model.id}/contents/${selectedDatasetId}/display` | ||
| ) | ||
| .then(response => { | ||
| this._setPreview(response.data); | ||
| }) | ||
| .catch(error => console.log(error)); | ||
| }, | ||
|
|
||
| _eventBuild: function() { | ||
| const selection = this.$(".upload-rule-source-content").val(); | ||
| this._buildSelection(selection); | ||
| }, | ||
|
|
||
| _buildSelection: function(content) { | ||
| const selectionType = this.selectionType; | ||
| const selection = { content: content }; | ||
| if (selectionType == "dataset" || selectionType == "paste") { | ||
| selection.selectionType = "raw"; | ||
| } else if (selectionType == "ftp") { | ||
| selection.selectionType = "ftp"; | ||
| } | ||
| selection.dataType = this.dataType; | ||
| Galaxy.currHistoryPanel.buildCollection("rules", selection, true); | ||
| this.app.modal.hide(); | ||
| }, | ||
|
|
||
| _setPreview: function(content) { | ||
| $(".upload-rule-source-content").val(content); | ||
| this._updateScreen(); | ||
| }, | ||
|
|
||
| _updateScreen: function() { | ||
| const selectionType = this.selectionType; | ||
| const selection = this.$(".upload-rule-source-content").val(); | ||
| this.btnBuild[selection || selectionType == "paste" ? "enable" : "disable"](); | ||
| this.$("#upload-rule-dataset-option")[selectionType == "dataset" ? "show" : "hide"](); | ||
| this.$(".upload-rule-source-content").attr("disabled", selectionType !== "paste"); | ||
| }, | ||
|
|
||
| _template: function() { | ||
| return ` | ||
| <div class="upload-view-default"> | ||
| <div class="upload-top"> | ||
| <h6 class="upload-top-info"> | ||
| Tabular source data to extract collection files and metadata from | ||
| </h6> | ||
| </div> | ||
| <div class="upload-box" style="height: 335px;"> | ||
| <span style="width: 25%; display: inline; height: 100%" class="pull-left"> | ||
| <div class="upload-rule-option"> | ||
| <div class="upload-rule-option-title">${_l("Upload data as")}:</div> | ||
| <div class="rule-data-type" /> | ||
| </div> | ||
| <div class="upload-rule-option"> | ||
| <div class="upload-rule-option-title">${_l("Load tabular data from")}:</div> | ||
| <div class="rule-select-type" /> | ||
| </div> | ||
| <div id="upload-rule-dataset-option" class="upload-rule-option"> | ||
| <div class="upload-rule-option-title">${_l("Select dataset to load")}:</div> | ||
| <div class="dataset-selector" /> | ||
| </div> | ||
| </span> | ||
| <span style="display: inline; float: right; width: 75%; height: 300px"> | ||
| <textarea class="upload-rule-source-content form-control" style="height: 100%"></textarea> | ||
| </span> | ||
| </div> | ||
| <div class="clear" /> | ||
| <!-- | ||
| <div class="upload-footer"> | ||
| </div> | ||
| --> | ||
| <div class="upload-buttons"/> | ||
| </div> | ||
| `; | ||
| } | ||
| }); |
Oops, something went wrong.