Rule based uploader issues / sharp edges

[hexylena]

I'm working on the tutorial @jmchilton on 18.05 on usegalaxy.eu. I'll update this issue throughout the day

Rule based uploader auto-closes if a rule-based job finishes while interface open

• create a rule-based job
• close the window / let it start processing in background
• started a new rule-based job
• first job finishes
• rule based uploader closes
• progress is lost

Cannot scroll far enough

• Cannot see the entire table
• Also scroll bar doesn't seem to be there? There's a gray space in chrome where it looks like it should be there's nothing I can click on

Accidentally closed/refreshed and lost progress

• If there's any way to store the current rule state it would be really more user friendly if we accidentally click anywhere outside the rule builder. My touchpad is awful :(

Regex only could be supplemented with "common" operations

e.g.

• split by character (rather than regex)

Rule Hovering

Some rules when you hover over them highlight the row and explain the change, but not all do.

• Set column as definition
• Add a new column (would be nice to know /which/ column was added by a particular operation if you've added a lot)
• Sort on column

[jmchilton]

Rule based uploader auto-closes if a rule-based job finishes while interface open

That is not tracked anywhere else and I didn't think about that but it makes a lot of sense given the implementation of this as a sort of singleton. That should be fixed.

Cannot scroll far enough

Yeah, crap. #6100

Accidentally closed/refreshed and lost progress

The lowest hanging fruit is to just track the last set of rules in browser history and add a little button to "Restore Last Rules". It isn't enough, but it would be a big step forward I think.

Regex only could be supplemented with "common" operations

Yeah - I mean it is a balancing act in every language design all the way down to processor - RISC vs. CISC. I like split on in general though - that should be very doable.

Some rules when you hover over them highlight the row and explain the change, but not all do.

Yeah - it will show it for column definitions but not for rules essentially. Things like Swapping columns, Removing columns, etc.. means I can't always be sure the target column is still correct. I could potentially track it - but if the rule says "Add Regex to Column E" and then column C is highlighted that might be just as jarring. I'm not sure.

[hexylena]

Just wrote these without providing any suggestions, sorry. I also should've started the issue off with some of the more important things about it:

• it works incredibly well
• it's really pretty intuitive to use and relatively user friendly. Having never used it before, following the examples, I'm feeling pretty confident in it.

"Restore Last Rules". It isn't enough, but it would be a big step forward I think.

Ooh that's a great idea. Would also let users re-run the same rules over many sets of input metadata, if for some reason they couldn't obtain all of the data in a single input.

balancing act in every language

definitely! and regex was a single feature you could implement that would support a huge range of possibilities without having to implement a ton of really manual operations. Just felt that some additional hardcoded operations might provide utility to users not yet comfortable with regex.

I could potentially track it ... might be just as jarring

Ah this is a good point, I hadn't considered that case. That would be jarring as well and could confuse users. Erring on the side of safety is good.

[hexylena]

Running example 4 fails

https://github.com/jmchilton/training-material/blob/rules/topics/introduction/tutorials/galaxy-intro-rules/tutorial.md#example-building-matched-collections

Here are my rules:

{"rules":[{"type":"add_filter_count","count":"1","which":"first","invert":false},{"type":"remove_columns","target_columns":[1,2,4,5,6]},{"type":"sort","target_column":0,"numeric":false},{"type":"add_column_regex","target_column":0,"expression":".*","replacement":"http://www.uniprot.org/uniprot/\\0.fasta"},{"type":"add_column_value","value":"fasta"},{"type":"add_column_value","value":"ASDF fasta"},{"type":"add_column_regex","target_column":0,"expression":".*","replacement":"https://www.uniprot.org/uniprot/\\0.gff"},{"type":"add_column_value","value":"gff3"},{"type":"add_column_value","value":"ASDF gff3"},{"type":"split_columns","target_columns_0":[2,3,4],"target_columns_1":[5,6,7]}],"mapping":[{"type":"info","columns":[1]},{"type":"list_identifiers","columns":[0],"editing":false},{"type":"url","columns":[2]},{"type":"collection_name","columns":[4]},{"type":"file_type","columns":[3]}],"extension":"csfasta"}

Fails like this:

https://sentry.usegalaxy.eu/share/issue/6daf1925340d4d1e89cdc7df51950353/

No error is shown despite rule builder note

After clicking "build" you get the notice "you can close this but you won't see any error messages" and even when errors happen (I guess this is harder to catch / unexpected since this is a JS error in the client code). But no errors show up. Maybe this is out of scope?

[hexylena]

I don't see "Apply Rule to Collection" on either of usegalaxy.*, is this expected to be in 18.05 and we're missing it?

[martenson]

@erasche

sorry, I got confused, John is right :)

[jmchilton]

@erasche Yes, it probably requires you to update your tool_conf.xml. https://github.com/galaxyproject/galaxy/blob/release_18.05/config/tool_conf.xml.sample#L38.

[jmchilton]

Also that tool type has a bug you can actually see in the screenshots, if you select a collection in the first step it will the rules against the collection you select but not preview against the correct collection. This is bad and I need to fix ASAP.

[hexylena]

@martenson neat, didn't see that. But unfortunately selecting a single collection and attempting that:

@jmchilton ah ok, thanks. I'll enable those. I don't see them on main either so you might want to enable them as well. (Could be me being blind.)

[martenson]

I don't see them on main either so you might want to enable them as well. (Could be me being blind.)

I will fix that

[jmchilton]

Running example 4 fails

Should be fixed with #6231, sorry about that.

[hexylena]

Apply Rules from File / Text

It might be nice if __APPLY_RULES__ could load rules from files or text input. I guess my usecase for this is thinking more about workflows / helping new users run things from example histories.

[shiltemann]

I just wanted to chime in to say that this last point by @erasche would be super helpful for me.

Use case: I have a project where we have sequenced 2 loci per patient, and each is paired-end data (so basically a list of pairs of pairs). The file names contain all information needed to match the files together .

It would be really nice if the clinicians could just upload all files into a simple list collection without worrying about the rule builder, and I could then put the collection building steps in the workflow (e.g. first pair on forward/reverse, do contig merging, then pair on patient and do rest of analysis)

[jmchilton]

@shiltemann If I'm understanding correctly (probably not) I think you can do that today with workflows. If they are uploading to a simple list collection - the file names should be preserved as list identifiers and then the "Apply Rules" tool can load those identifiers in and re-process them to rules to give the list depth or whatever structure you want derived from the identifiers as the first step in the workflow. (Or multiple steps in the workflow to build different structures for different parts of analysis.)

The key thing that prevents needing to load extra data in from text input is I think:

The file names contain all information needed to match the files together

That is exactly what the "Apply Rules" tool allows you to process.

Certainly there are going to be cases out there where extra information useful to building structure isn't found in file names - and then @erasche's request is totally still needed. Step 1 will be attaching tags to inputs as they come in for extra metadata and allow processing with that, but Step 2 will be what @erasche described I think. We will have to implement joins and stuff - and the effort to recreate Excel in Galaxy will become and effect to recreate SQL in Galaxy - but I don't think that will be needed for what you described - hopefully not any way.

[shiltemann]

@jmchilton oh, nope, I was the one misunderstanding, you are totally right, I went back and tried again and it does exactly what I need, this is great! :)

[jmchilton]

I think I've migrated the issues I have yet to fix in dev to that quite daunting checklist that in #5822 - so I'm going to close this and use that to track these usability issues.