Jsvis prototype

[francoismg]

Prototype for js visualization

Still to do :

• Add error bars processing for D3
• More robust error bar processing for Bokeh
• Code from data processor and wrappers needs to be improved
• Some inconsistencies to fix and inheritance to lessen code duplication
• Some clean up
• make an npm package
• custom columns on the error bar
• delete column cross does not work
• make bokeh a dependency
• add arbitrary digits processing to arithmetic columns

@volodymyrss @dsavchenko

[github-actions]

PR Preview Action v1.4.8
Preview removed because the pull request was closed.
2024-12-03 07:30 UTC

[volodymyrss]

Looks interesting, shows in the preview! How do I load file in there?

[francoismg]

Looks interesting, shows in the preview! How do I load file in there?

Right now it works like a galaxy plugin so file cannot be changed in the interface, if you want to try another file you have to add it to "test_files" and point to it in main.js

I could add a new box that is populated with a list of accessible files and let the user choose between them if you want.

It wasn't clear if the use case for a standalone version was a file or list of files or a specific endpoint is set in the code like with other js library and you instantiate the interface for a specific file or if it was up to the user to provide the files himself. Since there are probably way better tool to do that locally with local files I thought the main use would be that files are remote

[volodymyrss]

Right now it works like a galaxy plugin so file cannot be changed in the interface, if you want to try another file you have to add it to "test_files" and point to it in main.js

It makes sense if it works like AladinLite, with an option to load local file or by URL.

I could add a new box that is populated with a list of accessible files and let the user choose between them if you want.

This would be useful. Else the demonstration does not actually demonstrate.

It wasn't clear if the use case for a standalone version was a file or list of files or a specific endpoint is set in the code like with other js library and you instantiate the interface for a specific file or if it was up to the user to provide the files himself.

We wanted it to be like AladinLite in the file load process. Presumably it's not actually difficult to select the source from upload and URL, right?

Since there are probably way better tool to do that locally with local files I thought the main use would be that files are remote

There is really not actually. UIs largely moved to the Web/Browser experience, but this sort of tool, similar to what Andrii was mentioning (called FV), was not re-implemented for Web. This is why we kind of have to do it from scratch.
But I think it starts to look very interesting!

[francoismg]

Right now it works like a galaxy plugin so file cannot be changed in the interface, if you want to try another file you have to add it to "test_files" and point to it in main.js

It makes sense if it works like AladinLite, with an option to load local file or by URL.

I could add a new box that is populated with a list of accessible files and let the user choose between them if you want.

This would be useful. Else the demonstration does not actually demonstrate.

It wasn't clear if the use case for a standalone version was a file or list of files or a specific endpoint is set in the code like with other js library and you instantiate the interface for a specific file or if it was up to the user to provide the files himself.

We wanted it to be like AladinLite in the file load process. Presumably it's not actually difficult to select the source from upload and URL, right?

Since there are probably way better tool to do that locally with local files I thought the main use would be that files are remote

There is really not actually. UIs largely moved to the Web/Browser experience, but this sort of tool, similar to what Andrii was mentioning (called FV), was not re-implemented for Web. This is why we kind of have to do it from scratch. But I think it starts to look very interesting!

Ok no problem I will add a new box with an input that lets the user specify a file (url is the easiest right now but needs to come from a cors compliant server (or we can use an open cors proxy server), for local files uploaded directly I will just have to check fits-reader cause there was some specific things when dealing with local files) and once it's loaded it will be added to a list of available files that the user can select

[volodymyrss]

Would be really cool if there was something for the demo today!

[francoismg]

Would be really cool if there was something for the demo today!

I just finished error bars for d3 so not sure I will have the time but will try

[francoismg]

For reference :

• add the possibility for user to download and change current file (investigate possible way to do cross origin download)
• synchronize hdu choice between header and data components
• add extname to hdu name
• add arithmetic and mathematical operation for column choice

[francoismg]

• Bugs with bokeh graphs have been fixed and error bars added to D3

• I was able to plot the file from mmoda without changes to the code, only problem is the fact that timedel is from the header and changes regarding how this is handled are ongoing

• For url file upload I looked at bypassing cors policies but I can't find a way to do that in client side js only without relying on some server config or code, using a proxy server or local proxy, or routing the requests through some backend. If you still have the code from that data challenge you talked about I could have a look at it maybe I have missed something

Regarding file upload, changes to the code and interface are still in progress and it's not testable right now but an usable minimal version should be ready during the week

Local file upload works with fits reader you just have to do things a bit differently that you would normally do but it works fine. However there were other things to take into consideration since users will be able to upload file directly, that csv support should be added, plus arithmetic column and multiple datasets support.

Ongoing development :

• User will have to specify a file type before upload (fits and csv right now)
• User will be able to upload any number of local files that will be displayed in an available files list
• Files in the available list can be selected and parametrized to some extent (custom column names, header cards value change.....) + some general information about the files are displayed
• User can move files from the available list to a "current" files list that will be used for generating visualization
• User can remove files from the current list when necessary
• Settings let the user choose between any column of any hdu or files that are in the current files list (every column is prefixed by a file id (and an hdu index when applicable)). That will let the user be able to plot things using data from any number of files together and do column arithmetic with columns from any files
• Multiple datasets can be plotted on the same graph
• Header and data component will still be synchronized together but will be able to show data from any files in the current list

[dsavchenko]

@francoismg I made a small change so that the file that's in the repo is accessible in preview
But now there are two files in "available" and only the second one is really working. But I hope you can debug this easily

[francoismg]

@francoismg I made a small change so that the file that's in the repo is accessible in preview But now there are two files in "available" and only the second one is really working. But I hope you can debug this easily

yes I have the same thing locally it was just some test I made, I will remove that so it's not confusing

[volodymyrss]

check how it's done here https://github.com/oda-hub/dispatcher-app/blob/master/cdci_data_analysis/analysis/plot_tools.py

add range-zoom UI

[volodymyrss]

How is it going, @francoismg ?

[francoismg]

How is it going, @francoismg ?

Custom range is working for D3 still a few bugs but seems to work, I will push it so you can test that, just need to remove things that are not working. If you want to test it you need to select D3, x range is working fine but y range needs x range to be set to work (one of the bugs I'm working on).

Bokeh part should be good soon and hopefully the plugin part and the other unrelated bugs I have spotted could be ready for friday

[francoismg]

Arithmetic columns are now available for error bars. Updated npm package has been published.

Will test again to import the package into another webpack project locally and then to galaxy when it's working

[francoismg]

Support for numbers in arithmetic column has been added

Spotted a bug to fix -> deleted custom columns can still appear in dropdown

[francoismg]

Integration inside a galaxy plugin is mostly ok, had to make some small changes since it's now a dependency of another webpack app but there seems to be some issues with yarn that cannot find the package during the client build, not sure if it's really related to yarn or just one of the generic issues when trying to build a plugin like with the first version of the plugin

Will make a new branch for the galaxy pull request once the issue is fixed

[francoismg]

Build and file load issues inside galaxy have been fixed will make the new branch for pull request and put the link for the new branch here

[francoismg]

There seems to have some bugs on galaxy latest dev branch (unrelated to the plugin) since at least yesterday that prevent client build. I pushed the modifications so it's ready for pr but probably better to wait for dev branch to work to make a check before opening the pr

branch is available here: https://github.com/esg-epfl-apc/galaxy/tree/fits-graph-visualization-plugin-pr

[bgruening]

What is not working on the dev branch? We do have a few green PRs, so it should work.

[francoismg]

What is not working on the dev branch? We do have a few green PRs, so it should work.

There seems to be 2 typescript or vue errors (by forking latest dev or directly cloning https://github.com/galaxyproject/galaxy.git)

TS1484: 'Workflow' is a type and must be imported using a type-only import when 'verbatimModuleSyntax' is enabled

TS2339: Property 'username' does not exist on type '{ isAnonymous: false; deleted: boolean; email: string; id: string; is_admin: boolean; nice_total_disk_usage: string; preferences: Record<string, never>; preferred_object_store_id?: string

Coming from WorkflowInvocationHeader.vue.ts and raised after plugin build when building the whole app frontend (I think)

Build from last week was working fine on the same machine btw

[francoismg]

Issues with the latest dev build seem to have been resolved, will update the fork and test it locally

[francoismg]

Draft pr is available here galaxyproject/galaxy#19003

Spotted a small issue when working with multiple files and trying to remove a file from plot, was working before so shouldn't be too much trouble to fix, will make pr ready to review after that if it's ok with you

[francoismg]

Bug has been fixed, can make the pr ready for review if ok with you

[francoismg]

Just spotted some inconsistencies when using user defined arf and rmf files names for spectrum data (sometimes processed columns are not added to available columns for axis), will fix that and then mark galaxy pr ready for review

[francoismg]

Following issues related to previously mentioned bug have been fixed :

• Saving file specific settings (product type, rmf and arf file) now work as intended
• Processed spectrum columns are now created properly
• File settings are now correctly bound to the file when it moves from available and current lists
• Files settings are now correctly displayed when switching file details
• Bug randomly generating undefined files is fixed
• Issues with file being duplicated in lists and dropdowns is fixed

Some duplication issues seemed to be linked to some race condition and a component not being properly recycled so now a change in file state (loaded -> available <-> current) removes the file details component to avoid the issue

Will publish the new package, push changes, update the galaxy pr and mark it ready for review

[dsavchenko]

As long as the package is already published, probably it's time to merge this PR into master?

[volodymyrss]

@francoismg , as soon as you are ready!

[francoismg]

@francoismg , as soon as you are ready!
sorry had miss the notification, yes sure we can merge it, there's still one review pending before we can use the merge button though

[francoismg]

No my bad it's all good