Is there any performance difference between Dataview & DataviewJS?

[rishifter]

I'm using Dataview for a lot of aggregation and in one particular case, it is making my note take a long time to open and there's some scroll lag also. The file in question has the query below,

TABLE
  default(people, "") as People
, file.outlinks as Links
, sort(info.genre) as Genre
, link(info.actors) as Actors
, link(info.director) as Director
, link(info.writer) as Writers
, rating as "★"
, info.imdbRating as IMDb
, info.rottentomatoes as RT
, info.metascore as Meta
, info.runtime as "🕒"
, elink(info.imdbURL, "") as 🔗
FROM "Films"
sort file.name

The folder "Films" has only 66 files right now but each file has a lot of YAML (fetched using OMDB with Templater). I'd not assume it to slow down for such few files so I'm thinking if there can be some optimisation. I was wondering if some of the functions I'm using default(), sort(), link(), elink() can be avoided if they could cause performance issues. Or it could be the YAML with around 30 fields? Would you recommend switching to JS to test if that is more performative or have you done any comparisons yourself? I was also wondering if there was a way to comment lines in the query so I can disable certain columns from loading usually but still have access to them when required.

I'm willing to do any sort of testing to get this working and figure out the most optimised queries for either of too many files. columns or just using js. Thanks.

[blacksmithgu]

DataviewJS skips the query parsing/execution, so I would not be surprised if it is faster, though it may not be substantially so - I'm guessing the "slow" part here is the actual rendering of the query itself. See what happens if you remove a few fields in terms of performance.

I'll have to investigate alternatives for rendering faster; optimally I would make it multi-threaded though I'm not sure how easy that is to do.

[rishifter]

I'll do some tests both with removing a few fields as well as with DataviewJS over the weekend and report back if there are significant differences. Just to be sure, when you say remove fields, do you refer to removing them from the query or from the files themselves? Does Dataview have to parse all fields in the files even if the query doesn't call for them? If so, I could consider reducing the fields sooner than later since I'm only at 66 films now and once it grows, it becomes time-consuming to change things.

Another thing you could consider, which others have probably asked for as well is if there was a way to manually reload a table render, that could solve performance issues with large datasets. I'm imagining Dataview caching the data at first render and only showing that unless the user manually reloads or refetches the data. This could very likely be a query option which could work override the refresh time set globally in settings to infinity so it never reloads. This way even if one particular query takes 3-5 seconds, one could just render it manually whenever they need to.

[blacksmithgu]

Dataview amortizes out the cost of collecting all the data by constantly doing it in the background as you change files, so the number of fields in your pages shouldn't really matter. The only expense is the actual execution of the query over the results, and then rendering it. There should be no problem with having lots of fields on your pages.

As for the second question, dataview will only re-render a query if the dataview index changes in the next release, which will help with this somewhat. Changing the refresh interval per-query is on my docket.

[nfelnlp]

Hi @blacksmithgu ,
first of all, thank you very much for your hard work on this beautiful tool!
I was about to start a new thread, but I think my question fits here quite nicely:

I'm working with really large tables (~1k entries) and the dataviewjs scripts in the background (called via the amazing dv.view()) do lots of computations to produce numbers (similarity measures) shown in the tables. I have successfully optimized them a bit, but as of now, these computations still take about 20-30 seconds which gets super annoying when it recomputes every time I change a single character anywhere in the vault. Before it displays the actual resulting table, it's a blank page.

I thought about writing a script that produces CSV files from the resulting tables and then reading them in again via dv.io.csv().

However, maybe there's a better option: Is there a way in JS to stop a once rendered dataview table from rerendering?
I already use checkmarks (tasks) in some Markdown files whose status can be read using dv.current().file.tasks. Based on the status (active or not), some parts of the JS code called via dv.view() are executed or not. Ideally, this rerendering could somehow be controlled with this or an option in the settings of dataview.

Thank you!

[blacksmithgu]

@nfelnlp There isn't a way to directly block re-renders currently, though I may be able to hack something together for your specific use case. Would you mind making an issue for this?

[nfelnlp]

Wow, thanks so much for your instant reply!! Yes, I'll create one.