CPU and Memory Usage

[Subtixx]

As I've noted in my review, the CPU and Memory Usage is terrible when having a large project.

5000+ Files => 100% CPU Usage (12 Cores / 24 Threads @ 4.1Ghz!) and about 60-70% Memory Usage (64 GB Total Memory installed)

[ChrisCarini]

Hi @Subtixx - thank you for giving the plug-in a try and opening this issue!

Can you please share some detailed steps to reproduce what you are seeing?

[Subtixx]

I just opened a project with 5000+ Files and the CPU Usage spiked after ~10-15 Seconds

[ChrisCarini]

Do you have an example project you can share please?

[Subtixx]

For obvious reasons I cannot share our proprietary codebase. But I assume this would happen with any very large project. These images I took a couple of minutes ago, since it didn't happen until now again. However the RAM is not high here? I assume because this "burst" of 100% CPU was only like 10-20 Seconds long. The other time when I created this issue, the usage was 100% for ~10-15 Minutes and caused the whole operating system to crawl, and in that 10-15 Minutes it creeped up from 4GB all the way to ~34 GB

[ChrisCarini]

For obvious reasons I cannot share our proprietary codebase.

Of course 😆 I was more hoping you had an OSS example you could share.

But I assume this would happen with any very large project.

Share some that you can reproduce this on that I can try (aka, any OSS codebase)? I've used this plugin on/in https://github.com/JetBrains/intellij-community (which, is currently ~10G clone & ~550k files) and haven't noticed any issues (I'm using IntelliJ, not Rider, however).

Do you mind sharing more details on what you've done exactly to pin-point that it's specifically this plugin that's causing issues, and not something else?

[Subtixx]

Hmm I'll look for an open-source project that could replicate it, the project I'm currently working on is ~26.8GB big and I don't really know how many files. However most files are binary 3D Assets ala FBX, uasset, PNG etc. I'm honestly not doing something special, the full OS slow-downs have started appearing after installing the plugin so I assumed it was the plugin. The project did "fine" (as fine as this big of a IntelliJ Project can be) before

EDIT:
Thinking about it (althought not open-source) the Unreal Engine Codebase could be a candidate

[ChrisCarini]

However most files are binary 3D Assets ala FBX, uasset, PNG etc

Hmm, it's possible something related to the filetypes then. If there is a similar-ish project w.r.t. the type of files, that might be useful.

Do you happen to use git-lfs for any of these assets? If so, I wonder if there is something there, too... 🤔

the full OS slow-downs have started appearing after installing the plugin so I assumed it was the plugin

Sure - correlation != causation, however. Not saying its not the plugin - it certainly could be. But do you mind doing a bit more testing so we can build confidence one way or the other? To start off with, can you try disabling the plugin and see if you can reproduce the issue? Also try uninstalling the plugin and see if the issue goes away (then, try re-installing it and see if the issue comes back right away again).

Thinking about it (althought not open-source) the Unreal Engine Codebase could be a candidate

Essentially, any project I can easily get the source-code to to try out - is UE accessible in that means, or is it restricted? A quick search on GitHub isn't showing anything (but, that doesn't mean it's not on another git hosting service like gitlab). Similar thought to above; does it have a similar composition of file assets as part of the repo, similar to your project?

[Subtixx]

Hmm, it's possible something related to the filetypes then. If there is a similar-ish project w.r.t. the type of files, that might be useful.

Yes I assumed so too. Since some files are in >10MB.

Do you happen to use git-lfs for any of these assets? If so, I wonder if there is something there, too... 🤔

Yes.

Sure - correlation != causation, however. Not saying its not the plugin - it certainly could be. But do you mind doing a bit more testing so we can build confidence one way or the other? To start off with, can you try disabling the plugin and see if you can reproduce the issue? Also try uninstalling the plugin and see if the issue goes away (then, try re-installing it and see if the issue comes back right away again).

You're right though, it does not mean the plugin is the issue! I'll right the suggested things and will report back!

Essentially, any project I can easily get the source-code to to try out - is UE accessible in that means, or is it restricted? A quick search on GitHub isn't showing anything (but, that doesn't mean it's not on another git hosting service like gitlab). Similar thought to above; does it have a similar composition of file assets as part of the repo, similar to your project?

No the Unreal Engine does not have assets. However there are learning projects like Lyra which are huge too and also contain massive amounts of assets. The Repository for UE is only acessible by:

1. creating an Epic Games account
2. Linking the Github account to the epic games account
   More info can be found here