Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A lot of tokenization happening at the same time #7003

Open
tgodzik opened this issue Dec 3, 2024 · 1 comment
Open

A lot of tokenization happening at the same time #7003

tgodzik opened this issue Dec 3, 2024 · 1 comment
Assignees
@tgodzik
Labels
bug Something that is making a piece of functionality unusable fixed-upstream Tag a ticket with this when a fix upstream has been completed and we can now address the ticket

Comments

@tgodzik
Copy link
Contributor

tgodzik commented Dec 3, 2024

Describe the bug

User reported on Discord that Metals sometimes take a lot of CPU and managed to get a jstack:

https://gist.github.com/felher/9939f409ab3f7d0df5e5abe8f06ec1e2

Looks like there is too much threads at: scala.meta.internal.tokenizers.LegacyScanner.reportIllegalCharacter$1(LegacyScanner.scala:237)

We probably need to optimize it

Expected behavior

No response

Operating system

None

Editor/Extension

None

Version of Metals

v1.4.1

Extra context or search terms

memory cpu usage tokenization
@tgodzik tgodzik self-assigned this Dec 3, 2024
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 3, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
2ec7dc2 
Related to scalameta/metals#7003
@tgodzik tgodzik mentioned this issue Dec 3, 2024
Merged
@tgodzik tgodzik added bug Something that is making a piece of functionality unusable upstream-fix-needed Waiting on a fix upstream labels Dec 4, 2024
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 4, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
3db9500 
Related to scalameta/metals#7003
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 4, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
9e34d7b 
Related to scalameta/metals#7003
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 4, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
0b355e0 
Related to scalameta/metals#7003
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 4, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
ddd08aa 
Related to scalameta/metals#7003
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 4, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
ea9af07 
Related to scalameta/metals#7003
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 5, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
21adfce 
Related to scalameta/metals#7003
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 5, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
4179938 
Related to scalameta/metals#7003
tgodzik added a commit to tgodzik/scalameta that referenced this issue Dec 5, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars
1226c10 
Related to scalameta/metals#7003
kitbellew pushed a commit to scalameta/scalameta that referenced this issue Dec 5, 2024
@tgodzik
improvement: Don't use format since it seems to break at some chars (#…
5d08839 
…4053)

Related to scalameta/metals#7003
@kasiaMarek kasiaMarek added fixed-upstream Tag a ticket with this when a fix upstream has been completed and we can now address the ticket and removed upstream-fix-needed Waiting on a fix upstream labels Dec 11, 2024
@kasiaMarek
Copy link
Contributor

Possible solutions to investigate:

  • can we use trees to get tokens?
  • should we cache tokens (same as we do for trees).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tgodzik tgodzik

Labels
bug Something that is making a piece of functionality unusable fixed-upstream Tag a ticket with this when a fix upstream has been completed and we can now address the ticket
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tgodzik @kasiaMarek