Add ink! analyzer (phase 2)

[davidsemakula]

Project Abstract

This application is for a follow-up grant for ink! Analyzer (PR).

ink! analyzer is a collection of modular and reusable libraries and tools for semantic analysis of ink! smart contract code.

ink! analyzer aims to improve ink! language support in integrated development environments (IDEs), source code editors and other development tools by providing modular and reusable building blocks for implementing language features (e.g. diagnostic errors, quick fixes, code completion suggestions, code/intent actions and hover content e.t.c) for the ink! programming language which is used for writing smart contracts for blockchains built on Substrate.

Relying on only generic Rust language support in IDEs, code editors and other development tools has some significant limitations for the developer experience including:

• No language support (e.g. diagnostic errors/warnings and quick fixes) for ink!'s domain specific semantic rules for smart contracts (e.g. exactly one #[ink(storage)] struct, at least one #[ink(message)] method and the same for #[ink(constructor)], ink! attributes should be applied to items of the correct type e.t.c).
• Inconsistent editor experience with issues like no code completion and/or hover content for some ink! attribute arguments (e.g #[ink(payable)]) because macro expansion/name resolution and trait resolution are hard problems for generic IDE/code editor tools (see also https://rust-analyzer.github.io/blog/2021/11/21/ides-and-macros.html).
• No language support (e.g. go to definition, find references and rename/refactor) for ink! specific syntax like paths in ink! attribute argument values (e.g. env values).

To solve the above challenges and improve ink! language support in IDEs, code editors and other development tools, ink! analyzer created two main components:

• A modular domain-specific semantic analysis library for ink! built on a resilient and lossless parser.
• A Language Server Protocol (LSP) implementation built on top of the aforementioned semantic analysis library.

Along with a Visual Studio Code extension that ships with a bundled ink! Language Server as a showcase and reference implementation for leveraging ink! analyzer components to add ink! language support to any tool with robust LSP client libraries/APIs/modules.

This stage of the project will improve ink! language support features across all components by adding:

• Quick fixes for all existing diagnostic errors and warnings.
• ink_e2e macro support (i.e diagnostics, quickfixes, completions, code actions and hover content).
• Command for creating an ink! project and code/intent actions for inserting code stubs/snippets for relevant ink! entities.
• Go to definition, find references and rename/refactor support for path-based ink! attribute argument values (i.e. env values for #[ink::contract] and environment values for #[ink_e2e::test]).
• Diagnostics that verify that the value of env values for #[ink::contract] and environment values for #[ink_e2e::test] are impl Environment as well as quick fixes to impl Environment for the target item where necessary.
• Inlay hints and signature help/parameter hints for ink! attribute arguments.
• Code/intent actions for "flattening" ink! attributes.

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[dsm-w3f]

@davidsemakula thank you for the grant application. I saw that the scope of this grant is bigger than the previous one, and for me, there are two projects here, one for improving the current version of the Ink! Analyzer plugin and others to create a new version for IntelliJ IDE. Let me ask some questions to start the discussion about this grant proposal.

• Wouldn't be better to split this grant proposal in two with the specific scope of each project? Why did you decide to send them together?
• In the improvement part of the ink! analyzer plugin. Did you already seek users' feedback about the plugin? We really want to people to use Ink! to develop and run smart contracts, and a VSCode plugin such as Ink! analyzer can help with that. However, I would appreciate knowing if the changes proposed are based on user feedback or on assumptions of what the user will need.
• Why did you choose to support another IDE than VSCode at this moment, and why IntelliJ? Are people using Indelij to develop Ink! smart contracts?

[davidsemakula]

thank you for the grant application. I saw that the scope of this grant is bigger than the previous one, and for me, there are two projects here, one for improving the current version of the Ink! Analyzer plugin and others to create a new version for IntelliJ IDE. Let me ask some questions to start the discussion about this grant proposal.

@dsm-w3f Thanks for taking the time to review and for the detailed feedback/questions 🙂.

Just to properly orient the discussion, I want to start by emphasizing that while the VS Code extension is the most user facing part of the project at the moment, the 2 main components remain the semantic analyzer and the language server as these are the fundamental building blocks that also allow language support across a broad range of IDEs, code editors and other development tools.

So ink! analyzer is conceptually better viewed as the semantic analyzer and the language server (with the VS Code extension being just one user interface/frontend), and the proposed improvements are better characterized as first and foremost improvements to these main components (which can be reused by many IDEs, code editors and other development tools including the IntelliJ plugin) rather than just VS Code extension improvements. In this light, you'll notice that the VS Code extension only development accounts for a relatively small portion of the resources in this proposed phase of the project - about ~8% in terms of overall budget (or ~14% in terms non-IntelliJ targeted development) to be precise.

Moving on to the questions, I've re-arranged the order in a way that I think allows the answers to "flow better into each other".

• In the improvement part of the ink! analyzer plugin. Did you already seek users' feedback about the plugin? We really want to people to use Ink! to develop and run smart contracts, and a VS Code plugin such as Ink! analyzer can help with that. However, I would appreciate knowing if the changes proposed are based on user feedback or on assumptions of what the user will need.

Following on from the introductory comments, the improvements should be viewed as semantic analyzer (and language server) improvements first and foremost. From this perspective, I'd frame the first version (i.e. the first grant) of ink! analyzer components as an MVP (Minimum Viable Product) and the improvements in this proposal as taking the project to production readiness.

In this sense, while user feedback is important - and I've started to solicit that (here, here and here, in addition to an announcement/promotion with the Web3 Foundation PR team that should go out early next week) - the proposed improvements in this phase of the project are not subjective plugin/extension features but rather core language support features that users would expect from mature IDE/editor tooling for any programming language.

I'd like to refer to the Semantic Analyzer subsection of the Architectural description of the first grant application (it has a few more details and few more links than the current iteration that may be helpful 🙂) for a detailed explanation for why a separate "compiler-frontend/semantic analyzer" built on separate resilient and lossless parsing and syntax tree libraries is a necessity for a great IDE/editor experience.

This rationale is validated by the existence of rust-analyzer as a separate compiler-frontend/semantic analyzer from the rustc compiler for Rust itself and the relatively recent emergence of slang (also mentioned in the application) with the aim of serving a similar role for Solidity (i.e. a separate compiler-frontend/semantic analyzer from solc with significant funding from the Ethereum ecosystem). In general this a well established pattern industry-wide with similar compiler frontends/semantic analyzers existing for other languages (e.g. Roslyn for C# and lib/Syntax for Swift).

Additionally, I regard the addition of ink! analyzer to the official ink! documentation portal under "Third Party Tools and Libraries" (see https://use.ink/getting-started/ink-analyzer and use-ink/ink-docs#261) as a strong positive signal for the recognition of its utility for the ink! project as a whole.

• Why did you choose to support another IDE than VS Code at this moment, and why IntelliJ? Are people using Indelij to develop Ink! smart contracts?

IntelliJ platform-based IDEs (e.g. IntelliJ IDEA, Android Studio, PyCharm, Webstorm, PhpStorm, CLion, Rider, GoLand and RubyMine) already have excellent Rust support via the IntelliJ Rust plugin. In fact, rust-analyzer itself is largely inspired by IntelliJ Rust plugin as rust-analyzer's creator was a contributor to IntelliJ Rust before creating the rust-analyzer project.

Because of this, the IntelliJ Rust plugin (and thus JetBrains IDEs) has healthy usage among Rust developers (see "Plugins" section of https://blog.jetbrains.com/rust/2023/01/18/rust-deveco-2022-discover-recent-trends/#Plugins). This is in combination with IntelliJ platform-based IDEs still being generally popular among all developers (see https://survey.stackoverflow.co/2022/#most-popular-technologies-new-collab-tools).

Given that ink! is a Rust eDSL, I'm using the healthy usage of the IntelliJ Rust plugin as a proxy for similar interest from ink! developers if similar language support is provided via an IntelliJ plugin.

Anecdotally, while I'm comfortable using VS Code, I personally have paid subscriptions to a few JetBrains products and tend to prefer them for most use cases. However, I did start out my professional software engineering journey before VS Code even existed so that may be the reason for some of my preferences. I don't however believe it's a niche preference among Rust developers given the information above.

In general, having developed an architecture and core components that are IDE/editor agnostic, I think it'd be a shame if it ends up only used with VS Code simply because it's the only well-supported option. In this case, we'd be implicitly answering to the question "Are people using IntelliJ to develop Ink! smart contracts?" ourselves by only providing a great IDE/editor experience on VS Code. So I think it's worthwhile to support the other current major IDE platform at this stage of the project given the excellent support for Rust and healthy use by Rust developers for the IntelliJ platform.

However, I don't plan to develop any additional plugins/extensions/integrations beyond the VS Code extension and IntelliJ plugin as I believe these two cover a broad enough swath of Rust/ink! IDE/editor users while also clearly demonstrating and communicating (with showcases and reference implementations) the utility of ink! analyzer components across multiple IDEs and editors with vastly different architectures.

• Wouldn't be better to split this grant proposal in two with the specific scope of each project? Why did you decide to send them together?

While I understand the reason for this sentiment, given the architecture of the project, the IntelliJ plugin is as much a part of the same project as the VS Code extension as they are both essentially user interfaces/frontends for language features provided by the same semantic analyzer and language server components.

Any perception that the VS Code extension is part of the project while the IntelliJ plugin is another project is simply because the VS Code extension got released first and so now carries that mind share (it's actually a perception I'd like to break as the architecture is intentionally IDE/editor agnostic). You'll also notice that all IntelliJ plugin and VS Code extension milestones are defined in terms of semantic analyzer and language server milestones/deliverables as they both depend on and reuse those components.

So if it makes sense to support the IntelliJ platform, my preference would be to go through the approval process once and then focus on delivering features across the entire component stack at a regular cadence. Splitting off only the IntelliJ plugin, or both the IntelliJ plugin and VS Code extension as separate projects would just mean writing largely the same application again and referencing previously delivered semantic analyzer and language server deliveries (as I'm currently doing in milestone 1 for this application) which would just be largely unnecessary overhead on my end. I'm hoping that my previous deliverables from the first grant demonstrate my ability to deliver high quality, well-tested software with great documentation and at a regular cadence, and inspire enough confidence to approve a bigger grant this time around as this allows me to plan better and allocate time longer term for what I see as a strategic project 🙂.

[dsm-w3f]

@davidsemakula thank you for the answer. I notice that milestones 1, 7, and 8 are related to IntelliJ. Removing that would reduce the price by at least 35k, and still have useful features for the language server and VSCode plugin. Lower-price grants are usually easier to approve. In my opinion, these milestones (1, 7, and 8) could be performed as another project.

Furthermore, for the core functionalities, could you please detail which elements of the Ink! DSL would be included on each one? A list of the elements included could make more tangible the scope of this grant. If you could provide examples of usage and how these features should look in the final interface (plugin) would also be a good way to understand and visualize the scope of the grant.

[davidsemakula]

@dsm-w3f Thanks for feedback.

I notice that milestones 1, 7, and 8 are related to IntelliJ. Removing that would reduce the price by at least 35k, and still have useful features for the language server and VSCode plugin. Lower-price grants are usually easier to approve. In my opinion, these milestones (1, 7, and 8) could be performed as another project.

I've removed all IntelliJ milestones from the proposal.

However, it would still be great to get some feedback/thoughts on the detailed "[why IntelliJ plugin]" answer above for future planning purposes.

Furthermore, for the core functionalities, could you please detail which elements of the Ink! DSL would be included on each one? A list of the elements included could make more tangible the scope of this grant.

I've updated milestone tables to make this a bit more clear, either by describing the methodology or listing the target ink! entities, depending on whichever is more feasible to specify (e.g. creating a list of all the over 74 existing diagnostics errors/warnings for ink! attribute macros - e.g. #[ink::contract] - and ink! attribute arguments - e.g. #[ink(storage)] - for which quick fixes will be added would be rather tedious, so a methodology is described instead). In general, since the goal is core language support, features are defined to be exhaustive i.e. all relevant "elements of the Ink! DSL" are covered for each feature.

If you could provide examples of usage and how these features should look in the final interface (plugin) would also be a good way to understand and visualize the scope of the grant.

In general, the VS Code extension doesn't define any custom user interfaces as these are standard editor/LSP features and therefore just hook into the standard interfaces of the IDE/editor. However, I've added some links to VS Code documentation that includes visual demos of the IDE/editor features to the VS Code milestone details (or similar features if there isn't a demo that's an exact match as most demos are based on JavaScript/TypeScript examples).

[dsm-w3f]

@davidsemakula thank you for the answers and changes. I think now the scope is more clear and concise. I think would be great to have Ink! analyzer available in IntelliJ and other IDEs. However, this should be easier to fund when the tool is more mature and with more users. I'm happy to approve this proposal. I'll mark it as ready to review for the other members of the committee to evaluate it and I will also ask for feedback from Parity Smart Contracts team. Hope to see the results soon.

[davidsemakula]

@dsm-w3f Awesome! thanks for taking the time to review/approve and for the feedback/thoughts.

[keeganquigley]

Thanks for the application @davidsemakula and the deep dive @dsm-w3f I was also impressed with your previous delivery and would be happy to go forward with it as well.

[davidsemakula]

@keeganquigley Awesome! thanks for the swift review/approval and for the positive feedback 🙂

[semuelle]

Thanks for the thorough application and discussion, @davidsemakula. I am also happy to support this application.

Regarding IDEA: My initial thought was similar to @dsm-w3f's, that it would be useful to collect some user data first before moving onto the next platform. Given the immense popularity of VSCode (at least per the Stack Overflow Survey), I think it's okay to stick with it for now. However, IDEA is also fairly popular, so I agree that we should keep it in mind as an option.

Regarding follow-up grants: the next time you apply for a grant for this, feel free to reference your previous applications in relevant parts of the application, so you don't have to repeat everything that was already laid out previously.

[davidsemakula]

@semuelle Awesome! thanks for the swift review/approval and for the detailed feedback 🙂

Regarding IDEA: My initial thought was similar to @dsm-w3f's, that it would be useful to collect some user data first before moving onto the next platform. Given the immense popularity of VSCode (at least per the Stack Overflow Survey), I think it's okay to stick with it for now. However, IDEA is also fairly popular, so I agree that we should keep it in mind as an option.

Makes sense, VS Code definitely seems more popular and that's 100% why I chose to support it first 🙂.

But (Just for discussion 🙂), I also think the fragmented nature of JetBrains/IntelliJ IDEs (i.e. dedicated products for most popular languages/workflows) may make the platform seem smaller in unified surveys like Stack Overflow's than it is in reality. Because you get an aggregated number across all languages for VS Code while for JetBrains/IntelliJ platform the usage number is segmented among the many products for specific languages/workflows. The Rust specific survey suggests a much closer split between rust-analyzer (likely mostly VS Code) and IntelliJ Rust (all JetBrains/IntelliJ IDEs) usage for that specific segment.

Regarding follow-up grants: the next time you apply for a grant for this, feel free to reference your previous applications in relevant parts of the application, so you don't have to repeat everything that was already laid out previously.

Gotcha 👍

[Noc2]

I'm also happy to go ahead with it.

[davidsemakula]

@Noc2 Awesome! thanks for the swift review/approval 🙂

[davidsemakula]

@dsm-w3f @Noc2 @keeganquigley @semuelle There aren't any changes from the last approval(s).
I force pushed to remove changes from an unnecessary merge I'd made for #1926 🙂

[takahser]

@davidsemakula thanks for being flexible regarding the scope and putting emphasis on VS Code. Happy to approve as well!

[github-actions]

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at [email protected] and we'll be happy to collaborate on an announcement about the work you’re doing.

Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀

[davidsemakula]

@takahser Awesome! thanks for the swift review/approval 🙂

[Parasos]

bb