Add Binary APIs SIP

content/binary-api.md

@@ -0,0 +1,235 @@
+---
+layout: sip
+permalink: /sips/:title.html
+stage: pre-sip
+status: submitted
+title: SIP-NN - Binary APIs
+---
+
+**By: Author Nicolas Stucki**
+
+## History
+
+| Date          | Version            |
+|---------------|--------------------|
+| Feb 27 2022   | Initial Draft      |
+
+## Summary
+
+This proposal introduces the `@binaryAPI` and  `@binaryAPIAccessor` annotations on term definitions. The purpose of binary APIs is to have publicly accessible definitions in generated bytecode for definitions that are private or protected.
+
+
+## Motivation
+
+### Provide a sound way to refer to private members in inline definitions
+
+Currently, the compiler automatically generates accessors for references to private members in inline definitions. This scheme interacts poorly with binary compatibility. It causes the following three unsoundness in the system:
+* Changing any definition from private to public is a binary incompatible change
+* Changing the implementation of an inline definition can be a binary incompatible change
+* Removing final from a class is a binary incompatible change
+
+You can find more details in https://github.com/lampepfl/dotty/issues/16983
+
+### Avoid duplication of inline accessors
+
+Ideally, private definitions should have a maximum of one inline accessor, which is not the case now.
+When an inline method accesses a private/protected definition that is defined outside of its class, we generate an inline in the class of the inline method. This implies that accessors might be duplicated if a private/protected definition is accessed from different classes.
+
+### Removing deprecated APIs
+
+There is no precise mechanism to remove a deprecated method from a library without causing binary incompatibilities. We should have a straightforward way to indicate that a method is no longer publicly available but still available in the generated code for binary compatibility.
+
+```diff
+- @deprecated(...) def myOldAPI: T = ...
++ private[C] def myOldAPI: T = ...
+```
+
+
+## Proposed solution
+
+<!-- This is the meat of your proposal. -->
+### High-level overview
+
+This proposal introduces 2 the `@binaryAPI` and `@binaryAPIAccessor` annotations, and changes adds a migration path to inline methods.
+
+#### `@binaryAPI` annotation
+
+A binary API is a definition that is annotated with `@binaryAPI` or overrides a definition annotated with `@binaryAPI`.
+This annotation can be placed on `def`, `val`, `lazy val`, `var`, `object`, and `given` definitions.
+A binary API will be publicly available in the bytecode.
+
+This annotation cannot be used on `private`/`private[this]` definitions.
+
+Removing this annotation from a non-public definition is a binary incompatible change.
+
+Example:
+
+~~~ scala
+class C {
+  @binaryAPI private[C] def packagePrivateAPI: Int = ...
+  @binaryAPI protected def protectedAPI: Int = ...
+  @binaryAPI def publicAPI: Int = ... // warn: `@binaryAPI` has no effect on public definitions
+}
+~~~
+will generate the following bytecode signatures
+~~~ java
+public class C {
+  public C();
+  public int packagePrivateAPI();
+  public int protectedAPI();
+  public int publicAPI();
+}
+~~~
+
+#### `@binaryAPIAccessor` annotation
+
+A binary API with accessor is a definition that is annotated with `@binaryAPIAccessor`.
+This annotation can be placed on `def`, `val`, `lazy val`, `var`, `object`, and `given` definitions.
+The annotated definition will get a public accessor.
+
+This can be used to access `private`/`private[this]` definitions within inline definitions.
+
+Example:
+~~~ scala
+class C {
+  @binaryAPIAccessor private def privateAPI: Int = ...
+  @binaryAPIAccessor def publicAPI: Int = ...
+}
+~~~
+will generate the following bytecode signatures
+~~~ java
+public class C {
+  public C();
+  private int privateAPI();
+  public int publicAPI();
+  public final int C$$inline$privateAPI();
+  public final int C$$inline$publicAPI();
+}
+~~~
+
+Note that the change from `private[this]` to package private, protected or public is a binary compatible change.
+Removing this annotation is a binary incompatible change.
+
+#### Binary API and inlining
+
+If there is a reference to a binary API in an inline method we can use the definition without needing an inline accessor.
+
+Example 3:
+~~~ scala
+class C {
+  @binaryAPI protected def a: Int = ...
+  protected def b: Int = ...
+  inline def foo: Int = a + b
+}
+~~~
+before inlining the compiler will generate the accessors for inlined definitions
+~~~ scala
+class C {
+  @binaryAPI protected def a: Int = ...
+  protected def b: Int = ...
+  final def C$inline$b: Int = ...
+  inline def foo: Int = a + C$inline$b
+}
+~~~
+
+Note that if the inlined member is `a` would be private, we would generate the accessor `C$inline$a`, which happens to be binary compatible with the automatically generated one.
+This is only a tiny mitigation of binary compatibility issues compared with all the different ways accessors can be generated.
+
+### Specification
+
+<!-- A specification for the proposed changes, as precise as possible. This section should address difficult interactions with other language features, possible error conditions, and corner cases as much as the good behavior.
+
+For example, if the syntax of the language is changed, this section should list the differences in the grammar of the language. If it affects the type system, the section should explain how the feature interacts with it. -->
+
+We must add `binaryAPI` and `binaryAPIAccessor` to the standard library.
+
+```scala
+package scala.annotation
+
+final class binaryAPI extends scala.annotation.StaticAnnotation
+final class binaryAPIAccessor extends scala.annotation.StaticAnnotation
+```
+
+#### `@binaryAPI` annotation
+
+* Only valid on `def`, `val`, `lazy val`, `var`, `object`, and `given`.
+* TASTy will contain references to non-public definitions that are out of scope but `@binaryAPI`. TASTy already allows those references.
+* Annotated definition will be public in the generated bytecode. Definitions should be made public as early as possible in the compiler phases, as this can remove the need to create other accessors. It should be done after we check the accessibility of references.
+
+
+#### `@binaryAPIAccessor` annotation
+
+* Only valid on `def`, `val`, `lazy val`, `var`, `object`, and `given`.
+* An public accessor will be generated for the annotated definition. This accessor will be named `<fullClassName>$$inline$<definitionName>`.
+
+#### Inline
+
+* Inlining will not require the generation of an inline accessor for binary APIs.
+* Inlining will not require the generation of a new inline accessor, it will use the binary API accessors.
+* The user will be warned if a new inline accessor is automatically generated.
+  The message will suggest `@binaryAPI` or `@binaryAPIAccessor` and how to fix potential incompatibilities.
+  In a future version, these will become an error.
+
+### Compatibility
+
+<!-- A justification of why the proposal will preserve backward binary and TASTy compatibility. Changes are backward binary compatible if the bytecode produced by a newer compiler can link against library bytecode produced by an older compiler. Changes are backward TASTy compatible if the TASTy files produced by older compilers can be read, with equivalent semantics, by the newer compilers.
+
+If it doesn't do so "by construction", this section should present the ideas of how this could be fixed (through deserialization-time patches and/or alternative binary encodings). It is OK to say here that you don't know how binary and TASTy compatibility will be affected at the time of submitting the proposal. However, by the time it is accepted, those issues will need to be resolved.
+
+This section should also argue to what extent backward source compatibility is preserved. In particular, it should show that it doesn't alter the semantics of existing valid programs. -->
+
+The introduction of the `@binaryAPI` and `@binaryAPIAccessor` do not introduce any binary incompatibility.
+
+Using references to `@binaryAPI` and `@binaryAPIAccessor` in inline code can cause binary incompatibilities. These incompatibilities are equivalent to the ones that can occur due to the unsoundness we want to fix. When migrating to binary APIs, the compiler will show the implementation of accessors that the users need to add to keep binary compatibility with pre-binaryAPI code.
+
+A definition can be both `@binaryAPI` and `@binaryAPIAccessor`. This would be used to indicate that the definition used to be private, but now we want to publish it as public. The definition would become public, and the accessor would be generated for binary compatibility.
+
+### Other concerns
+
+<!-- If you think of anything else that is worth discussing about the proposal, this is where it should go. Examples include interoperability concerns, cross-platform concerns, implementation challenges. -->
+* Tools that analyze inlined TASTy code might need to know about `@binaryAPI`. For example TASTy MiMa.
+
+### Open questions
+
+<!-- If some design aspects are not settled yet, this section can present the open questions, with possible alternatives. By the time the proposal is accepted, all the open questions will have to be resolved. -->
+
+#### Question 1
+Should `@binaryAPIAccessor` accessors be named `<fullClassName>$$<definitionName>`? This encoding would match the names of `trait` accessor generated for private definition. We could use a single accessor instead of two. This would introduce an extra binary incompatibility with pre-binaryAPI code.
+
+#### Question 2
+```scala
+class A:
+  @binaryAPIAccessor protected def protectedDef: Int = ...
+class B extends A:
+  override protected def protectedDef: Int = ...
+  inline def inlinedDef: Int =
+    // Should this use the accessor of generated for `A.protectedDef`? Or should we warn that `protectedDef` should be a `@binaryAPI`
+    protectedDef
+```
+
+## Alternatives
+
+<!-- This section should present alternative proposals that were considered. It should evaluate the pros and cons of each alternative, and contrast them to the main proposal above.
+
+Having alternatives is not a strict requirement for a proposal, but having at least one with carefully exposed pros and cons gives much more weight to the proposal as a whole. -->
+
+### Only add `@binaryAPI`
+This would simplify the system and the user interaction with this feature. The drawback is that we could not access `private[this]` definitions in inline code. Users would need to use `private[C]` instead, which could cause name clashes.
+
+### Only add `@binaryAPIAccessor`
+This would simplify the system and the user interaction with this feature. The drawback is that we would add code size and runtime overhead to all uses of this feature. It would not solve the [Removing deprecated APIs](#removing-deprecated-apis) motivation.
+
+## Related work
+
+<!-- This section should list prior work related to the proposal, notably:
+
+- A link to the Pre-SIP discussion that led to this proposal,
+- Any other previous proposal (accepted or rejected) covering something similar as the current proposal,
+- Whether the proposal is similar to something already existing in other languages,
+- If there is already a proof-of-concept implementation, a link to it will be welcome here. -->
+* Proof of concept: https://github.com/lampepfl/dotty/pull/16992
+* Initial discussions: https://github.com/lampepfl/dotty/issues/16983
+
+<!-- ## FAQ -->
+
+<!-- This section will probably initially be empty. As discussions on the proposal progress, it is likely that some questions will come repeatedly. They should be listed here, with appropriate answers. -->