Macros lead to wrong coverage reports

[samuele-buro-sonarsource]

Compiler version

MWE based on 3.5.1.

Notes

• We also managed to reproduce it on 3.6.2.
• The issue also seems unrelated to the build plugin of choice as it appears with both sbt-scoverage and gradle-scoverage (reproducer uses the former).

Minimized code

An MWE can be found here. The reproducer is minimal but requires interaction between multiple files.

Code overview

The code is organized as follows:

src/main/scala
└── org
    └── example
        ├── App.scala
        ├── NumUtils.scala
        ├── greeting
        │   └── Greeting.scala
        └── macros
            └── Macros.scala

The key points are:

• Greeting.scala defines the logic to greet users (greet: Option[String] => String);
• Macros.scala defines a macro to decorate strings at compile time (decorate: String => String);
• App.scala greets an undefined user and decorates a string using the macro (println(decorate(greet(None))));
• NumUtils.scala defines a simple function isEven, which is completely unrelated to the rest of the code.

Output

The coverage report fails to include Greeting.scala, Macros.scala, and NumUtils.scala. Moreover, it shows 100%
coverage for App.scala even though it should be 0% (main method is never called in the tests).

Expectation

We expect to get an equivalent report as if macros were converted to ordinary methods. As explained in the README,

To make sure that macros are the cause of the issue, simply make the decorate macro into a regular method:

def decorate(s: String): String = s">>> $s <<<"

and re-run tests and report generation. Coverage is now correctly computed.

[jchyb]

Minimised for scalac:

// MacroDef.scala
object App:
  def main(args: Array[String]): Unit =
    println(Macros.decorate(Greeting.greet()))

// MacroRun.scala
import scala.quoted.{Expr, Quotes}

object Macros:
  inline def decorate(inline s: String): String = ${ decorateQuotes('s) }
  def decorateQuotes(s: Expr[String])(using Quotes): Expr[String] = '{ ">>> " + $s + " <<<" }

object Greeting:
  def greet() = "hello"

Running with scalac MacroDef.scala MacroCall.scala --coverage-out:. generates scoverage.coverage, where only possible calls to MacroRun.scala are recorded.:

# Coverage data, format version: 3.0
# Statement data:
# - id
# - source path
# - package name
# - class name
# - class type (Class, Object or Trait)
# - full class name
# - method name
# - start offset
# - end offset
# - line number
# - symbol name
# - tree name
# - is branch
# - invocations count
# - is ignored
# - description (can be multi-line)
# '' sign
# ------------------------------------------
0
MacroDef.scala
<empty>
App
Object
<empty>.App
main
70
112
4
println
Apply
false
0
false
println(Macros.decorate(Greeting.greet()))

1
MacroDef.scala
<empty>
App
Object
<empty>.App
main
78
111
4
+
Apply
false
0
false
Macros.decorate(Greeting.greet())

2
MacroDef.scala
<empty>
App
Object
<empty>.App
main
78
111
4
+
Apply
false
0
false
Macros.decorate(Greeting.greet())

3
MacroDef.scala
<empty>
App
Object
<empty>.App
main
94
110
4
greet
Apply
false
0
false
Greeting.greet()

4
MacroDef.scala
<empty>
App
Object
<empty>.App
main
28
36
3
main
DefDef
false
0
false
def main


What seems to happen is that our coverage compiler phase is not correctly handling file suspension mechanism caused by having macros and calls to them in the same compilation:

• First both files are being compiled, after running into a macro call, the compiler suspends the file with macro call.
• The file with macro def gets compiled, and has a scoverage.coverage file generated (with indexes 0-4)
• The file with a macro call gets compiled, also has a scoverage.coverage file generated (with indexes 0-4), incorrectly overwriting the previous file
• Not only do we lose information about indexes from the previously compiled files, those previously compiled classifies point to the wrong indexes

A workaround while we try to fix this is to have the macro calls and macro defs in separate projects