Migrate Mill to Scala 3

[Iltotore]

Scala 3 was released more than a year ago and the community quickly started to adopt it (according to Scaladex library count by version).

Moving to Scala 3 has several advantages for Mill:

• Make use of the new indentation based syntax which seems (to me) more suited for build scripts.
• Faster compilation
• Replace some macro usages like task-specific methods (e.g T.dest) in favour of context functions.
• Where macros are still needed, use Scala 3's macro, cleaner and less "magical".
• As far as I know, the move would mainly impact the implementation and not the API itself.

Note that the migration comes with cost:

• Need to rewrite all macros and possibly other places
• Make sure the dependencies used by Mill support Scala 3

I guess this issue will take time to be fully addressed but still a good goal.

[lefou]

Yeah, in the past some of us worked hard to get all dependencies of Mill migrated to Scala 3. Yet, as you already noticed, Mill relies on the macro system to lift the task notation into applicatives. I guess it need some brave souls to start to hack a proof of concept and show how we could do it in Scala 3. One result could also be, to clearly see what will not work, so we can start thinking about solutions.

[Iltotore]

I have to check how Mill exactly "registers" tasks. For the applicative part, I should be able to produce a small PoC/example once available (in ~5 weeks). It should be similar to Odersky's talk about representing monads with capabilities (record)

[Iltotore]

I have to check how Mill exactly "registers" tasks. For the applicative part, I should be able to produce a small PoC/example once available (in ~5 weeks). It should be similar to Odersky's talk about representing monads with capabilities (record)

School took more time than I expected. I will try to find the time to make a small PoC for December February.

[Iltotore]

I started to craft a small PoC few days ago (I am late 😅 ). I am currently succeeding and got the task name using the method (thanks Sourcecode) and dependency detection/applicative:

def sayHello = T {
  T.log("hello")
}

def whichDir = T {
  T.wd
}

def test = T {
  sayHello()
  whichDir()
}

test match //Print task name or the task object
  case Task.Named(name, _) => println(s"Task named $name")
  case task => println(s"Anonymous task: $task")

for input <- test.inputs do //Print deps or their name
  input match
    case Task.Named(name, _) => println(s"Dep named $name")
    case task => println(s"Anonymous dep: $task")

I get:

Task named test
Dep named sayHello
Dep named whichDir

Task-specific methods also work and are not callable outside of T using Scala 3's context functions:

def whichDir = T {
  T.wd //OK
}

T.wd

[error] 22 |    T.wd
[error]    |        ^
[error]    |        This method is only usable in a task or given a `TaskContext`
[error] one error found

I however have some questions about Mill's internal behaviour:

• Does Task#inputs have transitive inputs or only direct ones?
• Does Task#inputs can have duplicates? If so, why is it a Seq and not a Set?
• Are tasks dependencies/inputs pre-executed before executing the task or are they executed at call-site? For example:

def test = T {
  println("test")
  myDepTask()
}

Is myDepTask executed before test or at call-site after the println?

[lefou]

By using a task's apply method, you are expressing a dependency to that task, it will end up in the inputs of the current task.

When you run a task, Mill is first creating a task graph by traversing all dependencies. These form a DAG. Then we calculate an execution plan which means, task with dependencies run after their dependencies. The call to your dependencies apply method will return the actual result of that dependency task.

For your example, it means, myDepTask is already finish before the println.

[Iltotore]

Thank you for the explaination but I still don't understand why inputs is a Seq and not a Set because from what I understand, it shouldnt contain duplicates.

[lefou]

TBH, I still feel to have some incomplete understanding of the details, but I think it's because how we detect and map found dependencies to inputs in the macros. I think we simply enumerate all deps (even when they are used multiple times) and map them to the inputs. Later, at evaluation time, we map those inputs back to the results of these deps. I can't tell whether this is because of some technical limitation or to keep things at a manageable size or because it just worked good enough. (I myself tried one time to change it, but didn't managed it. But at least I tried to hide it by de-duplicating it when reporting the inputs to the user with the inspect command.) We never evaluate a target twice, so having the same input multiple times is probably no big issue performance wise.

[Iltotore]

Small update about the work:

Last week I had time to work on the task system. The project is a very minimalist version of Mill built from scratch (still picking some Mill utils like Agg or Graph) in Scala 3. The goal is to show how can some Mill parts (like macros) can be migrated to Scala 3.

I tried to make the project structure look as similar as possible to Mill's but again, this project is very minimalist.

The migration of Scala 3 involve two main parts: task dependency detection and task-restricted methods (like T.dest).

Task-restricted methods

I "simply" used Dotty's [Context Functions] to restrict make some functions only usable in a task body aka given a TaskContext.

Note: files are sampled/truncated.

Task.scala

enum Task[+T]:
  case Literal(inputs: Seq[Task[?]], body: TaskContext => Result[T])
  case Mapped[A, +B](source: Task[A], f: TaskContext => A => Result[B]) extends Task[B]
  case Sequence[+A](tasks: Seq[Task[A]]) extends Task[Seq[A]]
  case Named(name: String, underlying: Task[T])

TaskContext.scala

@implicitNotFound("This method is only usable in a task or given a `TaskContext`")
case class TaskContext(dest: Path, logger: Logger, results: Map[Task[?], Any], values: Seq[?]):

T.scala

//SourceCode is used to get the name of the task (aka method defining it)
inline def apply[T](inline body: TaskContext ?=> T)(using name: sourcecode.Name): Task[T] =
    Task.Named(name.value, applicative(body)) //applicative is a macro for task dependencies explained further.

def log(msg: String)(using ctx: TaskContext): Unit =
    ctx.logger.log(Level.INFO, msg)

def dest(using ctx: TaskContext): Path = ctx.dest

def sayHello = T {
  T.log("hello world") //OK
}

T.log("hello world") //Compile-time error: This method is only usable in a task or given a `TaskContext`

Task dependencies

This part is similar to Mill's. Term#children used in the applicative/dependency discovering macro no longer exists in Scala 3 so I had to make a small recursive visitor instead:

inline def applicative[T](inline f: TaskContext ?=> T): Task[T] =
  ${ applicativeImpl[T]('f) }

def applicativeImpl[T](expr: Expr[TaskContext ?=> T])(using Quotes, Type[T]): Expr[Task[T]] =

  import quotes.reflect.*

  def recTree(tree: Tree): Expr[List[Task[?]]] = tree match
    case statement: Statement => recStatement(statement)
    case caseDef: CaseDef     => recCase(caseDef)
    case _                    => '{ Nil }

  def recCase(caseDef: CaseDef): Expr[List[Task[?]]] = caseDef match
    case CaseDef(_, Some(guard), thenp) =>
      '{ ${ recTerm(guard) } ++ ${ recTerm(thenp) } }
    case CaseDef(_, None, thenp) => recTerm(thenp)

  def recStatement(statement: Statement): Expr[List[Task[?]]] = statement match
    case term: Term             => recTerm(term)
    case definition: Definition => recDef(definition)
    case _                      => '{ Nil }

  //Other similar methods like recTerm...

  '{ Task.Literal(${ recTerm(expr.asTerm) }, ctx => Result.Success(${ expr }(using ctx))) }

Then for this code:

def helloMessage = T {
  "Hello world"
}

def whichDir = T {
  T.dest
}

def both = T {
  T.log(helloMessage())
  T.log(s"whichDir's destination is ${whichDir()}")
}

both's dependencies are helloMessage and whichDir.

To choose between executing the task when using Task#apply (when in the REPL for example) and declaring a dependency/getting the task's value (when in a task body), an ApplyHandler typeclass is used just like Mill in its current state.

That's all for the moment. Another brave soul started to create the module system. I will help them once available again. The unfinished project (not even a README!) is here.

[Iltotore]

I finally had free time again to work on the project. I realized that module detection/management is entirely done via Java runtime reflection which should not cause any problems in Scala 3 (indeed, I tested).

Is there any other part in Mill involving macros that might be hard to migrate to Scala 3?

[lihaoyi]

Three things that come to mind:

1. Discover.scala
2. Cross.scala
3. Parsers.scala

[Iltotore]

Surprisingly enough, the make macro in Cross.scala is really harder to write in Scala 3. I also stumbled upon what seems to be a compiler bug. I will link the issue to this discussion if it is confirmed to be a bug.

EDIT: scala/scala3#17349

[Iltotore]

Another blocking issue: scala/scala3#17375

[lefou]

For a nice API experience, we currently use name shadowing for the inner Tests traits in JavaModule and derived modules. This technique is no longer supported by Scala 3. In preparation, we already define each of these Tests trait as a uniquely named trait, e.g. JavaModuleTests or ScalaModuleTests, and define a dummy trait Tests extends ScalaModuleTests.

/home/lefou/work/opensource/mill/scalalib/src/mill/scalalib/ScalaModule.scala:31:9: shadowing a nested class of a parent is unsupported but trait Tests shadows trait Tests defined in trait JavaModule; rename the class to something else
   trait Tests extends ScalaModuleTests
         ^

Maybe, we should deprecated these Tests traits in 0.11 to avoid later migration issues?

[lihaoyi]

I would be in favor of doing so. The shadowed Test traits are very error prone, because they look like they get overriden but actually get shadowed. This makes it terribly easy to accidentally be running tests on the wrong platform

trait Foo extends ScalaModule{
  def scalaVersion = "2.13.10"
  object tests extends Tests
}
object bar extends Foo
object qux extends Foo with ScalaJSModule // qux.run runs on node.js but qux.test runs on the JVM

This is made especially bad because running tests on the wrong platform often doesn't fail! Both Scala.js and Scala-Native do generate bytecode that mostly works, so unless you are explicitly exercising some JS/Native-specific functionality you may see your tests passing and be none-the-wiser that you're actually running them on Scala-JVM without exercising Scala.js code at all!

[lefou]

Ok, we can @deprecated it for 0.10.13 and remove all Tests traits in 0.11.0, as those won't compile even with -Xsource:3.

[yawaramin]

Is it a goal to migrate Mill to Scala 3? In that case, maybe we can convert this discussion into an issue?

[lihaoyi]

The work is largely complete in #3369, just waiting for Mill 0.12.0 to be cut before we merge it since it breaks bincompat

[Iltotore]

Nice. Can't wait for the talk about it in a week 😁