parallel operations never terminate in the REPL

[julienrf]

Tested with scala 2.13.0-M1 and parallel-collections 0.1.2.

The following expression does not terminate in the REPL:

(1 until 1000).par.filter(n => n % 3 == 0).count(n => n.toString == n.toString.reverse)

With 2.11.8 it terminates. With 2.12.3 it doesn’t terminate either.

However, if I put the same expression in a program its evaluation does terminate.

[retronym]

This is a consequence of the new lambda encoding:

https://stackoverflow.com/questions/15176199/scala-parallel-collection-in-object-initializer-causes-a-program-to-hang/15176433#15176433

scala/bug#8119

scala -Yrepl-classbased is a workaround:

⚡ scala -Yrepl-class-based
Welcome to Scala 2.12.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112).
Type in expressions for evaluation. Or try :help.

scala> (1 until 1000).par.filter(n => n % 3 == 0).count(n => n.toString == n.toString.reverse)
res0: Int = 36

However, that its own problems, so we can't make it the default. For the 2.13 REPL, we're aiming for a best-of-both-worlds solution.

[retronym]

Another workaround:

scala> def foo = (1 until 1000).par.filter(n => n % 3 == 0).count(n => n.toString == n.toString.reverse)
foo: Int

scala> foo
res0: Int = 36

[DarkDimius]

@retronym this is not inherent to the Java8 lambda encoding. This is simply a bug due to way how labmdalift works in Scalac. Dotty compiled code does use lambdas but does not have issue with the deadlock in static initializer.

Similar discussion happened here: typelevel/scalacheck#290
Fix for this issue in Dotty: https://github.com/lampepfl/dotty/pull/628/files

[retronym]

@DarkDimius The PR you linked doesn't describe the change in enough detail for me to understand.

I've produced a bytecode diff to contrast scalac and dotty for this test case. Could you help connect the dots for me?

[retronym]

Okay, now I see the difference. The lambda in scalac is backed by a static method, and in dotty it is backed by an instance method for a lambda that captures this. Seems like a tradeoff between how deadlock prone things are vs how much additional baggage is carried around when serializing the lambda. I can see that for lambdas lexically enclosed in a modules constructor the dotty approach might a better default. What rules is applied to decide whether static is used?

[odersky]

I believe the criterion is which this-references the lambda contains. If it contains references to an enclosing this, it goes in the same class. if it does not contain any this references, it becomes static.

[DarkDimius]

I believe the criterion is which this-references the lambda contains. If it contains references to an enclosing this, it goes in the same class. if it does not contain any this references, it becomes static.

There's one additional rule: it becomes static unless it's owned by a top-level module class.