Lambda in REPL (using object-wrappers) + concurrency = deadlock

[scabug]

The following code is a test of generating and using java.util.Spliterator to create a java.util.stream.Stream:

package xp

import java.util.Spliterator
import java.util.function.Consumer

class Aspi(a: Array[String], private val i0: Int, private var i1: Int) extends Spliterator[String] {
  private var i = i0
  def characteristics = Spliterator.SIZED | Spliterator.SUBSIZED | Spliterator.IMMUTABLE
  def estimateSize = i1-i
  override def getExactSizeIfKnown = i1-i
  def tryAdvance(f: Consumer[_ >: String]): Boolean = {
    println(s"Trying at $i [$i0, $i1)")
    if (i >= i1) false
    else { f.accept(a(i)); i += 1; true }
  }
  def trySplit(): Aspi = {
    if (i1-i < 4) {
      println(s"Could not split [$i $i1)")
      null
    }
    else {
      println(s"Splitting from [$i $i1)")
      val child = new Aspi(a, (i+i1+1)/2, i1)
      i1 = child.i0
      child
    }
  }
}

object Test { def main(args: Array[String]) {
  def nu = new xp.Aspi(Array("one", "two", "three", "four", "five"), 0, 5)
  val st = java.util.stream.StreamSupport.stream(nu, true)
  val c = new java.util.Comparator[String]{ def compare(s: String, t: String) = s compareTo t }
  println(st.map[String]((s: String) => s + s).max(c))

This compiles and runs successfully. However, when the contents of main are entered in the REPL, the last line hangs. Stack trace:

"ForkJoinPool.commonPool-worker-1" #14 daemon prio=5 os_prio=0 tid=0x00007f966855f800 nid=0x721f in Object.wait() [0x00007f964a18b000]
   java.lang.Thread.State: RUNNABLE
	at $line6.$read$$iw$$iw$$anonfun$1.apply(<console>:11)
	at $line6.$read$$iw$$iw$$anonfun$1.apply(<console>:11)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at xp.Aspi.tryAdvance(Test.scala:14)
	at java.util.Spliterator.forEachRemaining(Spliterator.java:326)
        ...

Explicitly creating the SAM like so:

println(st.map[String](new java.util.function.Function[String, String] {
  def apply(s: String) = s + s
}).max(c))

solves the issue. (Note: I have not tried this on 2.11.5; I assumed 2.12.x was up to date in anything relevant.)

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-9076?orig=1
Reporter: @Ichoran
Affected Versions: 2.12.0

[scabug]

@retronym said:
Enabling -Xprint:all in the REPL, shows the difference between your manually expanded version and the SAM-lambda version:

  object iw extends Object {
    private[this] val res0: scala.runtime.BoxedUnit = _;
    <stable> <accessor> def res0(): Unit = ();
    final def apply$body$1(s: String): String = s.+(s);
    def <init>(): type = {
      iw.super.<init>();
      iw.this.res0 = {
        Test.test({
          new <$anon: java.util.function.Function>()
        });
        scala.runtime.BoxedUnit.UNIT
      };
      ()
    }
  };
  @SerialVersionUID(value = 0) final class anonfun$1 extends Object with java.util.function.Function with Serializable {
    final override <synthetic> def apply(s: String): String = $line3.iw.apply$body$1(s);
    final override <synthetic> <bridge> <artifact> def apply(x$1: Object): Object = anonfun$1.this.apply(x$1.$asInstanceOf[String]());
    def <init>(): <$anon: java.util.function.Function> = {
      anonfun$1.super.<init>();
      ()
    }
  }

The apply method delegates to iw.apply$body$1. This blocks when called form the threadpool because the static initializer of iw$ has not completed, it is blocking on the entire operation to complete.

See http://stackoverflow.com/questions/23108731/why-is-scala-await-result-timing-out-in-repl-when-passed-the-same-future-twice#comment35339401_23111645

Indeed this terminates again under scala -Yrepl-class-based. But we can't switch to that as a default until we sort out problems like:

% qscala -Yrepl-class-based

scala> class C(val a: Any) extends AnyVal
<console>:7: error: value class may not be a member of another class
       class C(val a: Any) extends AnyVal
             ^

[scabug]

@retronym said:
BTW, a workaround is to indirect through a lambda:

scala> () => Test.test(s => s + s)
res0: () => Unit = <function0>

scala> res0()
Splitting from [0 5)
Could not split [0 3)
Could not split [3 5)
Trying at 3 [3, 5)
Trying at 0 [0, 3)
Trying at 1 [0, 3)
Trying at 4 [3, 5)
Trying at 2 [0, 3)
Trying at 5 [3, 5)
Trying at 3 [0, 3)
Optional[twotwo]

[scabug]

@retronym said:
See also: https://groups.google.com/forum/#!topic/scala-user/fljtxY9bJV0

/cc @som-snytt Do you think we could encode my workaround above into the object based wrappers to make the default REPL less prone to this problem?

[scabug]

@som-snytt said:
How about making it an App?

scala/scala#4246

[scabug]

@SethTisue said:
@retronym should this be retargeted for 2.12.1 or Backlog?

[scabug]

Dave Gurnell (davegurnell) said (edited on Mar 1, 2017 10:35:25 AM UTC):
Seth suggested I post this recreation here. Try this on a 2.12.1 REPL:

import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

// This works fine:
val x = Future(42)
Await.result(x, 1.second)

// This throws a TimeoutException:
Await.result(Future(42), 1.second)

Adding scalacOptions += "-Ydelambdafy:inline" to build.sbt makes the problem go away.

The problem can also result in a NoClassDefFoundError as shown here (running Tut on an ebook project):

[tut] compiling: /Users/dave/dev/projects/advanced-scala/src/pages/test.md
java.lang.NoClassDefFoundError: Could not initialize class $line4.$read$$iw$$iw$$iw$$iw$$iw$$iw$
    at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
    at scala.util.Success.$anonfun$map$1(Try.scala:251)
    at scala.util.Success.map(Try.scala:209)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
    ... manually elided for brevity
[tut] *** Error reported at /Users/dave/dev/projects/advanced-scala/src/pages/monads/test.md:6
java.util.concurrent.TimeoutException: Futures timed out after [1 second]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:255)
  at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
  at scala.concurrent.Await$.$anonfun$result$1(package.scala:190)
  at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  at scala.concurrent.Await$.result(package.scala:123)
  ... 88 elided
[error] (run-main-0) java.lang.Exception: Tut execution failed.
[trace] Stack trace suppressed: run last *:streams for the full output.
[trace] Stack trace suppressed: run last *:tutOnly for the full output.
[error] (*:tutOnly) Nonzero exit code: 1
[error] Total time: 5 s, completed 27-Feb-2017 09:38:19

[adriaanm]

Probably a wontfix in 2.12, but should consider this as we improve the repl in 2.13 (where we'll standardize on some form of class-based wrapping).

[SethTisue]

should this still be on the 2.13.0-RC1 milestone, or shall we push it out to 2.13.x or 2.14?

[smarter]

It's indeed fixed in Dotty: scala/scala-dev#195 :)

[adriaanm]

Re-scheduling for 2.13 in the context of scala/scala-dev#325 and a commercial support request.

First we should finish the jline 3 upgrade, but then let's look into improving the encoding.

[adriaanm]

@retronym suggested a few alternative solutions -- one is to type check using the current object wrappers, and then have a repl phase that does the class-based encoding? This would avoid breaking anyone that relies on the current repl tree shapes during typer. Have to check how this affects serialization in Spark.