A misuse of retry.Backoff()

I met a bug in an old project this week. retry.Backoff() is misused and the real exception in Future is hidden and override by TimeoutException. This introduces big difficulty for me to recognize the real failure reason.

softwaremill retry library is used in the code to do exponential retry. I guess the author didn’t clearly understand what Backoff does and just copied the code in sample directly. Below is the sample code.

implicit val success: retry.Success[Try[Connection]] =
      retry.Success[Try[Connection]](_.isSuccess)
    val future = retry.Backoff().apply {
      Future {
        Try(DriverManager.getConnection(connectionString))
      }
    }
    Await.result(future, 2 minutes) match {
      case Success(conn)      => conn
      case Failure(exception) => throw exception
    }

The above code waits for 2 minutes to connect the database. It expects connection related exception can be thrown from the Failure case. But it will never when the connection fails. This is due to Backoff() will retry 8 times by default with exponential delay which exceeds the 2 minutes waiting time. If the connection fails, it will always report java.util.concurrent.TimeoutException. Let’s see the code of Backoff.apply().

  /** Retry with exponential backoff for a max number of times */
  def apply(
    max: Int = 8,
    delay: FiniteDuration = Defaults.delay,
    base: Int = 2)
   (implicit timer: Timer): Policy =
    new CountingPolicy {
      def apply[T]
        (promise: PromiseWrapper[T])
        (implicit success: Success[T],
         executor: ExecutionContext): Future[T] = {
          def run(max: Int, delay: FiniteDuration): Future[T] = countdown(
            max, promise,
            count => Delay(delay) {
              run(count, Duration(delay.length * base, delay.unit))
            }.future.flatMap(identity))
          run(max, delay)
        }
    }

By default, the longest waiting time is 500 * (pow(2, 8) -1) milliseconds = 2.1 minutes. So be sure you are clear what the code does when using a third party library.