Recently I learned... Union Types

Union types can add powerful functionality in cases where it would be easy to give up and lose type safety.

Suppose you have a typed configuration key -- for example, myIds.
myIds stores a value of type Set[Int], but because it's a configuration key, sometimes you might set it in code with a Set[Int], but sometimes you might want to set it from a String value from a config file.

So let's say myIds has a setValue method. We would like to be able to call setValue with a Set[Int] or with a String, but nothing else, and it should be safe at compile-time.

e.g.,:
myIds.setValue(Set(1, 2, 3))
myIds.setValue("1,2,3")

We could give up type-safety:

private var internalValue: Set[Int]
def setValue(value: Object): Union = {
  if (value.isInstanceOf[String]) {
    internalValue = convertFromString(value)
  } else if (value.isInstanceOf[Set]) { // Set of what? Grr type erasure is so evil.
    internalValue = value
  } else {
    throw new UglyAndHorribleRuntimeException("Wish we could have prevented this at compile-time!")
  }
}

How can we make it compile-time safe instead of accepting Object?

Enter Union Types!
Scala has intersect types -- just using "with" and you get a type made by intersecting two types.
It is possible to do Union types as well, but it's not built into the language so it's a bit more work.

Ideally we want to do something like:

type SetOrString = Set[Int] ∨ String
def setValue(value: SetOrString): Unit
...

Thanks to some help from StackOverflow, this can be accomplished! It's not 100% clean, but pretty good.

// http://stackoverflow.com/questions/3508077/how-to-define-type-disjunction-union-types// http://www.edofic.com/posts/2013-01-27-union-types.html// http://milessabin.com/blog/2011/06/09/scala-union-types-curry-howard/
// We want to be able to define union types so we can have, for example, a method that// accepts either an Int or a String, and for this method to be type-safe at compile-time,// and for it to work on primitives rather than having to use a boxed type such as Either.// Scala allows us to define type intersection using the "with" operator.// Scala does not have a union operator.// It's possible to define union using DeMorgan's Law: X or Y = NOT(NOT(X) AND NOT(Y))// Another way to think about it - from StackOverflow.com:// Given type ¬[-A] which is contravariant on A, by definition given A <: B we can write ¬[B] <: ¬[A],// inverting the ordering of types.// Given types A, B, and X, we want to express X <: A || X <: B.// Applying contravariance, we get ¬[A] <: ¬[X] || ¬[B] <: ¬[X].// This can in turn be expressed as ¬[A] with ¬[B] <: ¬[X] in which one of A or B must be a supertype// of X or X itself (think about function arguments).

import scala.language.higherKinds
/**  * Negation  * @tparam A Type to negate  */sealed trait ¬[-A]

/**  * Type set  */sealed trait TSet {
  type Compound[A]
  type Map[F[_]] <: TSet
}

/**  * Null type set  */sealed trait ∅ extends TSet {
  type Compound[A] = A  type Map[F[_]] = ∅
}

/**  * Type union  * Note that this type is left-associative for the sake of concision.  * @tparam T  * @tparam H  */sealed trait ∨[T <: TSet, H] extends TSet {
  // Given a type of the form `∅ ∨ A ∨ B ∨ ...` and parameter `X`, we want to produce the type  // `¬[A] with ¬[B] with ... <:< ¬[X]`.  type Member[X] = T#Map[¬]#Compound[¬[H]] <:< ¬[X]

  // This could be generalized as a fold, but for concision we leave it as is.  type Compound[A] = T#Compound[H with A]

  type Map[F[_]] = T#Map[F] ∨ F[H]
}

type UnionType = ∅ ∨ T ∨ String
def setValue[SetOrString : UnionType#Member](value: SetOrString)
  if (value.isInstanceOf[String]) {
    internalValue = convertFromString(value)
  } else { // Set[Int]
    internalValue = value
  }

}

No more runtime exception!



setValue("1,2,3") // Compiles

setValue(Set(1,2,3)) // Compiles

setValue(1) // Does not compile!



In my production code, I have a generic type Key that has a generic type parameter, so in my case instead of Set[Int] I would have some generic type T that is different based on whether it's an IntKey or an IntSetKey or something more complicated. The generics all work with this, and as a bonus, StringKey works as well (so String union String still works).



Understanding the set theory behind this is not a prerequisite for using it, so try it out and have some clean, type-safe code!

Today I taught someone... Scala implicits

Implicit classes / implicit conversions:

Suppose you have a Spark RDD and you want to call rdd.groupByKey()… It will fail to compile because the RDD class does not have a groupByKey() method.

Add an import to org.apache.spark.rdd.PairRDDFunctions, and it will magically compile.

PairRDDFunctions is a class that defines extra functions you can use on an RDD. The class definition for PairRDDFuncitons is honestly rather confusing… But here is a simpler example that shows how implicit classes can add additional functionality to a class:

implicit class StringExtensions(val str: String) extends AnyVal {   /**     * Whether or not the string is comprised entirely of digits (doesn't accept negative numbers)     * @return Whether or not string is numeric     */    def isNumeric: Boolean = {     str.forall(Character.isDigit)   } }

After adding an import of StringExtensions, you can now write code like “12345”.isNumeric and it will compile even though that is not a method in java.lang.String. This is similar to C#’s extension methods.

Another handy use of this that you may come across is Java/Scala interoperability, to convert between the Java and Scala collection types.

Let’s say we’re in Java 7 (no streams, boo hoo) and want to do some functional programming on a java.util.List.

Maybe it’s a List<String> and we want to find all the numeric strings and do some crazy map, like for each string, get a tuple of the reversed string and the summation of the digits in the string (not sure why we would ever need to do this). Then we will return it back as a list of the numbers.

If we import scala.collection.JavaConverters._, then we can do this in one (long) line:

javaList.asScala.filter(_.isNumeric).map(str => (str.reverse, str.foldLeft(0)((accumulatedValue, char) => accumulatedValue + char.toString.toInt))).asJava

Here, asScala is an implicit conversion from a Java List to a Scala List, isNumeric is the implicit method we defined above, and reverse is defined in java.lang.String.

The map function turns the string into a 2-tuple of the reversed string and the summation of the digits in the string (done using foldLeft, but not the only way to do it).

The map returns a Scala sequence, so .asJava converts it back to a Java List.

If you import scala.collection.JavaConversions._ instead (-ions instead of -ers), then you can remove the .asScala and .asJava, and it will still work, but doing the conversions explicitly improves readability.

Implicit parameters 

Looking in the Spark documentation, it says you can define an accumulator of type double, given a Spark context, by doing context.accumulator(0.0).

But this does not work unless you add an import to org.apache.SparkContext._ -- this is because, although the SparkContext class does have an accumulator method, it takes in additional parameters, which are marked as implicit. The SparkContext object defines implicit objects of the types needed by the accumulator functions. So once you import those and get them in scope, you can call accumulator with a single parameter, and it will fill in the additional parameter(s) based on the implicit variables in scope.

// Example borrowed from a comment on http://docs.scala-lang.org/tutorials/tour/implicit-parameters.html object Main extends App {   def sayHello(name: String)(implicit greeting: String) {   println(s"$greeting, $name")   }   // sayHello("John") // does not compile because the greeting is not given   sayHello("Paul")("Hello")   implicit val greeting = "Hi"   sayHello("George") // Works because there is an implicit of the same type in scope }

Today I learned... why Scala usually doesn't use null

We Scala developers use the Option monad because it's so much better to assign None to something instead of null -- it's much safer and more explicit.

And we never have to see another NullPointerException!

Fatal exception:
None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:313)
at scala.None$.get(Option.scala:311)

Now we just have to worry about not trying to call get on an Option that's None.
This is why many people in the Scala community recommend against ever using Option.get.

It's Scala's version of the NullPointerException. But it can be avoided if we use methods like map, filter, fold, foreach, getOrElse, etc., on an Option monad... Or in the Java-style, if we really have to use get, to check if it's empty first.

But sometimes you might want it to throw an exception because it means an assumption has been violated.

Today I learned... A limitation of singletons in Scala

Singletons in Scala are pretty nice -- you can have the basic goodness of a Java singleton, but with none of the scaffolding. The static keyword is not available in Scala, so singletons are useful anytime you want a static type of behavior, but Scala singletons can extend abstract classes or traits, can be passed in, mocked, etc.

The limitation comes when you are doing unit testing and want to run multiple unit tests completely independently of each other. If your singleton has any sort of state associated with it -- not just a var or a mutable object in a val, but also something as innocuous as the current timestamp at initialization saved as a val, etc. -- then there is no way, with the possible exception of reflection, of resetting that singleton's state in-between unit tests.

These particular unit tests I'm writing are actually functional system tests -- I'm basically calling my main method with different command-line parameters once in each unit test. And since in the JVM world I can't just do something magical like shutdown and reload an AppDomain (I miss this concept from .NET), I'm stuck with state getting persisted between what are supposed to be multiple program runs, because they're all in the same JVM (of course, same as the unit test's JVM, because I'm calling them through the JVM and not through the command-line, so I can make sure they don't throw exceptions, verify side effects if desired, and so on). And the main culprit is the singleton object.

I wrote a trait that can be used to solve this problem, but unfortunately, it required refactoring all my usages of any singletons that had state, so instead of singleton.method(), it became singleton.instance.method(). Find/replace all solved this problem without any trouble for me.

The trait manages the singleton, and also requires anything extending it to have a reset method. I call the reset method for all stateful singletons in the finally block of my main method.

trait ResettableSingleton[T] {
  protected var instanceMaybe: Option[T] = None

  def isInstantiated: Boolean = instanceMaybe.nonEmpty

  /**    * Singleton instance    *    * @return Object of the class    */  def instance: T = {
    instanceMaybe synchronized {
      if (instanceMaybe.isEmpty) {
        instanceMaybe = Option(instantiate)
      }
      instanceMaybe.get
    }
  }

  /**    * Some might prefer shorter syntax, so this is just an alias for "instance"    *    * @return Instance    */  def get: T = instance

  /**    * Create an instance of the class    * This is abstract because there's no guarantee that every class has, e.g., a parameterless constructor.    *    * @return Object of the class    */  protected def instantiate: T
  /**    * Resets the singleton instance. Intended to be used if performing multiple test runs without restarting the JVM.    */  def reset(): Unit = {
    instanceMaybe = None
  }
}

Example usage:

// Take the original singleton, turn it into a class, and make the default constructor private.
class Blah private() {
  // ... Original singleton body goes here, and remains unchanged
}
/** * Resettable singleton of Blah */object Blah extends ResettableSingleton[Blah] {
/** * Create an instance of the class * @return Object of the class */ override def instantiate: Blah = {
new Blah
}
}

So I lose the syntactic sugar, but now my unit tests work. Hey, it could be worse... I could be writing Java.

Today I learned... Data-driven unit testing in Scala

I've done data-driven unit tests in C# before, but recently I learned how to do them in Scala, and it's very easy and clean.

An example unit test, using WordSpec, which follows a BDD-style syntax, is below:

"Multiplying two integers" when {
  "one is negative" should {
    "have a negative result" in {
      val positive = scala.util.Random.nextInt(10000)
      val negative = -1 * scala.util.Random.nextInt(10000)
      val result = positive * negative
      assert(result < 0, s"positive * negative should be negative, but is ${result}!")
    }
  }
}

Unit test methods can be created on-the-fly at runtime, and therefore you can take any data source and turn it into a set of unit tests. Here is an example using WordSpec, which is one of the several flavors of writing unit tests in Scala.

(1 to 100).foreach(testCaseNumber => {
  s"Test scenario $testCaseNumber" when {
    "doing something" should {
      "return success" in {
        doSomethingWithData(getDataForTest(testCaseNumber))
      }
    }
  }
}

When you run all unit tests in your class, ScalaTest will show each of these 100 data as a separate test case with separate pass/fails.

It's really slick, and if you have some common test that you want to run over and over with some slight modification that can be controlled based on some parameter, I recommend you give it a try!

Today I learned... tuple assignment in Scala

// This works
var (x, y) = try {
  (0, 0)
}

// This doesn't work... would be nice if it did!
var x: Int = 0
var y: Int = 0
(x, y) = try {
  (0, 0)
}

Today I learned... doing things the hard way when languages and OSes are glitchy

So I have some code that works just fine... most of the time. But there has been some strange stuff that doesn't make any sense and isn't easily reproducible.

Few examples I encountered recently were Scala's parallel HashMap and file renaming in Java.

Scala has a nice parallel HashMap that allows you to do gets, puts, and updates concurrently, and it handles the synchronization for you... except when it doesn't.

I put a tuple into a ParHashMap, which has no delete operations on it in my whole code, and later when I went to get that value, it wasn't there.

On another instance, when I went to put a tuple into a ParHashMap, I got an ArrayIndexOutOfBoundsException.

I replaced my ParHashMap with a single-threaded HashMap and did my own synchronization around it, and I haven't seen either problem since...


Another issue was renaming a file in Windows using Java. File.renameTo was working just fine, but all of a sudden I came across one occurrence where the file appeared to have been copied instead of renamed. So I changed my code to delete the source file after the rename, if it was still there. But then I got an exception that the source file was already in use, even though I had previously closed my writer that was operating on that file.

I ended up writing the following method to replace what should be a simple one-line file rename:

def rename(filePath: String, targetPath: String): Unit = {
    RetryHelper.retry(10, Seq(classOf[java.nio.file.FileSystemException])){
      val sourceFile = new File(filePath)
      val targetFile = new File(targetPath)
      if (!targetFile.exists) {
        Files.move(sourceFile.toPath, targetFile.toPath)
      }
      if (sourceFile.exists) {
        sourceFile.delete()
      }
    }(Thread.sleep(50))
}

That's right, the only way I could get it to work every single time, without fail (so far!) was to delete the source file after the "move" (in case it behaved as a copy for some reason), and to wrap that in a retry block that executes up to 10 times and catches FileSystemExceptions with a short sleep before retrying. Yikes!

For reference, here is my retry helper, which I discussed in a previous post but which has been revised since, as it now takes in a code block defining what to do in the exception handling block when catching one of the specified exceptions.

import scala.util.control.Exception._
// Motivation for this object is described in the following blog posts://   http://googlyadventures.blogspot.com/2015/05/what-else-i-learned-yesterday.html//   http://googlyadventures.blogspot.com/2015/06/today-i-learned-doing-things-hard-way.html
/**  * Helper methods to retry code with configurable retry count and exceptions to handle  */object RetryHelper {
  /**    * Retry any block of code up to a max number of times, optionally specifying the types of exceptions to retry    * and code to run when handling one of the retryable exception types.    *    * @param maxTries              Number of times to try before throwing the exception.    * @param exceptionTypesToRetry Types of exceptions to retry. Defaults to single-element sequence containing classOf[RuntimeException]    * @param codeToRetry           Block of code to try    * @param handlingCode          Block of code to run if there is a catchable exception    * @tparam T Return type of block of code to try    * @return Return value of block of code to try (else exception will be thrown if it failed all tries)    */  def retry[T](maxTries: Int, exceptionTypesToRetry: Seq[Class[_ <: Throwable]] = Seq(classOf[RuntimeException]))
              (codeToRetry: => T)
              (handlingCode: Throwable => Unit = _ => ()): T = {
    var result: Option[T] = None
    var left = maxTries
    while (!result.isDefined) {
      left = left - 1
      // try/catch{case...} doesn't seem to support dynamic exception types, so using handling block instead.
      handling(exceptionTypesToRetry: _*)
        .by(ex => {
          if (left <= 0) {
            throw ex
          } else {
            handlingCode(ex)
          }
        }).apply({
        result = Option(codeToRetry)
      })
    }
    result.get
  }
}