Showing posts with label Scala. Show all posts
Showing posts with label Scala. Show all posts

Thursday, June 7, 2012

Book review: Scala in Depth

I read Scala in Depth by Joshua D. Suereth over last week, which really is not giving it the time and attention it deserves. Furthermore I'm no Scala expert - although I'm not a complete novice either. Anyway here are my thoughts on the book.

Is it any good? Or TL;DR
Yes, it's good. It delivers on the promise of the title it is - as far as I can tell - really is Scala in depth. It does so with a mixture of some theoretical background for the different language constructs and a lot of practical Scala programming advice coming from the lessons the author has learned through his own use of the language. Where it sometimes lacks a bit is in introducing stuff before using it. This is not a problem if you have a basic working knowledge of Scala already, but I image it would be, if you didn't. Bottom line: I'd recommend Scala in Depth to anyone who - like me - has dipped their toes in Scala and want to learn more.

The language
The book starts off with a very quick introduction to the very basics of Scala - it's a statically blended OO/functional language with a very flexible syntax and grammar running on the JVM (and to some extent .NET if you really want). This introduction includes the REPL and a few tips like "prefer immutability". This should be enough to give you a feel for the language, and to get you startet playing in the REPL.
Once the basics are out of the way the book becomes somewhay more hard core - not in an academic language semantics kind of way, but in a practical way, where the reader through the book is taken through a number of sometimes hairy examples. These gives a thourough run through of the object oriented parts of the language (like classes, objects, traits and polymorphism), the type system (like generics, higher kinded types and existential types) and the functional parts (like functions, higher order functions, and some basic functional patterns).
All in all these parts of the book in themselves provide an in depth look at Scala. But not a very practical one: Knowing of the language features and even knowing the ins and outs of them is not the same as being able to use them well. These practicalities are addresed by the more style oriented parts of the book.

Style
The other part of the book - which is not pulled out as a separate part, but interleaved with the rest of the book - is the advice on style. To me these are the most interesting parts of the book. This where the book gets into things like demonstrating how to use actors effectively and safely, how to use implicits sanely, a map of the collections library and integrating with Java. These are things I suspect you'll need in real life development, and which are painful to go through learning by yourself.

All in all a very informative book that can take you from a basic knowledge of Scala to a thorough knowledge of the language - given that you take the time to do some serious coding along side your reading. If you just speed through the book - like I did - you'll learn less, but still a significant amount.

Monday, December 20, 2010

Options for DCI on .NET

Here's a short summary of the technical/language options for doing DCI on the .NET platform. The list below is loosely ordered in order of increasing "mainstreamness" - according to me :-).
The list notes the language, how roles and contexts are handled and how the solution is run on .NET.

  • Scala "cross-compiled" to MSIL/CLR:
    In Scala contexts are simply objects instantiated at runtime from context classes. The contexts have the responsibility of instantiating data objects and mapping roles to them. Roles are traits that are mixed into data objects at instantiation time. So typically the contexts receive ids for data objects, pull the corresponding data from e.g. a database, and instantiate the data object with a role trait mixed in. In Scala this is all strongly typed and looks like this:

    class MoneyTransferContext(sourceAccountId: int,
                                sinkAccountId: int){
       val source =
            new SavingsAccount(sourceAccountId) with TransferMoneySource
       val sink =
             new CheckingAccount(sinkAccountId) with TransferMoneySink
       def execute = {
         source.deposit(100000)
         source.transferTo(200, sink)
         }
    }

    To run this in a .NET setting the Scala code must "cross-compiled" to MSIL, and must use the .NET standard library along with the Scala APIs. For details on this see my earlier post, or the introduction on the Scala language web site.

  • Python running on IronPython:
    In Python contexts are objects, that take the responsibility of mixing roles into data objects dynamically. The data objects can be instantiated outside the context or inside the context depending on taste. When the context is done the roles can be yanked back out of the data objects. I dont know a whole lot of Python, so I wont go into details, but refer you to Serge Beaumonts sample.
    To run this on .NET use IronPython.
  • Ruby running on IronRuby:
    In Ruby contexts are also objects, that take the responsibility of mixing roles into data objects dynamically. Data objects can be instantiated inside the context or outside the context. Roles in Ruby are implemented as modules, which using the Ruby standard library can be mixed into any Ruby object. The Ruby code looks like this:
    class TransferMoneyContext
       attr_reader :source_account, :destination_account, :amount
       include ContextAccessor
       
       def self.execute(amt, source_account_id, destination_account_id)
         TransferMoneyContext.new(amt, source_account_id,
                                  destination_account_id).execute
       end
     
       def initialize(amt, source_account_id, destination_account_id)
         @source_account = Account.find(source_account_id)
         @source_account.extend MoneySource
         @destination_account = Account.find(destination_account_id)
         @amount = amt
       end
    
       def execute
         in_context do
           source_account.transfer_out
         end
       end
    end

    To run this on .NET just use IronRuby. It runs fine out of the box.

  • C# and Dynamic:
    Using the dynamic features of C#, contexts are objects that receive either data objects or IDs of data objects. In case of an ID the context instantiates the data object and populates it based on the ID and some data source. In both cases the data object has a role reference which will point to an expando object, into which the roles methods are injected. The code looks roughly like this:
    dynamic transfer = new ExpandoObject();
     transfer.Transfer = (Action)
      ((source, sink, amount) =>{ 
        dynamic mSource = source;
        dynamic mSink = sink;
        mSource.DecreaseBalance(amount);
        mSink.IncreaseBalance(amount);});
     Account sourceAcct = new Account { CashBalance = 100m };
     sourceAcct.Role = transfer;
     Account sinkAcct = new Account { CashBalance = 50m };
     sourceAcct.Role.Transfer(sourceAcct,sinkAcct,10m);
    Running on .NET? Well, it's C#, so just do it.

  • C# and extention methods:
    Using C# extension methods for roles contexts are objects that receive either data objects or IDs of data objects. In either case the context maps the data objects to roles. Roles are implemented as static classes containing extension methods for role interfaces. Data objects capable of playing a given role implement that role interface. The code looks like this:
    public class TransferMoneyContext { public TransferMoneySource Source { get; private set; } public TransferMoneySink Sink { get; private set; } public decimal Amount { get; private set; } public TransferMoneyContext(TransferMoneySource source, TransferMoneySink sink, decimal amount) { Source = source; Sink = sink; Amount = amount; } public void Execute() { Source.TransferTo(Sink, Amount); } } public interface TransferMoneySink { void Deposit(decimal amount); void Log(string message); } public interface TransferMoneySource { decimal Balance { get; } void Withdraw(decimal amount); void Log(string message); } public static class TransferMoneySourceTrait { public static void TransferTo(this TransferMoneySource self, TransferMoneySink recipient, decimal amount) { // The implementation of the use case if (self.Balance < amount { throw new ApplicationException("insufficient funds"); } self.Withdraw(amount); self.Log("Withdrawing " + amount); recipient.Deposit(amount); recipient.Log("Depositing " + amount); } }

    How to run this on .NET is ... well ... pretty obvious :-)

Saturday, October 2, 2010

Notes from getting Scala running on .NET

I just spend an evening getting Scala to run on .NET 4.0. Here my notes:

Scala 2.8.0

I started downloading Scala 2.8.0-final as a zip file (), and unpacking it to my scala playground directory. That gave me:
PS C:\Scala-playground\scala-2.8.0.final> tree
Folder PATH listing for volume OSDisk
Volume serial number is 00690066 6A5B:A532
C:.
+---bin
+---doc
¦   +---sbaz
¦   +---sbaz-setup
+---lib
+---meta
¦   +---cache
+---misc
¦   +---sbaz
¦   ¦   +---config
¦   ¦   +---descriptors
¦   +---sbaz-testall
¦   ¦   +---tests
¦   +---scala-devel
¦   ¦   +---plugins
¦   +---scala-tool-support
¦       +---a2ps
¦       +---bash-completion
¦       +---bluefish
¦       +---context
¦       ¦   +---Highlighters
¦       ¦   +---Template
¦       +---emacs
¦       ¦   +---contrib
¦       +---enscript
¦       +---geshi
¦       +---intellij
¦       +---jedit
¦       ¦   +---console
¦       ¦   ¦   +---commando
¦       ¦   +---modes
¦       +---latex
¦       +---notepad-plus
¦       +---scite
¦       +---subethaedit
¦       ¦   +---artwork
¦       ¦   +---Scala.mode
¦       ¦       +---Contents
¦       ¦           +---Resources
¦       ¦               +---English.lproj
¦       ¦               +---Scripts
¦       +---textmate
¦       ¦   +---Bundles
¦       +---textpad
¦       +---textwrangler
¦       +---ultraedit
¦       +---vim
¦           +---ftdetect
¦           +---indent
¦           +---plugin
¦           +---syntax
+---src

I then went on to install the Scala msil extension (i.e. the scala .NET compiler and runtime) as follows:
PS C:\Scala-playground\scala-2.8.0.final> bin/sbaz.bat install scala-msil

Then, following the "official" instructions on running scala on .NET (http://www.scala-lang.org/node/168), I copied mscorlib.dll to the scala lib folder:
PS C:\Scala-playground\scala-2.8.0.final> copy C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorlib.dll .\lib

I was now ready to try to compile something. E.g. this very simple hello world program using a little bit of .NET, namely Console.WriteLine:

import System.Console object test extends Application {   Console.WriteLine("Hello world!") }

I tried to compile it like this:
PS C:\Scala-playground\hello.net> ..\scala-2.8.0.final\bin\scalac.bat -target:msil .\test.scala

but got this error:
scala.tools.nsc.MissingRequirementError: class scala.runtime.VolatileBooleanRef not found.
        at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:513)
        at scala.tools.nsc.symtab.Definitions$definitions$.getClass(Definitions.scala:471)
        at scala.tools.nsc.symtab.Definitions$definitions$.newValueClass(Definitions.scala:620)
        at scala.tools.nsc.symtab.Definitions$definitions$.BooleanClass(Definitions.scala:92)
        at scala.tools.nsc.symtab.Definitions$definitions$.initValueClasses(Definitions.scala:643)
        at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:787)
        at scala.tools.nsc.Global$Run.(Global.scala:597)
        at scala.tools.nsc.Main$.process(Main.scala:107)
        at scala.tools.nsc.Main$.main(Main.scala:122)
        at scala.tools.nsc.Main.main(Main.scala)
error: fatal error: class scala.runtime.VolatileBooleanRef not found.

I tried like this to point out where mscorlib and the scala runtime dlls are:
PS C:\Scala-playground\hello.net> ..\scala-2.8.0.final\bin\scalac.bat -target:msil -Xassem-extdirs ..\scala-2.8.0.final\lib .\test.scala
scala.tools.nsc.MissingRequirementError: class scala.runtime.VolatileBooleanRef not found.
        at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:513)
        at scala.tools.nsc.symtab.Definitions$definitions$.getClass(Definitions.scala:471)
        at scala.tools.nsc.symtab.Definitions$definitions$.newValueClass(Definitions.scala:620)
        at scala.tools.nsc.symtab.Definitions$definitions$.BooleanClass(Definitions.scala:92)
        at scala.tools.nsc.symtab.Definitions$definitions$.initValueClasses(Definitions.scala:643)
        at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:787)
        at scala.tools.nsc.Global$Run.(Global.scala:597)
        at scala.tools.nsc.Main$.process(Main.scala:107)
        at scala.tools.nsc.Main$.main(Main.scala:122)
        at scala.tools.nsc.Main.main(Main.scala)
error: fatal error: class scala.runtime.VolatileBooleanRef not found.
..same error.

Then I resorted to frantic googling, but ended up deciding to downgrade the mscorlib.dll to 2.0, and to 1.1. But in both cases I got same depressing result.
Finally I decided to downgrade to scala 2.7.7.

Scala 2.7.7

I downloaded scala 2.7.7 unzipped it and got:
PS C:\Scala-playground\scala-2.7.7.final> tree
Folder PATH listing for volume OSDisk
Volume serial number is 00690066 6A5B:A532
C:.
+---bin
+---doc
¦   +---sbaz
¦   +---sbaz-setup
+---lib
+---meta
¦   +---cache
+---misc
¦   +---sbaz
¦   ¦   +---descriptors
¦   +---sbaz-testall
¦   ¦   +---tests
¦   +---scala-tool-support
¦       +---a2ps
¦       +---bluefish
¦       +---context
¦       ¦   +---Highlighters
¦       ¦   +---Template
¦       +---emacs
¦       ¦   +---contrib
¦       +---enscript
¦       +---geshi
¦       +---intellij
¦       +---jedit
¦       ¦   +---console
¦       ¦   ¦   +---commando
¦       ¦   +---modes
¦       +---latex
¦       +---notepad-plus
¦       +---scite
¦       +---subethaedit
¦       ¦   +---artwork
¦       ¦   +---Scala.mode
¦       ¦       +---Contents
¦       ¦           +---Resources
¦       ¦               +---English.lproj
¦       ¦               +---Scripts
¦       +---textmate
¦       ¦   +---Bundles
¦       +---textpad
¦       +---textwrangler
¦       +---ultraedit
¦       +---vim
¦           +---ftdetect
¦           +---indent
¦           +---plugin
¦           +---syntax
+---src
The I went through copying mscorlib and installing scala-msil again:
PS C:\Scala-playground\scala-2.7.7.final> copy C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorlib.dll .\lib
PS C:\Scala-playground\scala-2.7.7.final> .\bin\sbaz.bat install scala-msil
planning to install: scala-msil/2.7.7.final
Installing...

Then I tried compilling again:
PS C:\Scala-playground\hello.net> ..\scala-2.7.7.final\bin\scalac-net.bat .\test.scala
PS C:\Scala-playground\hello.net> ls


    Directory: C:\Scala-playground\hello.net


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----        30-09-2010     21:59            target
-a---        30-09-2010     22:33       3863 test.msil
-a---        30-09-2010     21:52        100 test.scala

which went well. The scala compiler made a .msil file. Yeah.

Next step was turning that .msil into an .exe:
PS C:\Scala-playground\hello.net> C:\Windows\Microsoft.NET\Framework\v4.0.30319\ilasm.exe .\test.msil

Microsoft (R) .NET Framework IL Assembler.  Version 4.0.30319.1
Copyright (c) Microsoft Corporation.  All rights reserved.
Assembling '.\test.msil'  to EXE --> '.\test.exe'
Source file is ANSI

Assembled method test::$tag
Assembled method test::main
Assembled method test$::.ctor
Assembled method test$::.cctor
Assembled method test$::$tag
Assembled method test$::main
Assembled global method Main
Creating PE file

Emitting classes:
Class 1:        test
Class 2:        test$

Emitting fields and methods:
Global  Methods: 1;
Class 1 Methods: 2;
Class 2 Fields: 1;      Methods: 4;
Resolving local member refs: 8 -> 8 defs, 0 refs, 0 unresolved

Emitting events and properties:
Global
Class 1
Class 2
Resolving local member refs: 0 -> 0 defs, 0 refs, 0 unresolved
Writing PE file
Operation completed successfully

Which gave:
PS C:\Scala-playground\hello.net> ls


    Directory: C:\Scala-playground\hello.net


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----        30-09-2010     21:59            target
-a---        30-09-2010     22:35       2560 test.exe
-a---        30-09-2010     22:33       3863 test.msil
-a---        30-09-2010     21:52        100 test.scala

So I tried running the program:
PS C:\Scala-playground\hello.net> .\test.exe

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'predef, Version=0.0.0.0, Culture=
neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
   at Main(String[] args)

but missed the predef assembly which is part of the scala-msil extension. To get passed that I simply copied all dlls from the Scala lib folder and ran again:
PS C:\Scala-playground\hello.net> mv .\test.exe .\target
PS C:\Scala-playground\hello.net> cp ..\scala-2.7.7.final\lib/*.dll .\target
PS C:\Scala-playground\hello.net> cd .\target
PS C:\Scala-playground\hello.net\target> ls

    Directory: C:\Scala-playground\hello.net\target


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        30-09-2010     22:30    2070528 mscorlib.dll
-a---        30-09-2010     22:30    1180160 predef.dll
-a---        30-09-2010     22:30       5632 scalaruntime.dll
-a---        30-09-2010     22:35       2560 test.exe
PS C:\Scala-playground\hello.net\target> .\test.exe
Hello world!

Success! but only on the next to newest Scala version. Either way I had fun :-)

Thursday, June 17, 2010

Divide and Accumulate with Scala Actors

In Scala we can easily do things in parallel with actors; actors are easily started up, they are cheap, and they can easily send objects to each other
.
Here I'll look at how to use actors to parallelize in situations where we need to do some sort of calculation and aggregation over a collection of objects. E.g. given a collection of RSS or ATOM feeds pull down the list recents items from each one and build the aggregated list of items from all the feeds ordered by publication time. Doing that sequentially is sort of trivial, and involves waiting for a request to each feed one at a time. It would by nice to fire all those requests in parallel, then wait for them to return thier results and build the aggregated list as the results arrive. The following diagram shows how to do that with actors:




















Let the program start an actor per feed. Let each actor go do a request for current items to the feed it is in charge of, and then send that list as an immutable object to the accumulating actor. The accumulating actor will maintain the aggregated list of feed items. When a new list of items is received from one of the other actors it merges that list into its aggregated list and at the end it has the full aggregated list.
Going a step further; why not have the accumulating actor send an immutable copy of the aggregated list to e.g. the view every tiume the aggregated list has been updated. That us allows us to show the user the partial results as they arrive.

Notice that the above did not involve any explicit starting of threads, no synchronization of threads, and no locking. Those things are handled behind the scenes by the actors and their message queues. Furthermore there is only mutable state in one place; inside the accumulating actor. Everything sent from one actor to another is immutable. So we've parallelized safely and easily.

The problem we're solving here is - admittedly - fairly easy to parallelize, but nonetheless I think the solution I described is nice because it is easy to understand and to code.

Wednesday, April 28, 2010

First Adventures in ScalaTest - Part III: Introducing Actors in the NetWorth Sample

This is the third post about my little net worth Scala sample. This third post really doesn't have a lot to do with ScalaTest, so the title is probably somewhat misleading. What this post is about is how to introduce some parallelism in the net worth calculation code by using Scala actors. The first post in the series showed some test written in ScalaTest, the second one showed the net worth sample itself, and this one starts where that post ended: The code for calculating the net worth based on an XML file of stock symbols and units is:
val stocksAndUnitsXml = scala.xml.XML.load(fileName)

 val tickersAndUnits =
    (Map[String, Int]() /: (stocksAndUnitsXml \ "symbol")) {
      (map, symbolNode) =>
        val ticker = (symbolNode \ "@ticker").toString
        val units = (symbolNode \ "units").text.toInt
        map(ticker) = units
    }

 def generateTotalNetWorthAndSimpleReport = {
    val report = new java.io.ByteArrayOutputStream
    Console.withOut(report) {

      println("Today is " + new java.util.Date)
      println("Ticket  Units  Closing Price($)  Total value($)")

      val startTime = System.nanoTime

      val netWorth = (0d /: tickersAndUnits) {
        (cumulativeWorth, symbolAndUnitsPair) =>
          val (symbol, units) = symbolAndUnitsPair
          val lastClosingPrice = getLatestClosingPriceFor(symbol)
          val value = lastClosingPrice * units

          println("%-7s  %-5d %-16f  %-16f".format(symbol, units, lastClosingPrice, value))

          cumulativeWorth + value
      }

      val endTime = System.nanoTime

      println("Total value of investments is $" + netWorth)
      println("Report tool %f seconds to generate".format( (endTime - startTime)/1000000000.0 ))

      (netWorth, report)
    }
  }
That code is explained in my last post.

Now I want to parallelize that calculation. Specifically I want to parallelize the web service calls to get the latests price for each stock symbol, so I'm going to fetch each price in a separate actor and then send the prices back to the main thread and accumulate the net worth there.

The code that gets the price for a single symbol and sends it to another actor called 'caller' is:
    caller ! (symbol, getLatestClosingPriceFor(symbol))
To do that in a separate actor for each symbol I call this method:
   private[this] def getLatestPricesForAllSymbols(caller: Actor) =
     tickersAndUnits.keys.foreach {
       symbol =>
         actor {
           caller ! (symbol, getLatestClosingPriceFor(symbol))
         }
    }
  }
the call to 'actor' starts a new actor an executes the function given to it asynchronously. The '!' method on the actor 'caller' sends the value given as an argument to the '!' method to 'caller'.

To receive a single symbol and price I do:
  receiveWithin(10000) {
    case (symbol: String, lastClosingPrice: Double) =>
      val units = tickersAndUnits(symbol)
      val value = lastClosingPrice * units
  }
'receiveWithin(10000)' blocks until a message arrives for the current actor, or times out after 10 seconds, so if this code is in the 'caller' actor it will receive one of the symbol/price pairs sent using '!' in the code above.

In order to accumulate the whole net worth I place the above in a function that also takes an 'cucumulativeWorth' and returns an updated cumulative worth:
   private[this] def receiveAndProcessOneSymbol(cumulativeWorth: Double) = {
    receiveWithin(10000) {
      case (symbol: String, lastClosingPrice: Double) =>
        val units = tickersAndUnits(symbol)
        val value = lastClosingPrice * units

        println("%-7s  %-5d %-16f  %-16f".format(symbol, units, lastClosingPrice, value))

        cumulativeWorth + value
    }
  }
and to receive all the symbol/price pairs I call that method as many times as there are symbols in the 'tickersAndUnits' map:
  private[this] def receiveAndAccumulateWorthForAllSymbol =
    (0d /: (1 to tickersAndUnits.size)) {
      (cumulativeWorth, index) =>
        receiveAndProcessOneSymbol(cumulativeWorth)
    }
Putting all that together the 'generateTotalNetWorthAndSimpleReport' becomes:
  def generateTotalNetWorthAndSimpleReport = {
    val report = new ByteArrayOutputStream
    Console.withOut(report) {
      calculateNetWorthAndWriteReportTo(report)
    }
  }

  private def calculateNetWorthAndWriteReportTo(report: ByteArrayOutputStream) = {
      writeReportHeader

      val startTime = System.nanoTime
      getLatestPricesForAllSymbols(self)
      val netWorth = receiveAndAccumulateWorthForAllSymbol
      val endTime = System.nanoTime

      writeReportFooter(netWorth, endTime, startTime)

      (netWorth, report)
    }

  private def writeReportHeader = {
      println("Today is " + new java.util.Date)
      println("Ticket  Units  Closing Price($)  Total value($)")
    }

  private def writeReportFooter(netWorth: Double, endTime: Long, startTime: Long): Unit = {
    println("Total value of investments is $" + netWorth)
    println("Report took %f seconds to generate".format((endTime - startTime) / 1000000000.0))
  }

  private[this] def getLatestPricesForAllSymbols(caller: Actor) =
    tickersAndUnits.keys.foreach {
      symbol =>
        actor {
          caller ! (symbol, getLatestClosingPriceFor(symbol))
        }
    }

  private[this] def receiveAndAccumulateWorthForAllSymbol =
    (0d /: (1 to tickersAndUnits.size)) {
      (cumulativeWorth, index) =>
        receiveAndProcessOneSymbol(cumulativeWorth)
    }

   private[this] def receiveAndProcessOneSymbol(cumulativeWorth: Double) = {
    receiveWithin(10000) {
      case (symbol: String, lastClosingPrice: Double) =>
        val units = tickersAndUnits(symbol)
        val value = lastClosingPrice * units

        println("%-7s  %-5d %-16f  %-16f".format(symbol, units, lastClosingPrice, value))

        cumulativeWorth + value
    }
Now the calculation is done by fetching prices in parallel using the lightweight Scala actors, and sending the results back as one way messages that are aggregated to the final result in the main thread. That was easy, don't you think?

Oh, and finally even in this little sample and on a small dataset this actually does speed up things. If I rerun the integration test shown in the last post:
class IntegrationSpec extends FlatSpec with ShouldMatchers {
  def withYahooStockPriceFinder(fileName: String)(testFunctionBody: (PortfolioManager) => Unit) {
    testFunctionBody(new PortfolioManager(fileName) with YahooStockPriceFinder)
  }
 
  "A PortFolioManager with the YahooStockPriceFinder" should "produce a asset report and calculate the total net worth" in {
    withYahooStockPriceFinder("src/configuration/stocks.xml") {
      pm =>
        val (totalNetWorth, report) = pm.generateTotalNetWorthAndSimpleReport
        totalNetWorth should be(85000d plusOrMinus 10000)

        print(report)
    }
  }
}
I get:
Today is Wed Apr 28 20:37:38 CEST 2010
Ticket  Units  Closing Price($)  Total value($)
MSFT     190   30,930000         5876,700000    
INTC     160   23,220000         3715,200000    
ALU      150   3,150000          472,500000     
ORCL     200   25,940000         5188,000000    
NSM      200   14,970000         2994,000000    
CSCO     250   27,170000         6792,500000    
AMD      150   9,550000          1432,500000    
IBM      215   130,060000        27962,900000   
VRSN     200   26,840000         5368,000000    
XRX      240   10,980000         2635,200000    
APPL     200   0,000000          0,000000       
HPQ      225   53,330000         11999,250000   
ADBE     125   35,420000         4427,500000    
SYMC     230   17,270000         3972,100000    
TXN      190   26,380000         5012,200000    
Total value of investments is $87848.55
Report took 0,837941 seconds to generate
which shows that the calculation took less than a second now, whereas the calculation took over 6 seconds in the sequential version.
Oh, and also notice that the symbols appear in a different order in the report than they did in the last post. The input data is exactly the same. The difference is that they are printed in the order that the main thread receives prices for them, and that ordering is not deterministic any more because it depends on how fast each web service call returns.

And that's it. I'm still having fun with learning Scala :-)

Thursday, April 15, 2010

First Adventures in ScalaTest - Part II

Following up on my last post, where I showed some test code written with ScalaTest for a simple PotfolioManager demo I'll show the PortfolioManager itself in this post. As mentioned in the last post the code is largely (almost entirely) based on the last chapter of Programming Scala by Venkat Subramaniam.

The PortFoliioManager is an abstract class with a default constructor that takes a path to an XML file and loads the contents:

abstract class PortfolioManager(fileName: String) extends StockPriceFinder {
    val stocksAndUnitsXml = scala.xml.XML.load(fileName)
  //more..
}


The data in the XML file is expected to look something like this:

<symbols>
<symbol ticker="APPL"><units>200</units></symbol>
<symbol ticker="ADBE"><units>125</units></symbol>
</symbols>







and is parsed through XPath queries applied to the XML with the '\' operator. This method on the PortfolioManager does the parsing:

def tickersAndUnits =  
  (Map[String, Int]() /: (stocksAndUnitsXml \ "symbol")) {
    (map, symbolNode) =>
      val ticker = (symbolNode \ "@ticker").toString
      val units = (symbolNode \ "units").text.toInt
      map(ticker) = units
  }

so what goes on there is that we get the list of symbol XML elements from the stocksAndUnitsXml value by use of operator '\' from the Scala standard library. We then iterate over that list with '/:' (aka foldLeft). Through the iteration we build up a map from strings to ints mapping from stock symbols to the number of units. Again the data is pulled out of the XML with XPath and '\'.

Thats all just a bit of warm up. What the PortfolioManager is supposed to do is calculate and report the net worth of the stock portfolio described in the XML. In itself that's not a big deal, but I think its fun to see how it's handled in Scala - it turns out to be sort of neat.

The net worth is calculated by fetching the latest price of each individual stock symbol multiply by the units and sum up. To get the stock prices the code must query some external source. E.g. download.finance.yahoo.com, but I don't want to do that directly because I want to be able to run my tests without depending or waiting for Yahoo. That's why the ProfolioManager is abstract: The call to get the latest price for a symbol is factored out to the StockPriceFinder traits and is abstract:

trait StockPriceFinder {
  protected def getLatestClosingPriceFor(tickerSymbol: String) : Double
}

Since Portfolio manager extends StockPriceFinder it has to be abstract. Client code has to either instantiate concrete subclasses or mix in a trait implementing the getLatestClosingPriceFor method at instantiation time. That's what the tests from the last post did. And that's what makes it easy to switch between a fake implementation for unit tests, and a real implementation for "production" code and integration test code. I'll show an integration test towards the end of the post, but for now lets return to calculating the net worth and print a simple report to an output stream:

def generateTotalNetWorthAndSimpleReport = {
    val report = new java.io.ByteArrayOutputStream
    Console.withOut(report) {

      println("Today is " + new java.util.Date)
      println("Ticket  Units  Closing Price($)  Total value($)")

      val startTime = System.nanoTime
      val netWorth = (0d /: tickersAndUnits) {
        (cumulativeWorth, symbolAndUnitsPair) =>
          val (symbol, units) = symbolAndUnitsPair
          val lastClosingPrice = getLatestClosingPriceFor(symbol)
          val value = lastClosingPrice * units
          println("%-7s  %-5d %-16f  %-16f".format(symbol, units, lastClosingPrice, value))
          cumulativeWorth + value
      }
      val endTime = System.nanoTime

      println("Total value of investments is $" + netWorth)
      println("Report took %f seconds to generate".format( (endTime - startTime)/1000000000.0 ))

      (netWorth, report)
    }
  }

Things to notice in the code above are:
  • Console.withOut(report) redirects printlns in its function parameter to the output stream 'report'
  • The method returns two values packaged up in a tuple simply by ending with the line '(networth, report)'
  • The fold left operation is used again this time to iterate over the map from symbols to units and accumulating the worth
  • The price of a symbol is found by calling the abstract getLatestClosingPriceFor

To use the code we provide an implementation for the StockPriceFinder, a path and call generateTotalNetWorthAndSimpleReport:

class IntegrationSpec extends FlatSpec with ShouldMatchers {
  def withYahooStockPriceFinder(fileName: String)(testFunctionBody: (PortfolioManager) => Unit) {
    testFunctionBody(new PortfolioManager(fileName) with YahooStockPriceFinder)
  }

  "A PortFolioManager with the YahooStockPriceFinder" should "produce a asset report and calculate the total net worth" in {
    withYahooStockPriceFinder("src/configuration/stocks.xml") {
      pm =>
        val (totalNetWorth, report) = pm.generateTotalNetWorthAndSimpleReport
        totalNetWorth should be(85000d plusOrMinus 10000)

        print(report)
    }
  }
}

which produces this output:

Today is Wed Apr 14 20:54:02 CEST 2010
Ticket  Units  Closing Price($)  Total value($)
XRX      240   10.540000         2529.600000   
NSM      200   15.650000         3130.000000   
SYMC     230   17.015000         3913.450000   
ADBE     125   34.875000         4359.375000   
VRSN     200   26.870000         5374.000000   
CSCO     250   26.870100         6717.525000   
TXN      190   26.737500         5080.125000   
ALU      150   3.400000          510.000000     
IBM      215   131.050000        28175.750000   
INTC     160   23.470000         3755.200000   
ORCL     200   26.290000         5258.000000   
APPL     200   0.000000          0.000000      
HPQ      225   54.530000         12269.250000   
AMD      150   9.940000          1491.000000   
MSFT     190   30.730000         5838.700000   
Total value of investments is $88401.975
Report took 6.151573 seconds to generate

Next post I'll introduce some actors in the code to parallelize it, and to see if that brings a speed up.

Friday, April 9, 2010

First Adventures in ScalaTest

As mentioned earlier I'm spending some time learning Scala. To that end I've read Programming Scala - which BTW is a book I'd recommend to anyone looking into Scala. Towards the end of the book there's an example of a litte application that can calculate the net worth of some stocks: Given a collection of ticker symbols and amounts it goes to Yahoo and finds the latests trading prices and adds up the net worth. Not too complicated really. I'm sort of following along with the book in my own code, but I'm also sort of straying. In later posts I'll show the actual application, but for know I just want to show some of the tests I've written along the way.


I'm using ScalaTest for testing which comes in a couple of flavors of which I've chosen the BDD styled FlatSpec style. And I'm really enjoying it. Let's look at some of the things I like about it. First, here are just two simple tests for the constructor of my code under test:


class PortfolioManagerSpec extends FlatSpec with ShouldMatchers {
  def withFakeStockPriceFinder(
        fileName: String)
       (testFunctionBody: (PortfolioManager) => Unit) {
    testFunctionBody(new PortfolioManager(fileName)
      with FakeStockPriceFinder)
  }


  "A PortfolioManager" should "be instantiable with a valid file name" in {
    withFakeStockPriceFinder("src/configuration/stocks.xml")
     {pm =>}
  }


  it should "throw an exception when given an invalid file name" in {
    evaluating {
      withFakeStockPriceFinder("fakefilename") {pm =>}
    } should produce[FileNotFoundException]
  }
  ...
}


The ScalaTest flavor is chosen simply by extending one of several traits, in this case FlatSpec. FlatSpec gives me the ability to specify test cases in " should in" fashion as seen in the first test above. To me doing that in a statically typed language is awesome, and goes to show how flexible Scala really is. The second test starts with "it", which means that it continues on the line of testing of the test above. In this case "it" lets me avoid repeating the "A PortFolioManager" part.


Even these simple tests already puts Scalas strong support for functions into play: The withFakeStockPriceFinder method at the top is a curried method that takes first a string argument, and then a function argument, testFunctionBody. The method instantiates the object under test and passes it into the testFunctionBody. Both test cases above use withFakeStockPriceFinder to contruct the object under test, and have it passed into an anonymous function where the actual test code is written. Furthermore the second test makes use of some of the things from the ShouldMatchers trait, namely the evaluating, should and produce methods. Evaluating takes a function argument and executes it, much like my own withFakeStockPriceFinder, but it handles any exceptions thrown and lets me set up expectations for exceptions by calling the should and produce methods in a fluent fashion. -Note that dots and parenthesis in method calls are optional in Scala, as long as there are no ambiguities. This is IMHO is also awesome, and again goes to show how flexible Scala is.


Before showing the rest of my PortfolioManagerSpec I want to touch on another point: TDD is - as we know - a very disciplined way of working, and can be hard to follow all the time but somehow I find that the style of testing promoted by the FlatSpec and ShouldMatchers nudges me towards shorter red-green cycles and a more stringent test-first practice, than I usually have with NUnit. I'm not sure why that is, but it has to do with the way tests are declared with strings rather that method names, I think.


Anyway here's the whole PortfolioManagerSpec:


class PortfolioManagerSpec extends FlatSpec with ShouldMatchers {
  def withFakeStockPriceFinder(
        fileName: String)
       (testFunctionBody: (PortfolioManager) => Unit) {
    testFunctionBody(new PortfolioManager(fileName)
      with FakeStockPriceFinder)
  }


  "A PortfolioManager" should "be instantiable with a valid file name" in {
    withFakeStockPriceFinder("src/configuration/stocks.xml") 
      {pm =>}
  }


  it should "throw an exception when given an invalid file name" in {
    evaluating {
      withFakeStockPriceFinder("fakefilename") {pm =>}
    } should produce[FileNotFoundException]
  }


  "A PortfolioManager for stocks.xml" should "find 15 symbols and units in the stocks.xml file" in {
    withFakeStockPriceFinder("src/configuration/stocks.xml") {
      pm =>
        val tickersAndUnits = pm.tickersAndUnits
        tickersAndUnits.size should be (15)
    }
  }


  it should "find APPL, NSM and XRX ticker symbols" in {
    withFakeStockPriceFinder("src/configuration/stocks.xml") {
      pm =>
        val tickersAndUnits = pm.tickersAndUnits
        tickersAndUnits should (contain key ("APPL") and contain key ("NSM") and contain key ("XRX"))
    }
  }


  it should "find 200 ORCL" in {
    withFakeStockPriceFinder("src/configuration/stocks.xml") {
      pm =>
        val tickersAndUnits = pm.tickersAndUnits
        tickersAndUnits("ORCL") should be (200)
    }
  }


  it should "produce a asset report and calculate the total net worth" in {
    withFakeStockPriceFinder("src/configuration/stocks.xml") {
      pm =>
        val (totalNetWorth, report) =                  
          pm.generateTotalNetWorthAndSimpleReport
        totalNetWorth should be (146250)

        print(report)
    }
  }
}

Tuesday, March 2, 2010

Redoing the ArgParser in Scala

So I'm learning Scala and one the first things I've done is re-write the little argument parser sample I did for the Declarative over Imperative piece on 97 Things Every Programmer Should Know. I expected the Scala version of the argument parser to turn out somewhat different, shorter and more readable than the C# version. Whether it did or not I'll get back to towards the end of the post.
Now lets turn to the code. First of all I have a type called Argument which represents an argument that the parser should recognize:

class Argument(val name: String,
val action: String => Unit,
val helpText: String)

This is the complete declaration of the Argument class which is a value type with three immutable properites; name, action and helpText. The parts worth noticing is first of all how Scala combines the type declaration, propery declaration and constructor into a single line, allowing for really DRY declarations of value types like this one. Secondly notice the action property. That's a function type. In this sample the action is the piece of code that should handle the arguments matching the name property.
The declaration of the ArgParser type itself (with the whole implementation yanked out) is:

class ArgParser(private val args: List[String],
argumentSpecification: List[Argument]) {
//code omitted
}
Again the declaration of the class itself and a private immutable property - args - are condensed to a single line. Notice the second argument; a list of Argument objects. That list defines the arguments the parser is able to recognize, but internally I would like to store those arguments in a map from argument names to argument objects instead of as a list. The code that does that transformation is:

private val argumentMap: Map[String, Argument] =
(Map[String, Argument]() /: argumentSpecification){
      (map, a) => map+((a.name, a))
}
Notice the /: method. In Scala a method can be called anything including names consisting of special characters, like /:. The /: method performs a fold left on the list argumentSpecification, i.e. it applies the code block immediately after the call to each element in the list, while passing the result of one such invocation as an argument to the next. The end result is the result of the last invocation. Again I get a pretty concise piece of code.
The actual parsing of the args parameter is done by looping through the args list, looking up matches in the argumentMap, and invoking the action:
def parse {
for(i <- 0 until(args.length, 2)) {
args(i) match {
case "--help" => prettyPrintHelpText
case x if argumentMap.contains(x) => argumentMap(x).action(args(i + 1))
case _ =>
}
}
_hasParsed = true
}
That's pretty similar to the C# version.

The complete argument parser looks like this:
class ArgParser(private val args: List[String],
           argumentSpecification: List[Argument]) {
   args.length match {
     case 1 if (args.first == "--help") =>
     case x if (x % 2) == 1 =>
        throw new IllegalArgumentException
     case _ =>
  }

private val argumentMap: Map[String, Argument] =
(Map[String, Argument]() /: argumentSpecification){
(map, a) => map+((a.name, a))
}
private var _hasParsed = false
private var _inputFileName = ""

def hasParsed = _hasParsed

def parse {
for(i <- 0 until(args.length, 2)) {
args(i) match {
case "--help" => prettyPrintHelpText
case x if argumentMap.contains(x) =>
argumentMap(x).action(args(i + 1))
case _ =>
}
}
_hasParsed = true
}

private def prettyPrintHelpText {
println("Usage:")
argumentMap.values.foreach { argument =>
       println("\t" + argument.name + ": " + argument.helpText)
     }
   }
 }
Apart from a couple of syntactical things, like Scalas way of declaring properties and constructors and its way of accepting closures as arguments to methods, the Scala ArgParser code is pretty close to the C# ArgParser code. Why? Well I see a couple of possible reasons:
  1. I'm only just learning Scala now, so maybe I'm not thinking in Scala idioms and style
  2. Using a map from names to arguments containing lambda function for the handling of the arguments is actually the way to implement this sample, regardless of the language.
  3. C# has sufficiently many functional features that moving to a more functional language doesn't make much of a difference for simple cases like this ArgParser.
Which one is the right explanation I don't know. If you have opion please leave a comment.

Saturday, February 6, 2010

Take Away Points from Javagruppens Annual Conference

Having just come back from Javagruppens annual conference, the main things on my mind from the conference are:

I Must Learn Scala!
Scale is so terse, so flexible, and oh so cool. I must sit down and learn it properly soon. In fact I sort of cant wait :-).
I dont expect to be using Scala for anything serious any time soon (although you never know, do you), but I hope to challenge my way of thinking about code and design.

Eclipse RCP
I'm starting to realize how much ground the Eclipse RCP (Rich Client Platform) actually covers . -Its really cool and really solid. The framework is mature and includes all sorts of enterpisy stuff. E.g. there was a Eclipse BIRT (Business Intelligence Reporting Tools) presentation at the conference, which showed a "from naught to reports and a nice report designer integrated in your app in 2 hours" demo. Quite convincing. Seeing that I've done much more .NET than anything else over last few years, this is foreign ground, but for LOB desktop apps it's differently very serious competition to WinForms. Must keep this in mind for coming projects.

Is Flex for Me? -Still on the Fence
Another cool demo showed how to code up Flex clients to Grails backends, which may seem sort of odd at first, but the argument is that Flex is easy to use, well tooled, Flash is ubiquitous, and Flex apps also run on Air, so they run cross OSs, cross browsers and out of browsers. Is that enough to take the leap? I don't know. It still seems aimed at very thin clients, so why not stick with html+css+javascript? For desktop apps (i.e. Air apps in this case) I find they tend to need complicated functionality in which case I'm not convinced the Flex+actionscript ecosystem is rich enough compared to say, .NET or Ecplise RCP.

And, BTW it was really fun to be out of my element for a few days!