Monday, December 20, 2010

Options for DCI on .NET

Here's a short summary of the technical/language options for doing DCI on the .NET platform. The list below is loosely ordered in order of increasing "mainstreamness" - according to me :-).
The list notes the language, how roles and contexts are handled and how the solution is run on .NET.

  • Scala "cross-compiled" to MSIL/CLR:
    In Scala contexts are simply objects instantiated at runtime from context classes. The contexts have the responsibility of instantiating data objects and mapping roles to them. Roles are traits that are mixed into data objects at instantiation time. So typically the contexts receive ids for data objects, pull the corresponding data from e.g. a database, and instantiate the data object with a role trait mixed in. In Scala this is all strongly typed and looks like this:

    class MoneyTransferContext(sourceAccountId: int,
                                sinkAccountId: int){
       val source =
            new SavingsAccount(sourceAccountId) with TransferMoneySource
       val sink =
             new CheckingAccount(sinkAccountId) with TransferMoneySink
       def execute = {
         source.deposit(100000)
         source.transferTo(200, sink)
         }
    }

    To run this in a .NET setting the Scala code must "cross-compiled" to MSIL, and must use the .NET standard library along with the Scala APIs. For details on this see my earlier post, or the introduction on the Scala language web site.

  • Python running on IronPython:
    In Python contexts are objects, that take the responsibility of mixing roles into data objects dynamically. The data objects can be instantiated outside the context or inside the context depending on taste. When the context is done the roles can be yanked back out of the data objects. I dont know a whole lot of Python, so I wont go into details, but refer you to Serge Beaumonts sample.
    To run this on .NET use IronPython.
  • Ruby running on IronRuby:
    In Ruby contexts are also objects, that take the responsibility of mixing roles into data objects dynamically. Data objects can be instantiated inside the context or outside the context. Roles in Ruby are implemented as modules, which using the Ruby standard library can be mixed into any Ruby object. The Ruby code looks like this:
    class TransferMoneyContext
       attr_reader :source_account, :destination_account, :amount
       include ContextAccessor
       
       def self.execute(amt, source_account_id, destination_account_id)
         TransferMoneyContext.new(amt, source_account_id,
                                  destination_account_id).execute
       end
     
       def initialize(amt, source_account_id, destination_account_id)
         @source_account = Account.find(source_account_id)
         @source_account.extend MoneySource
         @destination_account = Account.find(destination_account_id)
         @amount = amt
       end
    
       def execute
         in_context do
           source_account.transfer_out
         end
       end
    end

    To run this on .NET just use IronRuby. It runs fine out of the box.

  • C# and Dynamic:
    Using the dynamic features of C#, contexts are objects that receive either data objects or IDs of data objects. In case of an ID the context instantiates the data object and populates it based on the ID and some data source. In both cases the data object has a role reference which will point to an expando object, into which the roles methods are injected. The code looks roughly like this:
    dynamic transfer = new ExpandoObject();
     transfer.Transfer = (Action)
      ((source, sink, amount) =>{ 
        dynamic mSource = source;
        dynamic mSink = sink;
        mSource.DecreaseBalance(amount);
        mSink.IncreaseBalance(amount);});
     Account sourceAcct = new Account { CashBalance = 100m };
     sourceAcct.Role = transfer;
     Account sinkAcct = new Account { CashBalance = 50m };
     sourceAcct.Role.Transfer(sourceAcct,sinkAcct,10m);
    Running on .NET? Well, it's C#, so just do it.

  • C# and extention methods:
    Using C# extension methods for roles contexts are objects that receive either data objects or IDs of data objects. In either case the context maps the data objects to roles. Roles are implemented as static classes containing extension methods for role interfaces. Data objects capable of playing a given role implement that role interface. The code looks like this:
    public class TransferMoneyContext { public TransferMoneySource Source { get; private set; } public TransferMoneySink Sink { get; private set; } public decimal Amount { get; private set; } public TransferMoneyContext(TransferMoneySource source, TransferMoneySink sink, decimal amount) { Source = source; Sink = sink; Amount = amount; } public void Execute() { Source.TransferTo(Sink, Amount); } } public interface TransferMoneySink { void Deposit(decimal amount); void Log(string message); } public interface TransferMoneySource { decimal Balance { get; } void Withdraw(decimal amount); void Log(string message); } public static class TransferMoneySourceTrait { public static void TransferTo(this TransferMoneySource self, TransferMoneySink recipient, decimal amount) { // The implementation of the use case if (self.Balance < amount { throw new ApplicationException("insufficient funds"); } self.Withdraw(amount); self.Log("Withdrawing " + amount); recipient.Deposit(amount); recipient.Log("Depositing " + amount); } }

    How to run this on .NET is ... well ... pretty obvious :-)

Monday, December 6, 2010

RHoK'ing for Unicef

I was at the RHoK 2.0 Aarhus event this weekend, and got the opportunity to do a simple little web app for Unicef. The app itself is not that interesting for people outside of Unicef, but it will save real money for Unicef that they can use on something else...like helping children. The app is something like 80% done, and we did it in about 28 hours. Wow, it feels great to have been able to actually help Unicef just by doing a saturday to sunday hackathon along with a few other geeks.

The app was done in ASP.NET MVC 2, with Enitity Framework 4 and SQL Express underneath, and the experience was pretty good: The architecture just came out of the box with MVC. EF code first made thinking about the database schema a non-issue (and the load on the app is expected to be low, so I don't think will need to go in a tweak for performance later). We were able to get by with some of the auto generated CRUD views that Visual Studio can make for ASP.NET MVC model objects. All in all the development was pretty quick.
The code is on google code.

Friday, November 26, 2010

Slides from DCI talk at ANUG

I did a talk on DCI at my local .NET User Group, ANUG, the other night and had a great time. The slides from the talk were:




The slides were followed by three demos:

  1. A simple Scala example "cross"-compiled to run on .NET
  2. A simple Ruby example running on IronRuby
  3. A slightly larger ASP.NET MVC based C# example, which I will be posting details about in the furture.

Tuesday, November 16, 2010

Upcoming events

I just wanted to point to 2 events that are coming up soon that I am participating in:

  1. I am doing a talk on DCI in the Aarhus .NET User Group on the 24th of November 2010. Register at  http://on.fb.me/9QOa3s 
  2. I am participating in Random Hacks of Kindness during the weekend 4th and 5th of December 2010. More info and registration at: http://rhokaarhus.eventbrite.com/ 

Thursday, October 14, 2010

A Pinch of Functional in my C# - An Everyday Example

I'm working on a codebase that, like many other code bases,  has a number of data access objects(DAOs) that all inherit a common superclass. The other day I got tired of looking this pattern of code repeated a lot in the DAOs:
    1        public MyDTO GetMyDTO(...)
    2         {
    3             DataContext DB = NewDBContext;
    4             try
    5             {
    6                 DB.Connection.Open();
    7                 //Fetch something from the DB and return it
    8             }
    9             finally
   10             {
   11                 if (DB.Connection.State == ConnectionState.Open)
   12                     DB.Connection.Close();
   13             }
   14         }

I wanted the creation of the DBContext and the handling of the connection to be factored into somewhere reusable. So I applied the functional "suround with" pattern and introduced this method in the superclass:
    1        protected TReturn UsingDBConnection<TReturn>(Func<DataContext, TReturn> func)
    2         {
    3             DataContext DB = NewDBContext;
    4             try
    5             {
    6                 DB.Connection.Open();
    7                 return func(DB);
    8             }
    9             finally
   10             {
   11                 if (DB.Connection.State == ConnectionState.Open)
   12                     DB.Connection.Close();
   13             }
   14         }
and the DAO code then turns into this:
    1        public MyDTO GetMyDTO(...)
    2         {
    3             return UsingDBConnection(DB =>
    4             {
    7                 //Fetch something from the DB and return it
    7             });
    8         }

C# has supported this for years now, so why am I posting it? -Because I wish I saw more of this type of code. To me it's easy to read and I get to reuse the common part.


Saturday, October 2, 2010

Notes from getting Scala running on .NET

I just spend an evening getting Scala to run on .NET 4.0. Here my notes:

Scala 2.8.0

I started downloading Scala 2.8.0-final as a zip file (), and unpacking it to my scala playground directory. That gave me:
PS C:\Scala-playground\scala-2.8.0.final> tree
Folder PATH listing for volume OSDisk
Volume serial number is 00690066 6A5B:A532
C:.
+---bin
+---doc
¦   +---sbaz
¦   +---sbaz-setup
+---lib
+---meta
¦   +---cache
+---misc
¦   +---sbaz
¦   ¦   +---config
¦   ¦   +---descriptors
¦   +---sbaz-testall
¦   ¦   +---tests
¦   +---scala-devel
¦   ¦   +---plugins
¦   +---scala-tool-support
¦       +---a2ps
¦       +---bash-completion
¦       +---bluefish
¦       +---context
¦       ¦   +---Highlighters
¦       ¦   +---Template
¦       +---emacs
¦       ¦   +---contrib
¦       +---enscript
¦       +---geshi
¦       +---intellij
¦       +---jedit
¦       ¦   +---console
¦       ¦   ¦   +---commando
¦       ¦   +---modes
¦       +---latex
¦       +---notepad-plus
¦       +---scite
¦       +---subethaedit
¦       ¦   +---artwork
¦       ¦   +---Scala.mode
¦       ¦       +---Contents
¦       ¦           +---Resources
¦       ¦               +---English.lproj
¦       ¦               +---Scripts
¦       +---textmate
¦       ¦   +---Bundles
¦       +---textpad
¦       +---textwrangler
¦       +---ultraedit
¦       +---vim
¦           +---ftdetect
¦           +---indent
¦           +---plugin
¦           +---syntax
+---src

I then went on to install the Scala msil extension (i.e. the scala .NET compiler and runtime) as follows:
PS C:\Scala-playground\scala-2.8.0.final> bin/sbaz.bat install scala-msil

Then, following the "official" instructions on running scala on .NET (http://www.scala-lang.org/node/168), I copied mscorlib.dll to the scala lib folder:
PS C:\Scala-playground\scala-2.8.0.final> copy C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorlib.dll .\lib

I was now ready to try to compile something. E.g. this very simple hello world program using a little bit of .NET, namely Console.WriteLine:

import System.Console object test extends Application {   Console.WriteLine("Hello world!") }

I tried to compile it like this:
PS C:\Scala-playground\hello.net> ..\scala-2.8.0.final\bin\scalac.bat -target:msil .\test.scala

but got this error:
scala.tools.nsc.MissingRequirementError: class scala.runtime.VolatileBooleanRef not found.
        at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:513)
        at scala.tools.nsc.symtab.Definitions$definitions$.getClass(Definitions.scala:471)
        at scala.tools.nsc.symtab.Definitions$definitions$.newValueClass(Definitions.scala:620)
        at scala.tools.nsc.symtab.Definitions$definitions$.BooleanClass(Definitions.scala:92)
        at scala.tools.nsc.symtab.Definitions$definitions$.initValueClasses(Definitions.scala:643)
        at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:787)
        at scala.tools.nsc.Global$Run.(Global.scala:597)
        at scala.tools.nsc.Main$.process(Main.scala:107)
        at scala.tools.nsc.Main$.main(Main.scala:122)
        at scala.tools.nsc.Main.main(Main.scala)
error: fatal error: class scala.runtime.VolatileBooleanRef not found.

I tried like this to point out where mscorlib and the scala runtime dlls are:
PS C:\Scala-playground\hello.net> ..\scala-2.8.0.final\bin\scalac.bat -target:msil -Xassem-extdirs ..\scala-2.8.0.final\lib .\test.scala
scala.tools.nsc.MissingRequirementError: class scala.runtime.VolatileBooleanRef not found.
        at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:513)
        at scala.tools.nsc.symtab.Definitions$definitions$.getClass(Definitions.scala:471)
        at scala.tools.nsc.symtab.Definitions$definitions$.newValueClass(Definitions.scala:620)
        at scala.tools.nsc.symtab.Definitions$definitions$.BooleanClass(Definitions.scala:92)
        at scala.tools.nsc.symtab.Definitions$definitions$.initValueClasses(Definitions.scala:643)
        at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:787)
        at scala.tools.nsc.Global$Run.(Global.scala:597)
        at scala.tools.nsc.Main$.process(Main.scala:107)
        at scala.tools.nsc.Main$.main(Main.scala:122)
        at scala.tools.nsc.Main.main(Main.scala)
error: fatal error: class scala.runtime.VolatileBooleanRef not found.
..same error.

Then I resorted to frantic googling, but ended up deciding to downgrade the mscorlib.dll to 2.0, and to 1.1. But in both cases I got same depressing result.
Finally I decided to downgrade to scala 2.7.7.

Scala 2.7.7

I downloaded scala 2.7.7 unzipped it and got:
PS C:\Scala-playground\scala-2.7.7.final> tree
Folder PATH listing for volume OSDisk
Volume serial number is 00690066 6A5B:A532
C:.
+---bin
+---doc
¦   +---sbaz
¦   +---sbaz-setup
+---lib
+---meta
¦   +---cache
+---misc
¦   +---sbaz
¦   ¦   +---descriptors
¦   +---sbaz-testall
¦   ¦   +---tests
¦   +---scala-tool-support
¦       +---a2ps
¦       +---bluefish
¦       +---context
¦       ¦   +---Highlighters
¦       ¦   +---Template
¦       +---emacs
¦       ¦   +---contrib
¦       +---enscript
¦       +---geshi
¦       +---intellij
¦       +---jedit
¦       ¦   +---console
¦       ¦   ¦   +---commando
¦       ¦   +---modes
¦       +---latex
¦       +---notepad-plus
¦       +---scite
¦       +---subethaedit
¦       ¦   +---artwork
¦       ¦   +---Scala.mode
¦       ¦       +---Contents
¦       ¦           +---Resources
¦       ¦               +---English.lproj
¦       ¦               +---Scripts
¦       +---textmate
¦       ¦   +---Bundles
¦       +---textpad
¦       +---textwrangler
¦       +---ultraedit
¦       +---vim
¦           +---ftdetect
¦           +---indent
¦           +---plugin
¦           +---syntax
+---src
The I went through copying mscorlib and installing scala-msil again:
PS C:\Scala-playground\scala-2.7.7.final> copy C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorlib.dll .\lib
PS C:\Scala-playground\scala-2.7.7.final> .\bin\sbaz.bat install scala-msil
planning to install: scala-msil/2.7.7.final
Installing...

Then I tried compilling again:
PS C:\Scala-playground\hello.net> ..\scala-2.7.7.final\bin\scalac-net.bat .\test.scala
PS C:\Scala-playground\hello.net> ls


    Directory: C:\Scala-playground\hello.net


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----        30-09-2010     21:59            target
-a---        30-09-2010     22:33       3863 test.msil
-a---        30-09-2010     21:52        100 test.scala

which went well. The scala compiler made a .msil file. Yeah.

Next step was turning that .msil into an .exe:
PS C:\Scala-playground\hello.net> C:\Windows\Microsoft.NET\Framework\v4.0.30319\ilasm.exe .\test.msil

Microsoft (R) .NET Framework IL Assembler.  Version 4.0.30319.1
Copyright (c) Microsoft Corporation.  All rights reserved.
Assembling '.\test.msil'  to EXE --> '.\test.exe'
Source file is ANSI

Assembled method test::$tag
Assembled method test::main
Assembled method test$::.ctor
Assembled method test$::.cctor
Assembled method test$::$tag
Assembled method test$::main
Assembled global method Main
Creating PE file

Emitting classes:
Class 1:        test
Class 2:        test$

Emitting fields and methods:
Global  Methods: 1;
Class 1 Methods: 2;
Class 2 Fields: 1;      Methods: 4;
Resolving local member refs: 8 -> 8 defs, 0 refs, 0 unresolved

Emitting events and properties:
Global
Class 1
Class 2
Resolving local member refs: 0 -> 0 defs, 0 refs, 0 unresolved
Writing PE file
Operation completed successfully

Which gave:
PS C:\Scala-playground\hello.net> ls


    Directory: C:\Scala-playground\hello.net


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----        30-09-2010     21:59            target
-a---        30-09-2010     22:35       2560 test.exe
-a---        30-09-2010     22:33       3863 test.msil
-a---        30-09-2010     21:52        100 test.scala

So I tried running the program:
PS C:\Scala-playground\hello.net> .\test.exe

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'predef, Version=0.0.0.0, Culture=
neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
   at Main(String[] args)

but missed the predef assembly which is part of the scala-msil extension. To get passed that I simply copied all dlls from the Scala lib folder and ran again:
PS C:\Scala-playground\hello.net> mv .\test.exe .\target
PS C:\Scala-playground\hello.net> cp ..\scala-2.7.7.final\lib/*.dll .\target
PS C:\Scala-playground\hello.net> cd .\target
PS C:\Scala-playground\hello.net\target> ls

    Directory: C:\Scala-playground\hello.net\target


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        30-09-2010     22:30    2070528 mscorlib.dll
-a---        30-09-2010     22:30    1180160 predef.dll
-a---        30-09-2010     22:30       5632 scalaruntime.dll
-a---        30-09-2010     22:35       2560 test.exe
PS C:\Scala-playground\hello.net\target> .\test.exe
Hello world!

Success! but only on the next to newest Scala version. Either way I had fun :-)

Saturday, September 25, 2010

Spec Style Unit Tests in C#

Spec style unit tests are unit tests that focus on making the specs clear, and then relating some test code to each part of the spec. The Specs Scala unit test framework is really good at that (e.g. check this earlier post), but I want to be able to do the same in C#.


Lets look at an example: A method for deciding whether a given year is a leap year. The rules for leap years are:


A leap year should
    be any year divisible by 400
    but not other years divisible by 100
    and all other years divisible by 4


In other words the above are the specs for the method. So I would like that to be clear from the unit test. I would also like the unit test to make it clear what test code corresponds to what part of the spec. Therefore I would like the spec to be part of the test, and the test code to be mixed right into that spec text. Something like:


"A leap year" should
{
  "be any year divisible by 400" asIn
{
// test code here
}
  "but not other years divisible by 100" asIn
{
// test code here
}
  "and all other years divisible by 4" asIn
{
// test code here
}
}


such that the spec is clearly visible as the strings, and such that the 'asIn' indicates that the code in the following scope tests the part of the spec right before the 'asIn'. To achieve something close to this I've made three simple extension methods on string: 'should', 'asIn' and 'andIn'. Each one of these extension methods take a delegate as an argument. That makes it possible to write this code:

   16             "A leap year".should(() =>
   17             {
   18                 "be any year divisible by 400".asIn(() =>
   19                 {
   20                     // Test code here
   21 
   22                 })
   23                 .andIn(() =>
   24                 {
   25                     // More test code
   26                 });
   27 
   28                 "but not other years divisible by 100".asIn(() =>
   29                 {
   30                     // Test code here
   31                 });
   32 
   33                 "and all other years divisible by 4".asIn(() =>
   34                 {
   35                     // Test code here
   36                 })
   37 

Because this is C# there I need to the dots and the arrows ( ()=> ) in there, but I still think the spec is pretty clearly visible. The extension methods themselves are very simple: They basically just execute the delegate passed to them, which means that all the test code snippets in the above will be executed. The complete code for the extension methods is:




    1 using System;
    2 
    3 namespace Specs
    4 {
    5     public static class SpecExtensions
    6     {
    7         public static void should(this string unitUnderTest,
    8                                   Action executableSpecification)
    9         {
   10             Console.WriteLine(unitUnderTest + " should");
   11             executableSpecification();
   12         }
   13 
   14         public static string asIn(this string readableSpecification,
   15                                   Action executableSpecification)
   16         {
   17             Console.WriteLine(@"    " + readableSpecification);
   18             return RunExecutableSpecification(
   19                     readableSpecification, executableSpecification);
   20         }
   21 
   22         public static string andIn(
   23             this string readableSpecification,
   24             Action executableSpecification)
   25         {
   26             return RunExecutableSpecification(
   27                     readableSpecification, executableSpecification);
   28         }
   29 
   30         private static string RunExecutableSpecification(
   31             string readableSpecification,
   32             Action executableSpecification)
   33         {
   34             executableSpecification();
   35             return readableSpecification;
   36         }
   37     }
   38 }


Combining that with NUnit, the test for my leap year checker becomes:




    1 using NUnit.Framework;
    2 
    3 namespace Specs.Test
    4 {
    5     [TestFixture]
    6     public class LeapYearTest
    7     {
    8         [Test]
    9         public void LeapYearSpec()
   10         {
   11             "A leap year".should(() =>
   12             {
   13                 "be any year divisible by 400".asIn(() =>
   14                 {
   15                     Assert.That(LeapYearChecker.IsLeapYear(0),
   16                                 Is.True);
   17                     Assert.That(LeapYearChecker.IsLeapYear(400),
   18                                 Is.True);
   19                     Assert.That(LeapYearChecker.IsLeapYear(800),
   20                                 Is.True);
   21                 })
   22                 .andIn(() =>
   23                 {
   24                     Assert.That(LeapYearChecker.IsLeapYear(-400),
   25                                 Is.False);
   26                 });
   27 
   28                 "but not other years divisible by 100".asIn(() =>
   29                 {
   30                     Assert.That(LeapYearChecker.IsLeapYear(100),
   31                                 Is.False);
   32                     Assert.That(LeapYearChecker.IsLeapYear(100),
   33                                 Is.False);
   34                     Assert.That(LeapYearChecker.IsLeapYear(200),
   35                                 Is.False);
   36                     Assert.That(LeapYearChecker.IsLeapYear(1000),
   37                                 Is.False);
   38                 });
   39 
   40                 "and all other years divisible by 4".asIn(() =>
   41                 {
   42                     for (int i = 0; i < 100; i += 4)
   43                     {
   44                         Assert.That(LeapYearChecker.IsLeapYear(i),
   45                                     Is.True);
   46                     }
   47                 })
   48                 .andIn(() =>
   49                 {
   50                     for (int i = 1600; i < 1700; i += 4)
   51                     {
   52                         Assert.That(LeapYearChecker.IsLeapYear(i),
   53                                     Is.True);
   54                     }
   55                 })
   56                 .andIn(() =>
   57                 {
   58                     for (int i = 1601; i < 1700; i += 4)
   59                     {
   60                         Assert.That(LeapYearChecker.IsLeapYear(i),
   61                                     Is.False);
   62                     }
   63                 });
   64             });
   65         }
   66     }
   67 

The specs are not as easily visible once all the code is there, but I still think its easy to find the specs. What do you think? Is there too much clutter of dots and arrows and stuff? Is the calling back and forth between the test and extension methods too strange? I'm on the fence. One part me really, really likes this. One part of me really, really doesn't.
Oh, and for completeness the code under test is:




    1    public static class LeapYearChecker
    2     {
    3         public static bool IsLeapYear(int year)
    4         {
    5             return year >= 0 &&
    6                 (year % 400 == 0 ||
    7                 (year % 4 == 0 && year % 100 != 0));
    8         }
    9     }

Thursday, September 9, 2010

Don't Stay Stuck

If you write code professionally, you will have days where you find yourself utterly stuck on a problem. In fact, some days coding feels like trying to wade through quicksand: The harder you struggle to get out the more stuck you find yourself. Getting into one of those situations where you are stuck and can't find a way out is perfectly normal. And it is perfectly acceptable for a professional programmer as well. Every programmer finds themselves in those situations from time to time. Getting stuck is a natural part of creative work like programming, and as such we must simply accept that we get utterly stuck now and then. The important part is what you do about it. As with quicksand the trick is to relax a bit, sit back, and realize that you need to seek help. Not all programmers realize this, but the best ones do.
When faced with an insurmountable problem some programmers just press on and try to work through it. That usually means they start thrashing; they run around in circles without getting anywhere. While this might keep them warm, and even make them seem really busy and hard working it doesn't help the project. Some programmers sort of lose interest and start surfing the web or drinking inordinate amounts of coffee. That doesn't help the project either. To get out the situation you're stuck in, you probably need help - I know, I always do. So seek help.
Seeking help can mean a number of things depending on the problem you're facing. If the problem is on the code level try asking a teammate to come and pair with you - or if you're already pairing, ask to switch pairing partner - that extra set of eyes usually helps. If the problem is a design issue, again ask a teammate to help you out, but get up from the keyboard and go to the white board. Again the extra set of eyes - or even just the process of explaining the problem - usually helps. If the problem isn't technical but has to do with say access to the right tools, the right level of support from third parties, or unrealistic deadlines, ask your project manager for help. And if he or she is not able or willing to help, don't be afraid to escalate the problem to his or her boss - it's the professional thing to do. As I said: Seeking help can mean a number of different things.
But whatever you do, don't just stay stuck. Do something about it.

Friday, August 20, 2010

Immediately Functional? -Really?

Following up on my last post I want to focus one point in the second decision diagram in that post. The diagram is a rule of thumb for deciding what you want a framework to implement for you, and it looks like this:

I want to focus on the "immediately functional" box. It says that you only want to use features in a framework that work in your context as they are. What does that mean? What if the feature needs to be configured just right to work in your context? Or needs to be extended through some extension point? Or needs to be modified slightly by cracking open the framework code? Or will work in the next release according to the frameworks roadmap? Lets look a these in turn.
Is configuration OK? Yes. Of course it is...But only to a certain point beyond which you drown, then you're usually better off building the whole feature by yourself. And keep in mind that frameworks that dont ask you to do a lot of configuration, while still implementing enough are (or course) preferable. Conventions are good at that.
Is extending through extension points OK? Yes. Of course it is. Iff its done through supported extension points. If you extend through something that isn't meant to be an extension point, you will probably have a hard time maintaining that extension, and would usually be better off building the whole feature specifically for your purpose.
Is tweaking the framework code itself OK? No, and not even if you're in a position to merge your changes back into the framework. Because even if in that case, you're now the maintainer of part of the framework, which is not what you set out to be, when you chose to use a framework. In this case you're better off building your own and maintaining that at your own pace, and according to your own needs.
Is waiting for a feature in the next release OK? It depends...as they say :-). It depends on when you absolutely need the feature, and on when you need to start developing on the feature. In other words it depends on whether you have reached the last responsible moment or not. The last responsible moment is the moment at which you have to make a decision.It not just any moment at which you can make a decision. That's likely too early. It's not simply the last minute, because at that point you dont necessarily have a choice anymore. It's the when you reach the point where you need to start developing the feature but still have time to think your choices through and make a decision. If you dont really need the feature yet, postpone to a later iteration, and decide on whether to use the framework for that feature or not, at that time. Maybe you can postpone long a enough for that new release of the framework to come out. At that point deciding whether to use the implementation in the framework or to build your own is probably easy. If you can't wait that long before starting developing the feature, you might still be able to postpone the framework vs DYI decision: Create a clear interface to what you expect the framework to implement in the next release, and then - when you have to - either use the framework to implement the interface or roll your own implementation. You might even hedge the your bets by defining the interface and simultaneously develop against the interface and rolling your own implementation of the interface. If the framework should be released with a suitable feature while your still working on your own implementation, you have the option to switch without throwing too much away, on the other hand if the framework doesn't release anything suitable you're well underway too.

It's about making the decision at right moment - which is at the last responsible moment because that is when you have the most, and best, information to base the decision on, while still maintaining the freedom to actually decide for yourself.

Wednesday, August 11, 2010

What Should a Framework Implement?

In my last post I wrote how you want a framework to give you a way out, when parts of the framework turns out to be too constraining. In this post I want to talk a bit about what you actually want a framework to cover in the first place. But first a little detour.

Build or Buy?
When an organisation needs a new software system, should it build the system (possibly using contractors/consultants) og buy the system (as in find a suitable COTS solution)? That's one of those questions being asked all the time, and it's one those questions that - in general - can only be answered with "it depends...". It's also a question I'm faced with from time to time in my work, and the first answer is always "it depends..".
I watched this talk by Neal Ford on emergent design recently and about 3/4 through the talk he adresses the build og buy question using this diagram:



The diagram says the new system should only be bought if it is not strategic (i.e. not part of what differentiates the organization), if there is a fully functional candidate COTS solution readily available, and that candidate is extensible (i.e. provides ways out, when the solution is too constraining). Especially the first clause I find interesting. Everything that is strategic, and thus everything that is part of what differentiates the organization should be purpose build in order get all the details exactly right, so as to support and further the differentiation.

So in a build situation, then what? Well, often one of the early things done is to choose a framework to build on.

What Should a Framework Implement?
In choosing a framework the first thing to consider IMO is what you actually want the framework to cover. If you've already decided not to buy a COTS solution, then there must be some part of the system you think should be purpose build. On the other hand you dont want to build everything from scratch (e.g. if you're building a web app for renting cars you dont want to start by building a web server). You want to concentrate your efforts on the really value adding parts of the system. Those parts are the bits that set this system aside from all others (in the car rental example that would be the parts for selecting cars, handling special offers, making reservation and so on). These are the differentiating parts. So you want a framework that implements all (or most of) the things that are not differentiating the system. But only if it is fully funcitonal and readily available, and only if it provides ways out.

So I end up with this diagram for deciding whether a given part of the system should be implmented by a framework or purpose build:



Looks familiar?

Thursday, July 22, 2010

Frameworks and Way Out

Frameworks are a very big part of software today. Framworks often provide much of the architecture of a system because a framework comes with a large number of design decisions alreay made. E.g. a Grails project uses MVC and active record patterns, it runs on java infrastructure, and so on.

In this essay in design by Len Bass, three sources of initial design hypothesis for a future system are listed: Existing similar systems, frameworks, and patterns. The essay points out that existing similar systems probably provide most guidance for the new systems, frameworks a bit less guidance and patterns least guidance. This makes frameworks a more desirable source of initial design/architecture than patterns.

While I agree with this overall, I want to point out a particular caveat in this post: Frameworks should always provide a way out. E.g. a Grails project might start out being served well by active record, but at some point active record may turn out to be too constrained for a few parts of the system. It is crucial to be able to bypass the framework in those few parts while still using the framework in the rest of the system. To continue the example Grails opens the possibility to drop down to Hibernate if active record is too constrained, and Hibernate in turn opens the possibily to drop down to SQL if so needed. Not all frameworks open these ways out, which means that you have to work against and around the framework when the framework becomes too constraining. Doing so invariably results in something quite unmaintainable, because those workarounds are weird and hard to understand, and because new versions of the framework likely will invalidate the workarounds.

Lastly, in my experienc projects always run into a few of those cases where the chosen framework is too constaining. So my advice here is to remember to consider what the frameworks you conisder allows in thoses cases where you want a way out.

Thursday, June 17, 2010

Divide and Accumulate with Scala Actors

In Scala we can easily do things in parallel with actors; actors are easily started up, they are cheap, and they can easily send objects to each other
.
Here I'll look at how to use actors to parallelize in situations where we need to do some sort of calculation and aggregation over a collection of objects. E.g. given a collection of RSS or ATOM feeds pull down the list recents items from each one and build the aggregated list of items from all the feeds ordered by publication time. Doing that sequentially is sort of trivial, and involves waiting for a request to each feed one at a time. It would by nice to fire all those requests in parallel, then wait for them to return thier results and build the aggregated list as the results arrive. The following diagram shows how to do that with actors:




















Let the program start an actor per feed. Let each actor go do a request for current items to the feed it is in charge of, and then send that list as an immutable object to the accumulating actor. The accumulating actor will maintain the aggregated list of feed items. When a new list of items is received from one of the other actors it merges that list into its aggregated list and at the end it has the full aggregated list.
Going a step further; why not have the accumulating actor send an immutable copy of the aggregated list to e.g. the view every tiume the aggregated list has been updated. That us allows us to show the user the partial results as they arrive.

Notice that the above did not involve any explicit starting of threads, no synchronization of threads, and no locking. Those things are handled behind the scenes by the actors and their message queues. Furthermore there is only mutable state in one place; inside the accumulating actor. Everything sent from one actor to another is immutable. So we've parallelized safely and easily.

The problem we're solving here is - admittedly - fairly easy to parallelize, but nonetheless I think the solution I described is nice because it is easy to understand and to code.

Tuesday, May 25, 2010

Why Do I Like Spec-Style Unit Tests?

As I've written about earlier I've been spending some time lately learning Scala. A side effect has been learning and using two test frameworks I haven't used before; ScalaTest and Scala Specs. Both offer what I think of as a spec style of writting tests. That is, tests along the lines of:

"A unit under test" should {
  "be tested by specs like this" in {
     // Code  for the first actual case
  }
 
  "and like this one" in {
    // Code for the second test case
  }
}

where a string is used to tell what's under test, and strings are also used to give each test case a name. In contrast "traditional" [J|N]Unit tests use method names to give each test case a name, and either don't point out the unit under test or use a class name for that.

The spec style of testing, I find, makes a test first workflow feel more natural, and makes my red/green cycles shorter, leading to more interaction between tests and code, and to better coverage. Very nice :-). I noticed this after using ScalaTest for a while, and found the same later when I started using Specs. Somehow both these frameworks gently nudge me towards a more clean TDD workflow.

I think the reason is that the unit under test is very explicitly named and that the test cases are named with strings rather than method names. Using a strings rather than method names makes me write longer names, and makes me write names that form full sentences when combined with the unit under test and the 'should' from the 'should' method. As in the example above which reads as "A unit under test should be tested by specs like this and like this one". The full sentences turn the tests into something a bit closer to specs, or rather, for me, the full sentences turn the tests into the place where I think about what exactly I want my classes to do, which makes it natural to write them first. On the other hand the natural language of full sentences makes me a little more aware of details, and therefore makes each test smaller and more focused which is what makes my red/green cycles shorter when using the spec style.

Smaller red/green cycles and a more test first approach has the usual benefits of turning the test into a design tool that leads to low coupling and high cohesion in the code under test, of creating high test coverage, and of catching coding bugs very early on.

That's why I like the spec style approach to unit testing.

Wednesday, April 28, 2010

First Adventures in ScalaTest - Part III: Introducing Actors in the NetWorth Sample

This is the third post about my little net worth Scala sample. This third post really doesn't have a lot to do with ScalaTest, so the title is probably somewhat misleading. What this post is about is how to introduce some parallelism in the net worth calculation code by using Scala actors. The first post in the series showed some test written in ScalaTest, the second one showed the net worth sample itself, and this one starts where that post ended: The code for calculating the net worth based on an XML file of stock symbols and units is:
val stocksAndUnitsXml = scala.xml.XML.load(fileName)

 val tickersAndUnits =
    (Map[String, Int]() /: (stocksAndUnitsXml \ "symbol")) {
      (map, symbolNode) =>
        val ticker = (symbolNode \ "@ticker").toString
        val units = (symbolNode \ "units").text.toInt
        map(ticker) = units
    }

 def generateTotalNetWorthAndSimpleReport = {
    val report = new java.io.ByteArrayOutputStream
    Console.withOut(report) {

      println("Today is " + new java.util.Date)
      println("Ticket  Units  Closing Price($)  Total value($)")

      val startTime = System.nanoTime

      val netWorth = (0d /: tickersAndUnits) {
        (cumulativeWorth, symbolAndUnitsPair) =>
          val (symbol, units) = symbolAndUnitsPair
          val lastClosingPrice = getLatestClosingPriceFor(symbol)
          val value = lastClosingPrice * units

          println("%-7s  %-5d %-16f  %-16f".format(symbol, units, lastClosingPrice, value))

          cumulativeWorth + value
      }

      val endTime = System.nanoTime

      println("Total value of investments is $" + netWorth)
      println("Report tool %f seconds to generate".format( (endTime - startTime)/1000000000.0 ))

      (netWorth, report)
    }
  }
That code is explained in my last post.

Now I want to parallelize that calculation. Specifically I want to parallelize the web service calls to get the latests price for each stock symbol, so I'm going to fetch each price in a separate actor and then send the prices back to the main thread and accumulate the net worth there.

The code that gets the price for a single symbol and sends it to another actor called 'caller' is:
    caller ! (symbol, getLatestClosingPriceFor(symbol))
To do that in a separate actor for each symbol I call this method:
   private[this] def getLatestPricesForAllSymbols(caller: Actor) =
     tickersAndUnits.keys.foreach {
       symbol =>
         actor {
           caller ! (symbol, getLatestClosingPriceFor(symbol))
         }
    }
  }
the call to 'actor' starts a new actor an executes the function given to it asynchronously. The '!' method on the actor 'caller' sends the value given as an argument to the '!' method to 'caller'.

To receive a single symbol and price I do:
  receiveWithin(10000) {
    case (symbol: String, lastClosingPrice: Double) =>
      val units = tickersAndUnits(symbol)
      val value = lastClosingPrice * units
  }
'receiveWithin(10000)' blocks until a message arrives for the current actor, or times out after 10 seconds, so if this code is in the 'caller' actor it will receive one of the symbol/price pairs sent using '!' in the code above.

In order to accumulate the whole net worth I place the above in a function that also takes an 'cucumulativeWorth' and returns an updated cumulative worth:
   private[this] def receiveAndProcessOneSymbol(cumulativeWorth: Double) = {
    receiveWithin(10000) {
      case (symbol: String, lastClosingPrice: Double) =>
        val units = tickersAndUnits(symbol)
        val value = lastClosingPrice * units

        println("%-7s  %-5d %-16f  %-16f".format(symbol, units, lastClosingPrice, value))

        cumulativeWorth + value
    }
  }
and to receive all the symbol/price pairs I call that method as many times as there are symbols in the 'tickersAndUnits' map:
  private[this] def receiveAndAccumulateWorthForAllSymbol =
    (0d /: (1 to tickersAndUnits.size)) {
      (cumulativeWorth, index) =>
        receiveAndProcessOneSymbol(cumulativeWorth)
    }
Putting all that together the 'generateTotalNetWorthAndSimpleReport' becomes:
  def generateTotalNetWorthAndSimpleReport = {
    val report = new ByteArrayOutputStream
    Console.withOut(report) {
      calculateNetWorthAndWriteReportTo(report)
    }
  }

  private def calculateNetWorthAndWriteReportTo(report: ByteArrayOutputStream) = {
      writeReportHeader

      val startTime = System.nanoTime
      getLatestPricesForAllSymbols(self)
      val netWorth = receiveAndAccumulateWorthForAllSymbol
      val endTime = System.nanoTime

      writeReportFooter(netWorth, endTime, startTime)

      (netWorth, report)
    }

  private def writeReportHeader = {
      println("Today is " + new java.util.Date)
      println("Ticket  Units  Closing Price($)  Total value($)")
    }

  private def writeReportFooter(netWorth: Double, endTime: Long, startTime: Long): Unit = {
    println("Total value of investments is $" + netWorth)
    println("Report took %f seconds to generate".format((endTime - startTime) / 1000000000.0))
  }

  private[this] def getLatestPricesForAllSymbols(caller: Actor) =
    tickersAndUnits.keys.foreach {
      symbol =>
        actor {
          caller ! (symbol, getLatestClosingPriceFor(symbol))
        }
    }

  private[this] def receiveAndAccumulateWorthForAllSymbol =
    (0d /: (1 to tickersAndUnits.size)) {
      (cumulativeWorth, index) =>
        receiveAndProcessOneSymbol(cumulativeWorth)
    }

   private[this] def receiveAndProcessOneSymbol(cumulativeWorth: Double) = {
    receiveWithin(10000) {
      case (symbol: String, lastClosingPrice: Double) =>
        val units = tickersAndUnits(symbol)
        val value = lastClosingPrice * units

        println("%-7s  %-5d %-16f  %-16f".format(symbol, units, lastClosingPrice, value))

        cumulativeWorth + value
    }
Now the calculation is done by fetching prices in parallel using the lightweight Scala actors, and sending the results back as one way messages that are aggregated to the final result in the main thread. That was easy, don't you think?

Oh, and finally even in this little sample and on a small dataset this actually does speed up things. If I rerun the integration test shown in the last post:
class IntegrationSpec extends FlatSpec with ShouldMatchers {
  def withYahooStockPriceFinder(fileName: String)(testFunctionBody: (PortfolioManager) => Unit) {
    testFunctionBody(new PortfolioManager(fileName) with YahooStockPriceFinder)
  }
 
  "A PortFolioManager with the YahooStockPriceFinder" should "produce a asset report and calculate the total net worth" in {
    withYahooStockPriceFinder("src/configuration/stocks.xml") {
      pm =>
        val (totalNetWorth, report) = pm.generateTotalNetWorthAndSimpleReport
        totalNetWorth should be(85000d plusOrMinus 10000)

        print(report)
    }
  }
}
I get:
Today is Wed Apr 28 20:37:38 CEST 2010
Ticket  Units  Closing Price($)  Total value($)
MSFT     190   30,930000         5876,700000    
INTC     160   23,220000         3715,200000    
ALU      150   3,150000          472,500000     
ORCL     200   25,940000         5188,000000    
NSM      200   14,970000         2994,000000    
CSCO     250   27,170000         6792,500000    
AMD      150   9,550000          1432,500000    
IBM      215   130,060000        27962,900000   
VRSN     200   26,840000         5368,000000    
XRX      240   10,980000         2635,200000    
APPL     200   0,000000          0,000000       
HPQ      225   53,330000         11999,250000   
ADBE     125   35,420000         4427,500000    
SYMC     230   17,270000         3972,100000    
TXN      190   26,380000         5012,200000    
Total value of investments is $87848.55
Report took 0,837941 seconds to generate
which shows that the calculation took less than a second now, whereas the calculation took over 6 seconds in the sequential version.
Oh, and also notice that the symbols appear in a different order in the report than they did in the last post. The input data is exactly the same. The difference is that they are printed in the order that the main thread receives prices for them, and that ordering is not deterministic any more because it depends on how fast each web service call returns.

And that's it. I'm still having fun with learning Scala :-)