Why Not Use Custom Visual Studio Project Templates?

I got this question about the Nancy book the other day on twitter:

@chr_horsdal Hi Christian, in the book you don't seem to use the Nancy Visual Studio project templates (so far at least) just curious why?
— Richard Dalton (@richardadalton) October 6, 2013

The answer I gave was just a bit of hand waving in the general direction of the question:

@richardadalton in general I dont like using project templates a whole lot. I prefer using empty projects plus nuget packages.
— Christian Horsdal (@chr_horsdal) October 7, 2013

While it is true that, that is my preference, there is not much by way of explanation in that answer, so I'll try to explain a little more here.

Pre-baked project templates are there to help developers out. They are a way to get quickly up and running with a fully setup project. With some of the heftier templates everything is there from dependent assemblies, configuration files, and folder structures to skeleton code. Basically some template provide a full wlaking skeleton. Just like that. Click, click, click, and you're done.
I don't like that as a starting point for a code base. It's too much and the odds of such complete templates fitting just right to the project at hand are in my experience minuscule. So you end up changing the setup that came with the template. This is a battle I don't want to be fighting. Firstly because it will quickly off set any initial gain won by getting quickly up and running using the template, but secondly and more importantly because it is battle against accidental complexity that should never have been there in the first place. This is the wrong way around. We want to start out small and then only build up more stuff as it becomes needed. Not before, and definitely not from the get go. I saw a nice talk on the benefits of this at GOTO last week, the gist of which was how keeping it simple enabled evolving from a very simple - almost simplistic - initial system to a full blown system without loosing speed or ending up with a mess. In fact although not about agile that was the best talk on agile I've seen in a while.

How about smaller templates then? I like them a little better, but only a little. At the end of the day they suffer from the same problems.

How about really small focused templates? Those I can live with because there is so little left that there is a chance that it actually fits my situation, and if not the battle to overcome the template is small.

How do I prefer it then? In fact I'd prefer it if Visual Studio did not require project and solution files, but was able to just run off the build files I wrote - be it rake, psake, nant or even MSBuild. Since this is not the case the next best thing is starting off with an empty project template. That is the minimal amount of initial complexity. Then after that I can add the packages I need. With each of them I add some complexity too, but it is the complexity I've deemed worth while.

Of course there is a lot of gray scale here: When is a template close enough to empty? When does a NuGet package become a complete project template? These are case-by-case judgements that you will have to make for yourself.