Sunday, July 31, 2011

Developer Survey Results - Part 3

This is the third part of my walk through of the results of a survey I recently did (part 1, part 2). This time I focus on practices around continuous integration.

The survey shows that 58% of respondents use CI, which  is fewer than I expected.

Those who use CI have their CI servers do different things. The answers show respondents use their servers to:


Build
94%
Run unit tests86%
Run integration tests
75%
Run system tests
22%
Run UI tests
14%
Run acceptance tests
14%
Create release or deployment package
58%
Create tag/branch in source control on success
19%
Deploy to non-production environment (like test or staging)
22%
Deploy to production
11%
Other
8%

So - not surprisingly - people build their code on the CI server. Then they seem to run the tests they've written - the numbers for running different kinds of tests align nicely with the numbers for writing different kinds of tests (see part 1). Furthermore over half create releases of deployment packages. That's pretty high, I think - even if it's only just over half of the half of the respondents that use CI at all; i.e. about a quarter of all respondents. Only very few take the next step and actually deploy to production. This is as expected; continuous deployment is still pretty new.

Next question how often CI runs and it shows that by far most people have it running on every commit. 

Turning to how often things brake the distribution is as follows:

Most of the time11%
Several times a day3%
Once a day33%
Once a week42%
Once a month6%
Never
3%
Other
3%

So its seems something is broken sort of often. Is that a problem? -Not in my opinion. It just shows that the CI server finds stuff. What could be a problem is if this stuff isn't fixed. Asking about how long from something brakes to work to fix it is started the answer were:

Within 15 minutes
22%
Within an hour
47%
Within half a day
14%
Within a day
11%
Within a week
3%
Within a month
0%
Never
3%
Other0%
which I thinks shows that the issues found by CI are taken seriously.

All in all, it's a mixed impression on this one: Too few do any CI, but those who do have it build, run the tests they have, and create releases. And they react quickly to issues.