Measuring aircraft by weight

[brooksmoses]

Computer programmers tend to have a strong aversion to measuring a programming project by the number of lines of program in the source code. There's a famous quote by Bill Gates about the matter:

"Measuring programming progress by lines of code is like measuring aircraft building progress by weight."

Obviously, you make progress in designing an aircraft by figuring out how to make it lighter weight, not by making it heavier.

The thing is, though, that like any measurement the problem is typically not in making the measurement, but in what you do with it. In particular, though the number of lines of code in a project is certainly not be useful for determining how close to finished it is (given that much of the work is in the debugging and testing stages), it can be used to estimate the total amount of "progress" -- that is, programmer time and effort -- that's required in the project in total. For a given type of project, with a given programming team (or similar teams), the amount of effort is roughly proportional to the number of lines to be written.

You can argue this by handwaving about making estimates and in this case or that case it worked, but there's still the compelling imagery of that quote to contend with. So ... what really happens if you try to measure aircraft by weight that way? I was curious and decided to try it. Just like with software, we can't expect similar results over very different kinds of airplanes, but we can consider similar ones -- for instance, Boeing jetliners. Thus, an experiment: Take the operating empty weight of Boeing's current planes (I pulled the data from Wikipedia), and their cost as a proxy for effort required (Boeing helpfully has a 2010-average-price list on their website), and make a simple linear regression for everything from the 737 to the 777. Then, use that to predict what a new 787 Dreamliner will cost.

Can't possibly work very well, can it?

Here's the chart. The gray line is the linear fit to all the pre-787 planes.



And there you have it. Absolutely spot on (0.02% error) for the 787-8, and not too bad (11.2% error) on the 787-9. If I could estimate the effort in my software projects to within 12% of the actual effort that easily, I'd be absolutely thrilled.

Comments

[alecaustin.livejournal.com]

Wow, yeah. I *wish* we could get work estimates that accurate in games development. (Or when we have them, that we could actually tell our clients that what they've expressed a desire for is unworkable. *sigh*)

[ckd]

I suspect the real metric to look at is weight per seat, since OEW and price are both correlated with capacity.

[brooksmoses]

Yup -- and I expect there's also a pretty strong market force driving the price-per-seat to fall in the same place for different planes, too.

Actually, I could see what happens if I add some Airbus data ... and, yeah, it falls on the same line. The matchup between the 737 and A320-family points is remarkably close, and the A330 and A340 are pretty close to the same line -- a little under it, as it turns out, but not significant. What's a bit startling is that the regression overpredicts the cost of an A380 by almost 30%.

On an actual price-per-seat graph, the regression isn't really straight, but has a bit of concave-downwards curvature -- the 747 and A380 are both notably cheaper in price per seat than most everything else, which is a bit of economy of scale for you, I assume. I haven't done weight per seat yet.

[trom.livejournal.com]

I've always hated that quote. Because given the design phase airplane manufacturers know to a reasonable (if not high )degree what the final weight will be. So at some level if you know the current weight of the build in progress you know how much is left to do.

Out of curiosity how good a measure is LOC an indicator of closeness to the point where the only changes are due to qa/debugging etc?

I wonder what a similar chart to yours for LOC for the linux kernel, gcc or windows would look like.

[brooksmoses]

Out of curiosity how good a measure is LOC an indicator of closeness to the point where the only changes are due to qa/debugging etc?

That's an interesting question -- unfortunately, on my most recent project (which was rare in actually starting from empty files) I don't think we were doing version-control commits at a rate that would really give good data for that. Anecdotally, though, there's quite a bit at the beginning where people are doing design and writing small "sketch" sort of programs, with a fairly good bit of progress for few LOC. The overall curve is sort of an S-shape, flat at the ends and sloping more steeply in the middle, and I don't really have much idea how linear the range in the middle would be.

I also would expect it to vary a lot from project to project -- for example, this recent one has some "extra" bits that we can separate off and some people will be programming those towards the end while other people are debugging the core pieces, so that part of the curve won't be as flat as it would for a project that couldn't be separated like that.

The interesting trick for things like the Linux kernel and so forth is finding independent numbers for the effort involved. The only ones I've seen have been estimates based on LOC.