The Brad Hole

KotOR is a C++ based app, and is based on a cross-platform engine. It uses OpenGL natively on both the PC and the Mac. My rough estimate shows that it contains roughly 500 moderately-sized source files, and it uses precompiled headers in both the CodeWarrior and Xcode builds.

So let's establish the baseline time for CodeWarrior on my test Mac, a 2.5GHz G5 with 1 gig of RAM. I used CodeWarrior 9.2 and Xcode 1.5. This is the total build time in CodeWarrior and Xcode (dual-CPU):

Debug build:

CodeWarrior: 6 min, 30 seconds
gcc: 10 minutes

Release build:

CodeWarrior: 11 min, 35 seconds
gcc: 10 minutes, 10 seconds

Interestingly, gcc's compile time for the debug build (i.e. the optimizer turned off) is almost identical to the when the optimizer is turned to the max. Related to that, the performance in gcc's non-optimized build is much better than CodeWarrior's. For KOTOR at least, this means that a debug build in Xcode isn't worth it vs. the release build.

With that baseline in place, let's get to the meat. I had at my disposal a dual-800 MHz G4 Mac, a 1.25 GHz G4 PowerBook, and a sad little 500 MHz G3 iBook. :) I messed around with hosting the Xcode build on various Macs, as well as removing one CPU on the host to see how that affected performance. I had heard that sometimes its best in a distributed setup not to use both CPUs on the host to keep overhead manageable. Here are the results of all the permutations I tested:

800 MHz G4 x 2 (host), 2.5 GHz G5 x 2 (remote) : 12 minutes, 13 seconds.
800 MHz G4 x 1 (host), 2.5 GHz G4 x 2 (remote) : 12m 13s
2.5 GHz G5 x 2 (host), 800 MHz G4 x 2 (remote) : 8m 25s
2.5 GHz G5 x 1 (host), 800 MHz G4 x 2 (remote) : 10m 20s
2.5 GHz G5 x 2 (host), 800 MHz G4 x 2 (remote),
1.25 GHz G4 (remote) : 7m 33s
2.5 GHz G5 x 2 (host), 800 MHz G4 x 2 (remote),
500 MHz G3 (remote), 1.25 GHz G4 (remote) : 8m 20s

As you can see, I was able to start beating the performance of CodeWarrior when I hosted on the 2.5GHz G5 and added in the 800MHz G4. I trimmed a minute off that by adding in the 1.25GHz PowerBook. When I got greedy and threw the iBook into the mix, I lost any gains garnered by adding the PowerBook.

What's interesting about this is the time for the steps to generate the precompiled headers and do the final link. While running these tests, I noticed that these are significantly longer steps in Xcode than CodeWarrior. These steps are also performed solely on the host CPU, so the speed of the host is critical to the overall performance of the distributed build. Here are the times for those 2 steps combined on the 2 Macs I used as hosts:

2.5GHz host - linker/precompiled step time: 1m 45s
800 MHz G4 host - 3m 20s

If you subtract those steps from the aggregate build times above, you get a better feel for how the distributed compile affects things:

2.5 GHz G5 x 2 : 8m 15s
800 MHz G4 x 2 (host), 2.5 GHz G5 x 2 (remote) : 8m 53s
2.5 GHz G5 x 2 (host), 800 MHz G4 x 2 (remote) : 6m 40s
2.5 GHz G5 x 2 (host), 800 MHz G4 x 2 (remote),
1.25 GHz G4 (remote) : 5m 48s

Bottom-line, Xcode's distributed compiling option can beat CodeWarrior in release builds if you have a decent dual-CPU Mac sitting around to throw at things. Although I didn't do tests with the distributed compiler and the debug builds in Xcode, if the times hold fairly closely to the optimized builds, then you'll have to add more CPUs than I had available to approach CodeWarrior's performance. Most of our development turn-around is with the debug builds, and right now, CodeWarrior retains the edge.

For those of you curious about the actual performance of the game between CodeWarrior and Xcode, it's a virtual dead-heat. In my tests here, there is roughly a 1-2 frame per second advantage for CodeWarrior, which is well within the margin of error.

With gcc 4.0/Xcode 2.0 in Tiger, I hope to do these benchmarks again. I'm not expecting compile times to improve a whole lot, although I do hear that the optimizer in gcc 4.0 is much better now.