The Year of the Build
Is wrestling with your software builds a fact of life? Not necessarily.
by Peter Varhol
October 4, 2006
Building software used to be easy. You would compile the code you wrote, with few or no libraries and no dependencies. Moreover, if it built, it usually ran. Today, you might spend 20 hours or more building a complex application with hundreds of dependencies; and even if the build is successful, it probably has plenty of bugs, some of them serious.
Building software remains a black art. Enterprises developing custom applications build the software only at major milestones. While most see this as a convenience, it is instead putting off the day of reckoning. Once builds are attempted, they invariably fail. Getting the software to build takes developers away from writing code, and often requires costly rewrites of the software itself. When the build finally works, it's jury-rigged, almost certainly hard-wired, and likely to fail the next time anything changes in the build process.
Even development teams who set up nightly builds face issues. In most cases, the build system is cobbled together with an off-the-shelf compiler and difficult-to-read scripts that execute a motley combination of commercial, custom, and open source tools for compiling, linking, moving, and executing the application.
And branching? You plan for a month to create a branch for a specific point delivery, and then freeze all check-ins; you return the next week to start checking it again, and you continue checking in bug fixes to both branches for the next three months. And sometimes confusion reigns, so you check into both when you were only supposed to check into the mainline. The productivity hit taken by the development team for this conceptually simple exercise is incredible.
Take this scenario another step further. You can't assume that your application will run because it builds. So you use smoke tests. These run automatically at the conclusion of a successful build. If they fail, then the build processes must halt until the cause of the failure is found. Once again, delays ensue as efforts are turned toward tracing and fixing another spurious problem.
Taking these issues into account, the "Groundhog Day" scenario for most software developers can be summarized in a few sentences. You build at night, using an automated build process held together as if by magic. When everyone arrives in the morning, the build is broken. You blame the guilty developer, make him or her buy donuts for the team, fix the problem, check in the fix, and manually complete the build by late morning. Then the smoke tests fail. You find out what code failed, check in another fix, and manually complete the build again. It's now after lunch, and both developers and testers have been waiting for a clean build to continue work. The smoke tests finally succeed, and QA takes over. After an hour of productive activity, QA announces that there are two P1 blocking bugs that must be fixed before testing can continue. The development leads stop working, analyze the bugs, fix them or hand them off, and make the decision that it's too late in the day to kick off another build. The QA staff goes back to playing Solitare, and the rest of the development team, including you, completes their coding frantically and checks in modified files before the 6PM cutoff.
And this cycle begins again the next day. Every day.
Why You Build
What about building is so critical to the software development process? You can find the answer in the tenets of the Agile development movement. According to this movement, you prove that your software works by having working software. You create working software by building continuously, so that you can test the software to ensure it works. If you're using an Agile development process (XP, Scrum, or any other of the alternatives available), then you're intensely concerned about building and testing your software on an ongoing basis.
Reliable builds offer other advantages, too. They break the cycle of the Ground Hog scenario—a sequence of errors, delays, and temporary fixes that become permanent until the next time they break. You can focus on writing code rather than building software.
You also waste time and effort whenever a build fails. While fixing build breaks and other build issues is almost never an "all hands on deck" situation, the delays they cause to others in the development process can be massive. Build problems by themselves can hold up further development, testing, pilot projects, and, ultimately, the delivery of completed software.
For development organizations seeking certification under the auspices of the International Standards Organization (ISO) or the Software Engineering Institute's Capability Maturity Model Integration (CMMI), establishing and maintaining repeatable processes is a strict requirement. Certain industries, such as health care and aerospace, also face the need for definitive processes.
Solutions on the Horizon
The software build process is on the radar of vendors seeking further improvements and automation on the application development lifecycle (ALM). Several established vendors have just announced new build solutions as a key part of ALM processes—Borland with Gauntlet, and IBM with BuildForge. Both companies announced major initiatives into ALM, with build management being a keystone of these initiatives. Only recently have companies positioned build management as a critical aspect of their ALM solutions.
Back to top