Streamed Lines: Branching Patterns for Parallel Software Development

Brad Appleton	<brad@bradapp.net>	Motorola Network Solutions Group
Stephen P. Berczuk	<berczuk@acm.org>	NetSuite Development
Ralph Cabrera	<cabrerar@agcs.com>	AG Communication Systems
Robert Orenstein	<rlo@perforce.com>	Perforce Software

Copyright © 1998 by Brad Appleton, Stephen Berczuk, Ralph Cabrera, and Robert Orenstein.
Permission is granted to copy for the PLoP '98 conference.

Table of Contents
[printing/downloading instructions]
Introduction to SCM Patterns (diagram) Parallel Development Branching for Effective Parallel Development Forces of Branching and Parallel Development Branching Patterns and their Participants Using the Branching Patterns	General Advice and Recurring Themes Branching Traps and Pitfalls to Avoid Effects of the Branching Patterns Appendix A - Streamed Lines: The Patterns Acknowledgements References
Send us your comments!

Forces of Branching and Parallel Development

Parallel development raises several important issues and concerns for the success of the development projects. These risk-factors often become the driving forces which fundamentally shape many of the recurring problems and solutions of parallel development efforts.

Teamwork

Communication, effort, and interaction need to be effectively organized and executed for a parallel development effort to succeed. Important issues are:

Communication and Visibility: Team members need to stay reasonably abreast of what other team members are doing and why. Important changes and baselines need to be visibly and frequently communicated to the entire team, as well as to other important stakeholders. At the same time parallel efforts need to be insulated from one another and coordinated together so that changes can be successfully isolated and integrated.
One must be extra cautious not fall into the common trap of isolating team members from each other, or from their own efforts upstream. We must encourage frequent, visible communication of changes and status, even while carefully controlling and isolating parallel efforts and change-tasks.
Roles and Responsibility: If everyone is responsible for a thing, then often no one is. Features, changes, components, and milestones need responsible owners who understand their purpose and are held accountable for the success of their outcomes. These owners must also be visibly communicated to the whole team.
Workflow Organization: Large/complex tasks and changes need to be broken down into more manageable chunks and assigned to appropriate owners with appropriate milestones. Then the results must be continually recombined and reconciled into a working whole. Dependencies between tasks, changes, milestones, and their owners should be minimized to reduce the risks associated with safety, liveness, and reusability. And of course important tasks and milestones also need their status visibly tracked and communicated to the team (especially to the project manager).

Reusability

Reusability is the property that baselevels and changes can be readily used and incorporated in derived versions while still functioning correctly. We want to eliminate (or at least minimize) the amount of rework needed to incorporate someone else's changes into our own workspace, and into any new baselines. Whenever someone makes a change that other will want to see and use, we want others to be able to use those changes with little or no modification. Reusability concerns for parallel development include:

Reproducibility: If you can't reproduce the contents of a change or a baselevel, then you can't easily reuse it in developer's workspaces or in derived configurations.
Traceability: If you can't readily find and identify the contents of a change or a baselevel, then you can't easily reuse it.
Separability: If you can't readily disentangle desired changes from unwanted changes or baselevels, then you can't easily reuse those changes.

Safety

Safety is the property that nothing bad ever happens to your project. Safety concerns for parallel development are usually related to quality and include:

Codeline Consistency: You want to keep the codeline in a consistent state. That means it has all the changes it is supposed to have (no partial changes), and that they have been successfully integrated so they don't conflict with one another.
Codeline Reliability: Other developers rely upon the current state of the codeline to build and function correctly. If it doesn't, work may be held-up and efforts may be wasted.
>Codeline Integrity and Stability: At any given point in time, you want the codeline to be both consistent, and reliable. Over any given period of time, you want the codeline to be both reliably consistent, and consistently reliable. Otherwise changes which work correctly in isolation might fail when combined together; A single unanticipated change could destabilize the entire project.
Lost Changes: Parallel development without controlled interaction between concurrent changes can result in lost or corrupted changes, and wasted effort and rework.
Reappearing Bugs: Strategies need to be employed to ensure that defects in a prior version don't reappear in a later version of the product (especially when multiple versions are being developed concurrently)

Liveness

Liveness is the property that development progress ever takes place. Liveness is closely related to teamwork in general, and to development productivity in particular. Liveness concerns are also frequently at odds with concerns for safety (suggesting some kind of balance must be reached). Liveness concerns for parallel development include:

Increased Work Efficiency/Productivity: Increasing concurrency in a development project can permit more work to happen at once, resulting in decreased time-to-market.
Increased Coordination Efforts: Increasing the amount of concurrency increases the need for, and amount of, synchronization efforts (merging and integration). Merging and integration are often non-trivial activities which can be both time consuming and error prone. If taken too far or not planned intelligently, the synchronization overhead required by branching and merging can outweigh any productivity gains due to parallelizing development.
Contention and Work Stoppages: Excessive locking and "work queuing" may cause very long "busy waits" where people are forced to suspend some of their activities until a locked file becomes available. This could result in dormancy or even deadlock of parallel activities.
Build Time: The number of times the system needs to be integrated and built, and the duration of each system build can dramatically effect overall productivity and coordination. When system builds take only minutes to complete, many changes can be tested, integrated and communicated within a single day. When system builds take several hours, or even the better part of an entire day, significantly fewer changes can be tested, integrated, and communicated within a single day.
This affects not only communication and productivity, but also the visibility and frequency of new baselines. The fewer times we need to build, and the less that needs to be rebult each time, the more productive we can be, and the more we need to visibly communicate what has changed in the latest stable baseline of the system.
Merge Complexity: When merging concurrent changes, the nature and complexity of those changes can make a huge difference in the amount of collaborative effort needed to reconcile them. When each changes affects a set of lines or an algorithm that is distinct from the other changes, successfully merging the changes can usually be accomplished by a single developer using a decent file comparison tool. But when concurrent changes affect the same sets of lines or algorithms, successfully resolving the conflicts can easily require two or more developers.
In the extreme case, conflicting changes that affect fundamental portions of a subsystem or module may require the collaboration of many developers (perhaps even one or more managers) to conduct a redesign effort, and (re)build a shared understanding of the subsystem or component. This last scenario is one we desperately want to avoid by effectively managing teamwork and workflow.

SCM Tool Support

Support mechanisms provided (or lacking) in your SCM tools for change control/tracking and version control have a huge impact upon the success or failure of parallel development efforts. Some of these mechanisms are:

Branching Support: Not all VC tools provide a conceptually easy way of employing branches and using them to group functional changes. Many VC tools (such as RCS [Tichy85] and SCCS [Rochkind75]) indicate branches using a hierarchical version numbering scheme such as 1.2.3.4, where each subsequent component identifies a branch of its predecessor. Other VC tools (such as ClearCase [Leblang94], and Perforce [Seiwald96]) provide more symbolic hierarchical names, like /main/rel_1.1/fix_file_menu, which more easily lend themselves to comprehension by humans. CVS (see [CVS]) allows mnemonic "tags" to be used for branch names, but such branch names are not hierarchically composed into branch paths.
Merging Support: Merging is probably the most feared, and highest-risk aspect of parallel development. The degree of visual and automated merging assistance offered by a VC tool can make all the difference in the world. Most tools offer a file comparison program capable of showing changes between 2-3 different versions of the same text file. More VC tools are now offering GUI support for text-file merging and comparison, which is one of the main reasons why many are becoming more comfortable with branching for parallel development.
The better VC tools not only provide GUI merging assistance, they are capable of tracking merge-history information and dependencies, and take advantage of this by making automated merging easier to perform, and more likely to be correct. Most tools, however, seem to provide merge-assistance for text-files only. A select few tools are capable of providing (or being extended to provide) merge assistance for non-text files, such as word-processor documents and spreadsheets. Merging of non-text files can be extremely difficult (and often ill-advised) without such support.
Logical Change Grouping: Some SCM tools directly support the notion of logical changes (sometimes called change-tasks or change-packages): They are able to group together modifications to multiple files for the same coherent purpose as a single, logical change. This can be extremely useful for managing the functional configuration of a software product, and for identifying, tracking, and integrating modifications for a particular fix or feature.
Derived Object Reuse: Most software build tools (e.g., "make") are capable of reusing existing target object files within the workspace to avoid any unnecessary recompilation and/or relinking. Some software build tools (like ODIN and ClearCase's "clearmake") are capable of reusing target object files that were previously built in some other workspace to avoid even more unnecessary recompilation and/or relinking. These capabilities can greatly reduce the build-time for developers to test their changes before promoting them to be used for a new baselevel or baseline.
Customizability: Some SCM tools are more configurable than others. As nice as a tool might be, it can't possibly provide everything you could want with the invocation of a single command, macro, or mouse-click. Some tools provide customization and extension primitives using a programming and/or GUI interface.
Pluggable Event/Action Hooks: Some tools provide "triggers": user-definable "hooks" or "plug-ins" that execute at a specific time during an SCM tool event or operation. Triggers can help provide a seamless integration between your tools and processes. If your tool doesn't support triggers, or user-defined extensions, for a desired operation, you may have to resort to implementing "wrappers": home grown scripts or commands that serve as a front-end to certain tool features, or which compose those features together in a useful manner.

[back to the table of contents]

Send us your comments!