Streamed Lines: Branching Patterns for Parallel Software Development

• Introduction to SCM Patterns (diagram)
• Parallel Development
• Branching for Effective Parallel Development
• Forces of Branching and Parallel Development
• Branching Patterns and their Participants
• Using the Branching Patterns

• General Advice and Recurring Themes
• Branching Traps and Pitfalls to Avoid
• Effects of the Branching Patterns
• Appendix A - Streamed Lines: The Patterns
• Acknowledgements
• References

• T1. Merge-a-phobia
• T2. Branch-a-holic
• T3. Merge-mania
• T4. Continual Cascading
• T5. Spaghetti Branching
• T6. The Unknown Branch
• T7. The Never-ending Branch

• T8. "Big Bang" Integration
• T9. The Wrong-Way Merge
• T10. Run-away Merge
• T11. Development Freeze
• T12. Codeline Pudding
• T13. Integration Wall

Sometimes strict locking is appropriate; but when it isnt (and for parallel development efforts it typically is not) then attempting strict locking often results in a lot of people twiddling their thumbs, waiting for someone else to checkin the file they need to checkout. The resulting deadlock and resource contention can be far more costly (in terms of wasted time and effort) then the time and effort that goes into integration and merging (provided the latter is done intelligently - which is what "Streamed Lines" is all about).

Patterns like selecting appropriate branching styles and merging styles should get you started on the right foot. Making sure you integrate early and often will help keep you on track. There is no one pattern presented here that will cure all of these fears and problems; but almost every single pattern presented here provides some way of addressing these issues in various situations. Using the patterns together addresses the larger issue.

Sometimes its better to split an oft-used file into two or more files (if it tried to hold too much at once). Similarly, branching directories is something not to be taken lightly. Also, branches are often not the best way to resolve multi-platform issues (though I admit there are times when they are).

But I strongly suspect such "overbranching" may be more of an attempt to use branches as a "hammer" to make every problem look like a nail. Something that seemed good to do in one place is hastily applied universally to other places without stopping to think if it's really appropriate in these other situations (or if it's "okay," but something else might be better).

For shops performing more loosely serialized development (rather than massively parallel efforts), variants like Branch per Major Task, Codeline per Major Release, may be a better fit than branching for every task and release. Similarly, a LAG Line or Docking Line may be a better fit than Staged Integration Lines. Sometimes it may even make more sense to use a Virtual Codeline instead of a normal branch.

Although it might seem counterintuitive at first, this problem is also a frequent side-effect of failing to merge early and often enough, or from restricting merges to be performed only by the codeline owner. The frequent changes that are merged all at once, or else all by the same person can simply be too large a cross to bare. Sharing the burden with others, or distributing it more evenly over time may be the answer.

Most of the recovery and prevention strategies mentioned for overbranching can be applied here as well (like LAG Line and Virtual Codeline). For merging problems with hard-to-eliminate branches and codelines, there are essentially three alternatives to address this problem (and they may be used together in concert wherever appropriate):

Redistribute the merging burden across time

The merging interval (or integration rhythm) may be too long, involving too many complex changes at once. merge early and often or one of its variants can address this. If it seems the integration rhythm is too fast and frenetic, you might try increasing the interval, or you might consider one of the other strategies below. Using Deferred Branching rather than Early Branching can also make a big difference if it seems like codelines are still necessary but are being created too fast or too soon.

Redistribute the merging burden across people

If codelines are very restrictive and codeline owners are too overwhelmed trying to perform all the merging, try sharing the burden with others. Use more relaxed access lines and employ patterns like MYOC and Docking Line.

Redistribute the merging burden across space

Refactor the organization of your codelines to more effectively support collaboration and communication by spreading, or else more centrally coordinating merge efforts across functional areas (system space) or geographic areas (physical space). Patterns like Mainline, Subproject Line, Component Line, Remote Line, Third Party Line, and Inside/Outside Lines, can help you achieve this.

Other possible, but less frequent causes are overuse, or inappropriate use, of subproject lines, component lines, or platform lines (or all of the above).

This can also be true when checking out files to the workspace. It often requires more time (and more storage) to construct versions far removed from the main trunk than those on the main trunk or very close to it. Cascading to the extreme can cause noticeable performance degradation for checout/checkin, often before branch pathnames begin to apprpoach tool and system limits.

Sometimes this results from referring to changes from one unfinished or unintegrated task within some other change-task. Sometimes it results from creating change-task branches off of other change tasks branches in unorderly ways.

Another possibility is that the branch may indeed have a single purpose, but it might not be very coherent. Perhaps the "end-criteria" are vague. But there is some kind of "fear of commitment": that one last "tweak" that is always deemed necessary. It meanders on and on and we're never quite sure if its actually "completed" or in limbo.

In one case, the main problem is that the branch lacks internal coherence, whereas in the other case, the branch primarily lacks internal cohesion. In both cases, the conceptual integrity of the branch has been compromised: it fails to encapsulate a single purpose.

A similar situation arises in programming when the same variable gets used for many purposes in the same subroutine or block of code. The resulting confusion and multi-purpose use of the variable can be a frequent source of coding errors and runtime mishaps. Such situations also arise in the case of modules and functions that are too large and try to do too much instead of splitting things up into smaller and more manageable "chunks" delegated to other modules or functions. Similarly, using branches for many purpose at the same time can also result in integration errors and merging mishaps.

It is possible to take the "single purpose" concept too far. The purpose must be not only coherent and cohesive, but it has to be both feasible and appropriate for the given project. Just because every branch should have a purpose does not imply that every purpose should have a branch. A branch "for every purpose under heaven" would be overkill, and would probably qualify for branch-a-holic (another case of using branches as a hammer to make every problem look like a nail).

One way to guard against runaway-branches is with a branch-cutoff upon submit. This is the same thing as a Codeline Freeze/Retirement for a single task-branch. When a branch is submitted or completed, it is locked against all further access. Deliberate actions must be taken to unlock the branch if it really does need to be "reopened" or "resumed".

To a certain extent, Codeline Policies and Codeline Conventions can help ensure that the work performed on a branch remains consistent with its originally intended purpose. Self-Documenting Codelines can also be used to help ensure that the nature of work that is supposed to take place on the branch is readily visible or accessible for frequent reminders. Perhaps most importantly, Merging Early and Often can be used to establish frequent and regular integration intervals which serve as periodic checkups to ensure that developers don't try to cram too many changes into a branch in between integrations.

A Self-Documenting Codeline can be particularly useful here because it makes the information more visible and more readily accessible. And the locality of reference between the codeline and its policy makes it easier to keep the policy current.

So the never-ending merge is either caused by earlier problems which make the merge so difficult as to be incredibly time-consuming, or else it is caused by lack of conceptual integrity of the integration effort.

One alternative is to freeze the release-line, but allow it to sync-up with the mainline so another codeline can be spawned for work on subsequent releases (as in overlapping release lines. Another common solution is spawn off a subbranch for the release-engineering effort, while letting subsequent development efforts continue on the original codeline. This is essentially parallel releasing/development lines where the release-engineering codeline becomes a maintenance codeline once the release goes out the door.

Using either of the above strategies may require the use of either Codeline per Release or else Codeline per Major Release if you aren't already using either one. And of course you should already be using a mainline as well.

Other times, the codeline instability may simply be a case of when the latest versions on the codeline aren't necessarily the greatest versions to rely upon for subsequent work.

If developers are checkin in their changes to the codeline one file at a time over a period of time, then their changes may not sense in the context of the codeline until all the changes are completed and checked-in. In this case the solution may simply be to use a change-transaction model of checkin: this ensures that all changes to all files for a single change-task are checked-in all at once as a group-checkin (preferably as an atomic operation or transaction if at all possible). This will ensure that no change-tasks are ever partially checked-in to the codeline.

If, despite using group-checkin for all revisioins in a change-task, the problem is that "latest is greatest" no longer holds true for the codeline, then stop trying to use the latest version of everything on the branch as the most recent stable configuration of the codeline. In which case you'll have to use labels or "tags" to indicate the most recent stable configuration or "baselevel" upon which to base development work carried out on that codeline. Sometimes a Docking Line can achieve a good balance between developers merging early and often, and requiring stable baselevels as the basis of codeline changes.

Branches are good for isolating risk into separate paths off the overall workstream, but if we take it too far we isolate the workers from each other a bit too much, and we let them become too far removed from how their roles and tasks fit into the overall team, system, and lifecycle. If we lose awareness of others tasks and roles and how each of those pieces of the puzzle fit together to form the "whole" team, and the "whole" system, and the "whole" lifecycle, then people stop needing to care about those things, and become concerned only with their own roles and tasks and little else.

Social Isolation: Development roles and tasks are hierarchically organized in a way that not only isolates development tasks from one another, but isolates project team members from one another. The right hand doesn't know (or can't appreciate) what the left hand is doing. People forget (or never even understood) how all their different roles are supposed to fit together.

To promote and maintain healthy team communication and interactions in parallel development efforts, we must strive to isolate work, not people. Otherwise instead of team synergy we get team lethargy, with divisive and derisive behavior among the troops. So the most important "patterns" here are patterns of project management (rather than configuration management) to maintain team synergy and social health.

Patterns like MYOC (Merge Your Own Code) and Relaxed-Access Line can help if they are sufficient to mitigate the risk of unstable codelines, or if the associated cost is not too terribly tragic. In higher-risk projects requiring greater control, adding another level of integration such as a Docking Line or Staged Integration Lines can work well when they are combined with strategies like having integrators and developers work together side-by-side to merge non-trivial changes; having integrators notified when changes are ready to be integrated; ensuring that developers work in close proximity with integrators and communicate frequently; and having developers also serve as integrators and codeline-owners for at least one codeline, or else periodically switch/rotate between the roles of integrator and developer, so that everyone who needs to gets a chance to play both roles and appreciate what each one does.

And of course integrate early and often can help ensure that these collaborations and rotations take place frequently, to ensure constructive and productive team behavior as well as correct and consistent codelines.