10.2 Speed up Build Process

Scenario

In a continuous integration environment, it is very important that a build can complete as quick as possible to provide feedback to developers as early as possible after they submit a build verification request.

Solution

You can execute CPU intensive build steps in parallel, and you run them on different agent nodes. The results from different nodes can be collected back to server for further processing. Let's assume a product comprises of component1 and component2, and both of them take a long time to build. You can execute the build of component1 and component2 in parallel as follows:

The node match condition property for these steps can be specified as follows:

• master: node.isAgent()
  this tells the master step to run only on agent node (in order to [reduce server load])
• checkout: node==current.parent.node
  this tells the step checkout to run on the same agent as the master step
• build components: node==current.parent.node
  this tells the step build components to run on the same agent as the master step
• build component1: node.isAgent()
  this tells the step build component1 to run on agent node selected by load balancing algorithm
• build component2: node.isAgent()
  this tells the step build component2 to run on agent node selected by load balancing algorithm
• *package: node==current.parent.node
  this tells the step package to run on the same agent as the master step
• publish: node==current.parent.node
  this tells the step publish to run on the same agent as the master step

The idea here is that step build component1 and build component2 are designed to run on the least loaded agent nodes, while all other steps are running on the same agent node.

There are still a few problems. The nodes running step build component1 and build component2 are not guaranteed to execute on the same node that executes the step checkout. How can we make sure that the necessary files are available when the build of component1 and component2 starts? The same issue exists when running step package, how can we make sure that build results of component1 and component2 are available for packaging if the build steps and package step do not run on the same agent?

This is solved by specifying file requirements for steps. Let's assume that the agent node to run master step is agent1, and the checkout step retrieves files into two folders into workspace that resides on agent1: component1-src, and component2-src. Folder component1-src contains all files used to build component1, while folder component2-src contains all files used to build component2. Build results of comonent1 are saved to folder component1-binary, and build results of component2 are saved to folder component2-binary. Now we can specify following file requirements:

• for step build component1
  
  This will transfer folder component1-src (including all files inside it) from workspace of the agent running checkout step to the agent running step build component1

• for step build component2
  
  This will transfer folder component2-src (including all files inside it) from workspace of the agent running checkout step to the agent running step build component2

• for step package
  
  This will transfer:
  1. folder component1-binary (including all files inside it) from workspace of the agent running the step build component1 to workspace of the agent running the package step.
  2. folder component2-binary (including all files inside it) from workspace of the agent running the step build component2 to workspace of the agent running the package step.