CI/CD for TwinCAT: The Real Cost of Scaling Self-Hosted Builds

TL;DR Self-hosted TwinCAT CI does not mean one Jenkins agent. Past a handful of developers you need a pool of real Windows machines, and that pool has to be provisioned, patched, and kept in a state where CI always knows which machines are actually available. At that scale, two builds can land on the same physical machine and corrupt each other’s TwinCAT runtime, producing failures that vanish on retry. The fix is a three-part Jenkins pattern: lock a specific machine, pin the build to it, and keep the lock registry synced with your real node pool automatically. Zeugwerk CI/CD already runs the machines and handles all of this, so you don’t have to.

Why one build machine stops being enough

This post is about what happens once you decide to run CI on your own infrastructure and it has to grow past one machine.

The first TwinCAT CI setup is usually simple: one Windows box with TwinCAT installed, one Jenkins agent or GitLab runner registered, pipelines pointed at main. On a small team with few concurrent builds, that can last quite a while.

Capacity is the obvious limit. Ten developers pushing to the same pipeline on one machine means queue time. A release branch and main both building at once means queue time. A hotfix on release/1.x while feature work continues on main means queue time. CI that developers learn to ignore because “it’ll run eventually” is CI that is not doing its job.

TwinCAT also adds constraints that most web teams never think about:

• Multiple runtime versions. Machines in the field may run TC3.1.4024 while new development targets 4026. Those need separate build pools, not just more slots on one machine.
• Parallel release lines. If you adopt Release Flow, release/1.x and release/2.x each need their own CI runs, at the same time, by design.
• Real hardware in the loop. A TwinCAT build is not just compilation. The runtime has to actually run so compiled code can be deployed and unit tests executed. That ties every build to a specific machine for the duration of the run.

So a team that takes CI seriously stops asking “do we need CI?” and starts asking “how many build machines do we need?” For a machine builder with several projects, several TwinCAT versions, and more than a handful of engineers, the honest answer is: several. Often more than that.

That is not a Jenkins detail. It is the starting cost of self-hosted TwinCAT CI.

What a build pool actually means

Container CI scales by adding pods. TwinCAT CI scales by adding Windows machines. Each one is a persistent asset you now own:

Adding a machine is not “install the agent and forget it.” A new node has to match the TwinCAT version label your pipelines expect, carry the right tools at the right versions, and stay online. When a machine goes down for patching, capacity drops. When someone provisions a node by hand and skips a step, you get builds that pass on one agent and mysteriously fail on another.

Most teams budget time for writing the pipeline. Nobody budgets time for running a small fleet of Windows build servers indefinitely. When the farm is not kept healthy, CI stops feeling like a safety net and starts feeling like a slot machine: flaky failures, long queues, developers re-running builds until something green sticks, and nobody quite trusting the result. That is also why managed TwinCAT CI exists: same build capacity, but everything in the table above is the vendor’s problem, not yours.

If you take on maintaining the farm yourself, provisioning is only half the work. The other half is a real, specific problem: using the pool without TwinCAT builds tripping over each other. Once you have more than one machine, a fleet in name if not yet in glory, the natural move is to register them under a shared label and let Jenkins, GitLab, or GitHub Actions dispatch each build to whichever host has a free slot. That sounds like it solves everything, and it is also exactly where things go wrong. Here is why, and the pattern that actually fixes it.

Why a shared pool label is not enough

Mainstream CI assumes build agents are stateless. Spin up a container, run the job, throw it away. Two jobs never share an environment because there is no environment to share. Pooling works because any slot is as good as any other.

TwinCAT breaks that assumption. A build needs a real Windows machine with the right TwinCAT version, a runtime actually running, and tooling activated on that specific host. You cannot spin up a disposable TwinCAT runtime in a container. The machine carries state between builds: runtime configuration, installed libraries, license activation. Two builds on the same machine at the same time are not isolated from each other, no matter what your pipeline definition says.

The straightforward Jenkins setup is to label all the agents with a pool name and let the scheduler pick one:

agent { label 'TC3.1.4024' }

Five machines carry that label. Jenkins is free to schedule on whichever has an open executor slot. If nodes allow multiple executors, two builds can legally land on the same physical machine at once.

Here is what that looks like in practice. Two pipelines trigger a few seconds apart. Both land on build-agent-03, on different executor slots. Build A starts a TwinCAT compilation. Build B, right behind it, restarts the runtime to get a clean test environment. Build A’s compilation was mid-flight when the runtime reset. Its project file is gone. Build A fails with a cryptic error that looks like a code problem. The developer re-runs it, unchanged, and it passes. Nobody ever connects the failure to two builds sharing a machine they should not have shared.

That kind of failure does not throw a clean error. It produces intermittent results that disappear on retry, which is exactly what makes it take days to track down instead of minutes.

Executor counts don’t fix this

The natural first response is to drop executor slots per node to one. That kills the collision, but opens a different problem.

Build nodes are rarely dedicated to TwinCAT alone. The same machine might also run integration tests or packaging jobs that do not need exclusive access and would happily run in parallel. Forcing every node down to one executor to protect TwinCAT builds serialises all of that for no reason.

There is also a second failure mode even with one executor per node. TwinCAT pipelines often have a nested stage that tries to grab another executor from the same pool while the outer stage is still holding its slot. That inner stage waits for a slot that will never free up, because the only thing holding it is the pipeline itself. A few concurrent builds doing this and the whole queue looks frozen, with no obvious error anywhere.

Either way, tuning executor counts is trying to solve a scheduling problem by adjusting a number. The real issue is that scheduling (find any free slot) and exclusivity (only one TwinCAT build touches this machine at a time) are not the same problem, and Jenkins does not connect them automatically.

The fix: lock the machine, then pin the build to it

Jenkins has two mechanisms that are easy to conflate.

An executor is a build slot on a node. agent { label '...' } asks the scheduler for any node with a matching label that has a free slot. It is looking for an available slot, not a specific machine.

A lockable resource is a named mutex, managed by the Lockable Resources plugin. A pipeline that calls lock(...) blocks until a specific resource, or one resource from a labelled pool, is exclusively available. While the lock is held, nobody else can acquire it.

Combining the two gives you exclusive machine access, but the order matters. If a pipeline declares a pool-level agent first and calls lock inside a stage, the executor gets assigned before the lock is checked. Two builds can both land on the same machine on different slots and then race for the same lock. The one that loses is already sitting on the machine, holding an executor, having arrived before any exclusivity check ran. Small window, same problem as before.

The order has to be: acquire the lock first, assign the agent second. The pipeline-level agent is none, so nothing is scheduled yet. The lock runs in options, before any stage, and writes the machine it granted into a variable. One outer stage pins to that exact machine. Everything else runs nested underneath it, on the same locked host:

pipeline {
  // No executor is consumed at pipeline start.
  // The lock must be acquired before any agent is assigned.
  agent none

  options {
    // Acquire one machine from the pool before any stage runs.
    // 'resource_name' receives the exact node name that was granted,
    // e.g. "build-agent-03".
    lock(label: 'TC3.1.4024', quantity: 1, variable: 'resource_name')
  }

  stages {
    stage('Make') {
      // Pin to the specific machine the lock resolved to, not the pool label.
      // Jenkins cannot schedule this on any other node.
      agent { label env.resource_name }

      stages {
        stage('Prepare')   { /* checkout, restore dependencies */ }
        stage('Build')     { /* compile, run tests              */ }
        stage('Artifacts') { /* archive outputs                 */ }
      }
      // Lock is held for the entire 'Make' stage and released when it exits.
    }
  }
}

The lock step resolves to one resource from the TC3.1.4024 pool, say build-agent-03, and writes that name into env.resource_name. The outer stage('Make') pins to that exact name, so Jenkins has no choice but to schedule everything on build-agent-03. The lock holds until the whole Make stage finishes or the pipeline is aborted.

The difference is small in code and large in behaviour: agent { label 'TC3.1.4024' } alone asks for a machine from the pool. Lock-then-pin guarantees one specific machine, held exclusively, no matter how many executor slots that machine happens to have.

The part people forget: keeping the pool in sync

The lock and the pin only work if every Jenkins node has a matching lockable resource with the right labels. Add a machine and forget to register its resource, and it silently never joins the pool. Let a node go offline without updating its resource, and pipelines will happily queue for a machine that is never coming back.

Doing that by hand does not survive contact with a real fleet. We use a small Groovy script, syncLockableResources.groovy, dropped into $JENKINS_HOME/init.groovy.d/. Jenkins runs everything in that folder at startup, and this script also registers a listener that fires on every node state change, so the registry never drifts:

// $JENKINS_HOME/init.groovy.d/syncLockableResources.groovy

import hudson.slaves.ComputerListener
import hudson.slaves.OfflineCause
import hudson.model.Computer
import hudson.model.Hudson
import hudson.model.Node
import hudson.model.TaskListener
import jenkins.model.Jenkins
import org.jenkins.plugins.lockableresources.LockableResourcesManager

def logger = java.util.logging.Logger.getLogger("lockable-resource-sync")

def computeLabels = { Computer c, Node node ->
    if (!c.isOnline()) return "${node.name} OFFLINE"
    return node.labelString + " " + node.name
}

def syncResource = { Computer c ->
    if (c instanceof Hudson.MasterComputer) return

    Node node = c.getNode()
    if (node == null) return

    def manager = LockableResourcesManager.get()
    def resource = manager.fromName(node.name)

    if (resource == null) {
        manager.createResource(node.name)
        resource = manager.fromName(node.name)
        resource.setEphemeral(false)
    }

    resource.setLabels(computeLabels(c, node))
    manager.save()
    logger.info("Synced '${node.name}' → '${resource.labels}'")
}

Jenkins.get().getComputers().each { c -> syncResource(c) }

def sync = syncResource

ComputerListener.all().add(new ComputerListener() {
    @Override void onOnline(Computer c, TaskListener listener)          { sync(c) }
    @Override void onOffline(Computer c, OfflineCause cause)            { sync(c) }
    @Override void onTemporarilyOnline(Computer c)                      { sync(c) }
    @Override void onTemporarilyOffline(Computer c, OfflineCause cause) { sync(c) }
})

logger.info("lockable-resource-sync listener registered")

Three things worth knowing about what it does:

It creates the resource if one does not exist. The first time a node comes online, manager.fromName(node.name) returns null. The script creates a lockable resource named after the node, and from then on the lock step can find it.

It mirrors the node’s labels onto the resource. This is why agent { label env.resource_name } works at all: the resource is named build-agent-03, the node is named build-agent-03, and the label string includes that name plus the pool labels (TC3.1.4024 and so on) pulled straight from the Jenkins node config. Moving a machine between pools is just a label change in Jenkins, nothing to update separately.

It marks offline nodes so nobody can lock them. A node going offline becomes node-name OFFLINE, which does not match any pool label, so it silently drops out of rotation until it comes back.

The script runs once at startup to catch anything that changed while Jenkins was restarting, then stays alive as a listener for as long as Jenkins runs.

How other platforms handle it

Jenkins needs a plugin, a pipeline pattern, and an init script to get here. Other platforms bake it in more directly.

GitLab CI has resource_group: a project-level mutex. Give the same resource_group name to multiple jobs and GitLab serialises them automatically; the rest queue. It has no idea what a machine is, so you pair it with a runner tag that pins the job to a specific self-hosted runner. Tag picks the machine, resource group serialises access to it.

GitHub Actions has concurrency at the workflow or job level: a repo-wide mutex with an option to cancel the queued run instead of waiting. Pair it with a runs-on label mapped to a specific self-hosted runner and you get the same combination: label picks the machine, concurrency group serialises it.

Both are simpler than what Jenkins needs. That is not an argument for switching CI systems, the Jenkins pattern works fine, it just needs more pieces to get there.

Three parts, no shortcuts, and just one line item

Lock claims one specific machine before any work starts. Pin forces the build onto that exact machine, not just any free slot. Sync keeps the lock registry honest as the fleet changes, so nobody has to remember to update it by hand. Skip any one of the three and you are back to intermittent collisions that vanish on retry and reappear a week later on a different machine.

None of that is exotic, and none of it is the whole job either. Locking buys you exclusive machine access. It says nothing about getting the right tools onto every node, triggering pipelines correctly across release branches, or resolving library dependencies across release/1.x and release/2.x. Each of those is its own problem, with its own setup and its own upkeep.

Teams that end up with this whole list rarely planned for it. They planned for “set up CI/CD” as a project with an end date. The farm is the part that keeps running long after that.

What Zeugwerk CI/CD does instead

Zeugwerk CI/CD, the managed option from the landscape post, exists mainly to take that whole list off your desk. The build nodes are TwinCAT-capable, dedicated to your team (a fleet of them if that is what you need), and wiped clean the moment a build finishes, so exclusivity is just how it works rather than something you had to engineer. Dependency resolution, artifact generation, and test reporting come along with it, and your code never has to leave GitHub, GitLab, or Bitbucket, a lightweight proxy handles the rest. Setup is closer to an afternoon than a farm-engineering project.

Zeugwerk are providing a valuable and unique service that contributes to modernising the PLC software development experience. The build service is well integrated for GitHub and has been reliable, it just works. Their team has provided helpful onboarding and responsive ongoing support.

— Brianna Laugher · Principal Software Engineer · Celleo

None of this makes self-hosted the wrong call for everyone. Strict air-gapped policies, an existing Jenkins investment, or regulatory requirements that rule out external infrastructure are all real reasons to keep running your own machines. Where none of those apply, it mostly comes down to whether operating Windows build servers is something your team wants to get good at. And if you do stay self-hosted, you are not entirely on your own either: our DevTools CLIs, the ones this whole locking pattern is built to run, ship to your nodes the same way described in the Scoop post, so at least that part updates itself instead of becoming one more thing to maintain by hand.

When self-hosted still makes sense

Run your own farm if every byte has to stay on your network, if a mature DevOps team already maintains Windows infrastructure for you, or if CI needs to reach on-prem systems a cloud build simply cannot touch.

Otherwise, count the machines before you count the pipeline lines. “We’ll start with one and add more later” is a fine plan, as long as “more” is budgeted honestly: not just hardware, but locking, tooling, sync scripts, and someone who keeps the whole pool healthy.

Running a self-hosted TwinCAT build farm, or wondering whether managed CI would fit? Get in touch.