Autoscaling

Add burst runners to a GitLab runner on Hetzner Cloud while CI jobs queue up, and drain them when the load falls, near the end of a paid hour.

Autoscaling adds GitLab runners on Hetzner Cloud while one of your runners has few free job slots, and removes them again when the load fits on fewer. The runner you set it up on is the anchor. The runners it adds are burst runners: Hetzner servers registered with the anchor’s GitLab runner authentication token, so GitLab sees them as additional machines of the same runner and gives them the same jobs, with the same tags.

The anchor stays as it is and always runs; autoscaling never scales to zero. It can be a runner created with Manage Runners or an external runner, for example your own dedicated server.

Requirements

• GitLab runners only. GitHub Actions runners cannot be anchors.
• A Pro or Enterprise plan for the organization, like schedules. See Billing and plans. Without one, the dashboard shows nothing about autoscaling.
• Metrics from the anchor. Decisions use the anchor’s Monitoring samples, so turn on Show in dashboard for it. Burst runners always have Monitoring.
• External anchors need the agent version 1.1.0 or later, which reports GitLab Runner’s job limit (its concurrent setting), and GitLab Runner’s metrics endpoint, listen_address = "127.0.0.1:9252" in /etc/gitlab-runner/config.toml; see Running jobs on GitLab runners. Without a job limit, free job slots cannot be counted and the dashboard says so.

Set it up in the dashboard

GitLab runner cards show an Autoscaling line next to the schedule. Owners and editors select Add autoscaling, or the policy or the pencil next to it, to open the runner’s autoscaling page, and choose:

• When to scale: Eager, Balanced or Conservative, or Custom rules. Below the choice, the page explains what it means for the job slots the runner has now, for example With 4 job slots now: a burst runner starts when all 4 are busy for 3 minutes.
• Maximum burst runners: 1 to 20. Burst runners are counted until they are deleted, including those still starting or draining.
• Runners added per step: 1 to 5, default 1.
• Remove after low load for (minutes): how long the load must stay low before one burst runner is removed, default 10.
• Wait after ready (minutes): how long to wait after a burst runner became ready before adding more, default 3.
• Enabled: for a new policy, whether it starts scaling once saved. An existing policy has Enable and Disable at the top of the page, and a switch on the card. Disabling drains and deletes every burst runner but keeps the settings.

The ⓘ next to each limit shows a short explanation on hover. Select it to see that explanation together with a longer one.

For an external runner you also describe its burst runners, since Manage Runners does not know how your server is set up: server type and location, your Hetzner API token, the GitLab instance and the runner’s authentication token (glrt-…), executor, concurrency, the Hetzner SSH keys to add (listed by name once the token is entered, or with the stored token when you edit the policy), labels, and optionally a cache and a Metrics connection. The Hetzner token is checked with Hetzner and the runner token with GitLab before the policy is saved, so your GitLab instance must be reachable over HTTPS at a public address. The tokens are stored encrypted and never shown again; leave the fields empty later to keep them. Burst runners of a Manage Runners runner copy that runner when they are created: server type, location, Hetzner project, GitLab settings, concurrency, SSH keys, labels, cache and Metrics connection.

The card shows the policy, for example Autoscaling · Balanced · up to 3 burst runners, with the group’s job slots, for example 3 of 8 job slots busy · 5 free, the number of burst runners and the last decision. The autoscaling page also lists the newest 50 events and has Delete policy. Events are kept for 90 days; the API and list_scaling_events return up to 200 of them.

Free job slots

A job slot is room for one CI job. A runner’s job slots are its concurrency; an external runner’s job slots are the job limit its agent reports. The group is the anchor and its burst runners that are ready, and its free job slots are its job slots minus its running jobs.

The free job slots rule has a number K and a duration:

• Scale out when at most K job slots are free for the whole duration, for example when all of them are busy (K = 0).
• Scale in when, without one burst runner, more than K job slots would still be free for the whole Remove after low load time. The jobs of that runner count as moved to the others. A burst runner is also removed when the group ran no job at all for the whole Remove after low load time, so a rule whose K is as large as the group’s job slots still scales back.

This keeps the number of burst runners from swinging back and forth. A burst runner is added only while at most K job slots are free and the group is not idle. It is removed in one of two cases, and neither can add it straight back:

• More than K job slots would still be free without it, so the group stays above the point where runners are added.
• The group ran no job at all for the whole Remove after low load time. While the group stays idle, nothing is added.

Presets

The presets are one free job slots rule:

The page offers only presets that make sense for the anchor: a runner with a single job slot never has more than one free, so Eager would add a burst runner as soon as any job is seen, and it is not offered there. Presets that would behave the same for the anchor are offered once. While an external runner has not reported its job limit, all presets are offered. A policy that already uses a preset keeps showing it; with Eager on a runner with one job slot, its burst runners are removed again once the group has run no job for the Remove after low load time.

Examples

A runner with 1 job slot, Balanced: while its job runs for 3 minutes, a burst runner with 1 slot is added. With both busy for another 3 minutes, a second is added, up to the maximum. When the anchor and one burst runner are idle, removing that burst runner still leaves at least one free slot for 10 minutes, so it is removed in the last 15 minutes of its paid hour.

A runner with 4 job slots, Balanced: a burst runner is added when all 4 are busy for 3 minutes. With 8 slots and 3 jobs, removing a 4-slot burst runner would leave 1 free slot, more than 0, so it is removed once that has been true for 10 minutes. With Eager, a burst runner is added already when 3 of the 4 are busy for 2 minutes, and removed only when, without it, at least 2 slots would stay free.

Custom rules

A policy has one to three rules, one per metric:

Each rule also has a duration of 2 to 60 minutes. In the editor, a free job slots rule reads Scale out when at most [K] job slots are free for [M] minutes.

How decisions are made

Manage Runners checks every policy once a minute. The group is the anchor plus its burst runners that are ready and report metrics; burst runners that are still starting count only to wait for them.

• Scale out when any rule holds in every minute of its duration: at most K job slots free, or CPU or memory at or above the scale-out threshold. Each report counts for one minute only, and a minute without a report does not count, so a short spike does not count. Nothing is added while the group is idle, whatever the rules say: when the newest report of the anchor and of every ready burst runner says that no job is running. If a runner reports no job count, for example an external runner without GitLab Runner’s metrics endpoint, the group is not known to be idle, so a CPU or memory rule can still add burst runners. Nothing is added either while a burst runner is starting, within Wait after ready after the newest became ready, at the maximum, during a wait after a failure, or without an active plan.
• Scale in when every rule allows it without one burst runner for the whole Remove after low load time: more than K job slots would stay free, counting that runner’s jobs on the others, or no job ran in the group at all, and CPU and memory would stay below their scale-in thresholds, with the average of the others taken as the average × N ÷ (N − 1). One burst runner is removed at a time, and the load is checked again before the next one. A burst runner is only removed when its own load is known for the whole Remove after low load time; otherwise the card says that data is missing.
• No decisions without metrics: when the anchor has not reported for about three minutes, nothing is added or removed, except that burst runners are removed when the policy is disabled or the anchor is paused.

Billing-aware scale-in

Hetzner bills a server for every hour it has started. Removing a burst runner just after a new hour began would waste most of that hour, so a burst runner is only removed in the last 15 minutes of its current paid hour, counted from the time Hetzner created its server. If several qualify, the one closest to the end of its hour goes first; if that is a tie, the one with fewer running jobs. Until one is in its last 15 minutes, the card shows that the load is low and autoscaling is waiting.

A removed burst runner first takes no new jobs and finishes the running ones, like Pause after running jobs, for at most one hour; see Pause and resume. Then it is deleted completely, with its server, its IP addresses and its entry in the dashboard.

Burst runners in the dashboard

Burst runners appear after their anchor with a Burst badge and show how long they have been running, counted from the time Hetzner created their server, for example Running for 1 h 12 min · since 14:02, or Starting for 3 min while they start, and Draining · in front while they are removed. Times are shown in your browser’s local time format. The autoscaling page lists the same for each burst runner, oldest first. They can only be deleted: editing, pausing, resuming, duplicating and schedules are not available, also not through the CLI, the API or AI assistants. If you delete one by hand, autoscaling adds another one when the group is busy again. Their servers are in your Hetzner project and billed by Hetzner like any other runner, with the anchor’s Hetzner project for Manage Runners runners and the template’s for external ones.

GitLab runner managers

GitLab shows every machine registered with a runner token as a runner manager of that runner, with its own system ID. When a burst runner’s server is deleted, Manage Runners removes its runner manager through GitLab’s API, so the runner’s page in GitLab does not fill up with machines that are gone. Only that machine is removed; the runner, the anchor and the other burst runners stay registered. The same happens when any Manage Runners GitLab runner is paused or deleted, since a resumed runner registers as a new machine.

Manage Runners contacts your GitLab instance for this only over HTTPS and only at public addresses, and tries again for about 20 seconds if GitLab is unavailable. A self-managed GitLab instance that is reachable only from a private network, or a removal that still fails, keeps the old managers until GitLab removes them itself after a week without contact.

When autoscaling stops

• Disabling the policy drains and deletes every burst runner. Enabling it again also forgets earlier failures.
• Deleting the policy, or deleting the anchor, does the same.
• Pausing a Manage Runners anchor, for example by its schedule, drains and deletes every burst runner, and nothing is added until it is running again. A burst runner whose creation started just as the anchor began pausing is marked for removal at a later check and then leaves like any other: once it has finished starting, it drains and is deleted.
• Failures: when a burst runner cannot be created, or is not ready within 15 minutes, is unknown or fails to register, it is deleted and counts as a failure. After the first failure Manage Runners waits 5 minutes before adding runners again, after the second 10 minutes; the third failure in a row disables the policy, drains its burst runners, shows the reason on the card and emails the organization’s owners and editors once.
• Unknown burst runners, and burst runners whose creation was interrupted, are never just removed from the dashboard, since their server could keep running. Every server Manage Runners creates carries the Hetzner label managerunners-runner with the runner’s id, and so do its primary IP addresses. Manage Runners finds the server by its id or that label and deletes it with its IP addresses. Only the burst runner’s own Hetzner token can show that the server is gone; if Hetzner refuses it, the anchor’s or template’s token is used only when it finds the labelled server itself. If the cleanup still fails, the card shows that it is blocked and why, the runner keeps counting against the maximum and Manage Runners tries again every minute. Restore the token’s access, or delete the server and its IP addresses in your Hetzner project and then the runner in the dashboard.
• Without an active plan no burst runners are added. Existing ones start draining in the last 15 minutes of their paid hour and get up to one hour to finish running jobs before they are deleted.

Limits

• One policy per runner, up to 20 burst runners and up to three rules.
• Burst runners only on Hetzner Cloud, in the anchor’s location (Manage Runners runners) or the template’s (external runners).
• Decisions use the metrics reported once a minute, so autoscaling reacts within a few minutes, not seconds. It does not look at jobs waiting in GitLab’s queue.

With an AI assistant

The remote MCP server has get_scaling_policy, list_scaling_events, set_scaling_policy, enable_scaling_policy, disable_scaling_policy and delete_scaling_policy. They never take tokens: for an external runner without stored tokens, set_scaling_policy returns a dashboard link to a page that shows the requested policy and its organization, where you enter the tokens yourself and save exactly that policy. list_runners and get_runner show which runner a burst runner belongs to.