This post describes how to define and use rule templates with semantic names using
!reference tags, how
to define manual jobs using the same templates and how to use gitlab-ci
inputs as macros to give names to regular expressions used by rules.
Basic rule templates
I keep my templates in a
rules.yml file stored on a common repository used from different projects as I mentioned on
my previous post, but they can be defined anywhere, the important thing is that the
files that need them include their definition somehow.
The first version of my
rules.yml file was as follows:
.rules_common: # Common rules; we include them from others instead of forcing a workflow rules: # Disable branch pipelines while there is an open merge request from it - if: >- $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE != "merge_request_event" when: never .rules_default: # Default rules, we need to add the when: on_success to make things work rules: - !reference [.rules_common, rules] - when: on_success
The main idea is that
.rules_common defines a
rule section to disable jobs as we can do on a
in our case common rules only have
if rules that apply to all jobs and are used to disable them. The example includes
one that avoids creating duplicated jobs when we push to a branch that is the source of an open MR as explained
To use the rules in a job we have two options, use the
extends keyword (we do that when we want to use the rule as is)
or declare a
rules section and add a
!reference to the template we want to use as described
here (we do that when we want to add
additional rules to disable a job before evaluating the template conditions).
As an example, with the following definitions both jobs use the same rules:
job_1: extends: - .rules_default [...] job_2: rules: - !reference [.rules_default, rules] [...]
Manual jobs and rule templates
To make the jobs manual we have two options, create a version of the job that includes
when: manual and defines if we
want it to be optional or not (
allow_failure: true makes the job optional, if we don’t add that to the rule the job is
blocking) or add the
when: manual and the
allow_failure value to the job (if we work at the job level the default
when: manual, so it is optional by default, we have to add an explicit
allow_failure = true it to make it blocking).
The following example shows how we define blocking or optional manual jobs using rules with
.rules_default_manual_blocking: # Default rules for optional manual jobs rules: - !reference [.rules_common, rules] - when: manual # allow_failure: false is implicit .rules_default_manual_optional: # Default rules for optional manual jobs rules: - !reference [.rules_common, rules] - when: manual allow_failure: true manual_blocking_job: extends: - .rules_default_manual_blocking [...] manual_optional_job: extends: - .rules_default_manual_optional [...]
The problem here is that we have to create new versions of the same rule template to add the conditions, but we can avoid it using the keywords at the job level with the original rules to get the same effect; the following definitions create jobs equivalent to the ones defined earlier without creating additional templates:
manual_blocking_job: extends: - .rules_default when: manual allow_failure: false [...] manual_optional_job: extends: - .rules_default when: manual # allow_failure: true is implicit [...]
As you can imagine, that is my preferred way of doing it, as it keeps the
rules.yml file smaller and I see that the
job is manual in its definition without problem.
Unluckily for us, for now there is no way to avoid creating additional templates as we did on the
when: manual case
when a rule is similar to an existing one but adds changes,
variables to it.
So, for now, if a rule needs to add any of those fields we have to copy the original rule and add the keyword section.
Some notes, though:
- we only need to add
allow_failureif we want to change its value for a given condition, in other cases we can set the value at the job level.
- if we are adding
changesto the rule it is important to make sure that they are going to be evaluated as explained here.
- when we add a
needsvalue to a rule for a specific condition and it matches it replaces the job
needssection; when using templates I would use two different job names with different conditions instead of adding a
needson a single job.
Defining rule templates with semantic names
I started to use rule templates to avoid repetition when defining jobs that needed the same rules and soon I noticed that giving them names with a semantic meaning they where easier to use and understand (we provide a name that tells us when we are going to execute the job, while the details of the variables names or values used on the rules are an implementation detail of the templates).
We are not going to define real jobs on this post, but as an example we are going to define a set of rules that can be
useful if we plan to follow a scaled trunk based
development workflow when developing, that is, we are going to put the releasable code on the
main branch and use
short-lived branches to test and complete changes before pushing things to
Using this approach we can define an initial set of rule templates with semantic names:
.rules_mr_to_main: rules: - !reference [.rules_common, rules] - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main' .rules_mr_or_push_to_main: rules: - !reference [.rules_common, rules] - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main' - if: >- $CI_COMMIT_BRANCH == 'main' && $CI_PIPELINE_SOURCE != 'merge_request_event' .rules_push_to_main: rules: - !reference [.rules_common, rules] - if: >- $CI_COMMIT_BRANCH == 'main' && $CI_PIPELINE_SOURCE != 'merge_request_event' .rules_push_to_branch: rules: - !reference [.rules_common, rules] - if: >- $CI_COMMIT_BRANCH != 'main' && $CI_PIPELINE_SOURCE != 'merge_request_event' .rules_push_to_branch_or_mr_to_main: rules: - !reference [.rules_push_to_branch, rules] - if: >- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME != 'main' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main' .rules_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG =~ /^([0-9a-zA-Z_.-]+-)?v\d+.\d+.\d+$/ .rules_non_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG !~ /^([0-9a-zA-Z_.-]+-)?v\d+.\d+.\d+$/
With those names it is clear when a job is going to be executed and when using the templates on real jobs we can add additional restrictions and make the execution manual if needed as described earlier.
inputs as macros
On the previous rules we have used a regular expression to identify the release tag format and assumed that the
general branches are the ones with a name different than
main; if we want to force a format for those branch names
we can replace the condition
!= 'main' by a regex comparison (
=~ if we look for matches,
!~ if we want to define
valid branch names removing the invalid ones).
When testing the new gitlab-ci inputs my colleague Jorge noticed that if you keep their default value they basically work as macros.
The variables declared as
inputs can’t hold YAML values, the truth is that their value is always a string that is
replaced by the value assigned to them when including the file (if given) or by their default value, if defined.
If you don’t assign a value to an input variable when including the file that declares it its occurrences are replaced
by its default value, making them work basically as macros; this is useful for us when working with strings that can’t
managed as variables, like the regular expressions used inside
With those two ideas we can add the following prefix to the
rules.yaml defining inputs for both regular expressions
and replace the rules that can use them by the ones shown here:
spec: inputs: # Regular expression for branches; the prefix matches the type of changes # we plan to work on inside the branch (we use conventional commit types as # the branch prefix) branch_regex: default: '/^(build|ci|chore|docs|feat|fix|perf|refactor|style|test)\/.+$/' # Regular expression for tags release_tag_regex: default: '/^([0-9a-zA-Z_.-]+-)?v\d+.\d+.\d+$/' --- [...] .rules_push_to_changes_branch: rules: - !reference [.rules_common, rules] - if: >- $CI_COMMIT_BRANCH =~ $[[ inputs.branch_regex ]] && $CI_PIPELINE_SOURCE != 'merge_request_event' .rules_push_to_branch_or_mr_to_main: rules: - !reference [.rules_push_to_branch, rules] - if: >- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ $[[ inputs.branch_regex ]] && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main' .rules_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG =~ $[[ inputs.release_tag_regex ]] .rules_non_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG !~ $[[ inputs.release_tag_regex ]]
Creating rules reusing existing ones
I’m going to finish this post with a comment about how I avoid defining extra rule templates in some common cases.
The idea is simple, we can use
!reference tags to fine tune rules when we need to add conditions to disable them
simply adding conditions with
when: never before referencing the template.
As an example, in some projects I’m using different job definitions depending on the
DEPLOY_ENVIRONMENT value to make
the job manual or automatic; as we just said we can define different jobs referencing the same rule adding a condition
to check if the environment is the one we are interested in:
deploy_job_auto: rules: # Only deploy automatically if the environment is 'dev' by skipping this job # for other values of the DEPLOY_ENVIRONMENT variable - if: $DEPLOY_ENVIRONMENT != "dev" when: never - !reference [.rules_release_tag, rules] [...] deploy_job_manually: rules: # Disable this job if the environment is 'dev' - if: $DEPLOY_ENVIRONMENT == "dev" when: never - !reference [.rules_release_tag, rules] when: manual # Change this to `false` to make the deployment job blocking allow_failure: true [...]
If you think about it the idea of adding negative conditions is what we do with the
.rules_common template; we add
conditions to disable the job before evaluating the real rules.
The difference in that case is that we reference them at the beginning because we want those negative conditions on all
jobs and that is also why we have a
.rules_default condition with an
when: on_success for the jobs that only need to
respect the default workflow (we need the last condition to make sure that they are executed if the negative rules don’t