This post describes how I’m using
semantic-release on
gitlab-ci to manage versioning automatically
for different kinds of projects following a simple workflow (a develop
branch
where changes are added or merged to test new versions, a temporary
release/#.#.#
to generate the release candidate versions and a main
branch
where the final versions are published).
What is semantic-release
It is a Node.js application designed to manage project
versioning information on Git Repositories using a
Continuous integration
system (in this post we will use gitlab-ci
)
How does it work
By default semantic-release
uses semver for versioning
(release versions use the format MAJOR.MINOR.PATCH
) and commit messages are
parsed to determine the next version number to publish.
If after analyzing the commits the version number has to be changed, the command
updates the files we tell it to (i.e. the package.json
file for nodejs
projects and possibly a CHANGELOG.md
file), creates a new commit with the
changed files, creates a tag with the new version and pushes the changes to the
repository.
When running on a CI/CD system we usually generate the artifacts related to a release (a package, a container image, etc.) from the tag, as it includes the right version number and usually has passed all the required tests (it is a good idea to run the tests again in any case, as someone could create a tag manually or we could run extra jobs when building the final assets … if they fail it is not a big issue anyway, numbers are cheap and infinite, so we can skip releases if needed).
Commit messages and versioning
The commit messages must follow a known format, the default module used to
analyze them uses the
angular
git commit guidelines, but I prefer the
conventional commits one, mainly because
it’s a lot easier to use when you want to update the MAJOR
version.
The commit message format used must be:
<type>(optional scope): <description>
[optional body]
[optional footer(s)]
The system supports three types of branches: release
, maintenance
and
pre-release
, but for now I’m not using maintenance
ones.
The branches I use and their types are:
main
as release branch (final versions are published from there)develop
as pre release branch (used to publish development and testing versions with the format#.#.#-SNAPSHOT.#
)release/#.#.#
as pre release branches (they are created fromdevelop
to publish release candidate versions with the format#.#.#-rc.#
and once they are merged withmain
they are deleted)
On the release branch (main
) the version number is updated as follows:
- The
MAJOR
number is incremented if a commit with aBREAKING CHANGE:
footer or an exclamation (!
) after the type/scope is found in the list of commits found since the last version change (it looks for tags on the same branch). - The
MINOR
number is incremented if the MAJOR number is not going to be changed and there is a commit with typefeat
in the commits found since the last version change. - The
PATCH
number is incremented if neither the MAJOR nor the MINOR numbers are going to be changed and there is a commit with typefix
in the the commits found since the last version change.
On the pre release branches (develop
and release/#.#.#
) the version and
pre release numbers are always calculated from the last published version
available on the branch (i. e. if we published version 1.3.2
on main
we need
to have the commit with that tag on the develop
or release/#.#.#
branch
to get right what will be the next version).
The version number is updated as follows:
The
MAJOR
number is incremented if a commit with aBREAKING CHANGE:
footer or an exclamation (!
) after the type/scope is found in the list of commits found since the last released version.In our example it was
1.3.2
and the version is updated to2.0.0-SNAPSHOT.1
or2.0.0-rc.1
depending on the branch.The
MINOR
number is incremented if the MAJOR number is not going to be changed and there is a commit with typefeat
in the commits found since the last released version.In our example the release was
1.3.2
and the version is updated to1.4.0-SNAPSHOT.1
or1.4.0-rc.1
depending on the branch.The
PATCH
number is incremented if neither the MAJOR nor the MINOR numbers are going to be changed and there is a commit with typefix
in the the commits found since the last version change.In our example the release was
1.3.2
and the version is updated to1.3.3-SNAPSHOT.1
or1.3.3-rc.1
depending on the branch.- The pre release number is incremented if the
MAJOR
,MINOR
andPATCH
numbers are not going to be changed but there is a commit that would otherwise update the version (i.e. afix
on1.3.3-SNAPSHOT.1
will set the version to1.3.3-SNAPSHOT.2
, afix
orfeat
on1.4.0-rc.1
will set the version to1.4.0-rc.2
an so on).
How do we manage its configuration
Although the system is designed to work with nodejs
projects, it can be used
with multiple programming languages and project types.
For nodejs
projects the usual place to put the configuration is the project’s
package.json
, but I prefer to use the .releaserc
file instead.
As I use a common set of CI templates, instead of using a .releaserc
on each
project I generate it on the fly on the jobs that need it, replacing values
related to the project type and the current branch on a template using the
tmpl command (lately I use a
branch of my own fork while I wait
for some feedback from upstream, as you will see on the Dockerfile
).
Container used to run it
As we run the command on a gitlab-ci
job we use the image built from the
following Dockerfile
:
How and when is it executed
The job that runs semantic-release
is executed when new commits are added
to the develop
, release/#.#.#
or main
branches (basically when something
is merged or pushed) and after all tests have passed (we don’t want to create a
new version that does not compile or passes at least the unit tests).
The job is something like the following:
semantic_release:
image: $SEMANTIC_RELEASE_IMAGE
rules:
- if: '$CI_COMMIT_BRANCH =~ /^(develop|main|release\/\d+.\d+.\d+)$/'
when: always
stage: release
before_script:
- echo "Loading scripts.sh"
- . $ASSETS_DIR/scripts.sh
script:
- sr_gen_releaserc_json
- git_push_setup
- semantic-release
Where the SEMANTIC_RELEASE_IMAGE
variable contains the URI of the image built
using the Dockerfile
above and the sr_gen_releaserc_json
and
git_push_setup
are functions defined on the $ASSETS_DIR/scripts.sh
file:
- The
sr_gen_releaserc_json
function generates the.releaserc.json
file using thetmpl
command. - The
git_push_setup
function configuresgit
to allow pushing changes to the repository with thesemantic-release
command, optionally signing them with a SSH key.
The sr_gen_releaserc_json
function
The code for the sr_gen_releaserc_json
function is the following:
sr_gen_releaserc_json()
{
# Use nodejs as default project_type
project_type="${PROJECT_TYPE:-nodejs}"
# REGEX to match the rc_branch name
rc_branch_regex='^release\/[0-9]\+\.[0-9]\+\.[0-9]\+$'
# PATHS on the local ASSETS_DIR
assets_dir="${CI_PROJECT_DIR}/${ASSETS_DIR}"
sr_local_plugin="${assets_dir}/local-plugin.cjs"
releaserc_tmpl="${assets_dir}/releaserc.json.tmpl"
pipeline_runtime_values_yaml="/tmp/releaserc_values.yaml"
pipeline_values_yaml="${assets_dir}/values_${project_type}_project.yaml"
# Destination PATH
releaserc_json=".releaserc.json"
# Create an empty pipeline_values_yaml if missing
test -f "$pipeline_values_yaml" || : >"$pipeline_values_yaml"
# Create the pipeline_runtime_values_yaml file
echo "branch: ${CI_COMMIT_BRANCH}" >"$pipeline_runtime_values_yaml"
echo "gitlab_url: ${CI_SERVER_URL}" >"$pipeline_runtime_values_yaml"
# Add the rc_branch name if we are on an rc_branch
if [ "$(echo "$CI_COMMIT_BRANCH" | sed -ne "/$rc_branch_regex/{p}")" ]; then
echo "rc_branch: ${CI_COMMIT_BRANCH}" >>"$pipeline_runtime_values_yaml"
elif [ "$(echo "$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME" |
sed -ne "/$rc_branch_regex/{p}")" ]; then
echo "rc_branch: ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}" \
>>"$pipeline_runtime_values_yaml"
fi
echo "sr_local_plugin: ${sr_local_plugin}" >>"$pipeline_runtime_values_yaml"
# Create the releaserc_json file
tmpl -f "$pipeline_runtime_values_yaml" -f "$pipeline_values_yaml" \
"$releaserc_tmpl" | jq . >"$releaserc_json"
# Remove the pipeline_runtime_values_yaml file
rm -f "$pipeline_runtime_values_yaml"
# Print the releaserc_json file
print_file_collapsed "$releaserc_json"
# --*-- BEG: NOTE --*--
# Rename the package.json to ignore it when calling semantic release.
# The idea is that the local-plugin renames it back on the first step of the
# semantic-release process.
# --*-- END: NOTE --*--
if [ -f "package.json" ]; then
echo "Renaming 'package.json' to 'package.json_disabled'"
mv "package.json" "package.json_disabled"
fi
}
Almost all the variables used on the function are defined by gitlab
except the
ASSETS_DIR
and PROJECT_TYPE
; in the complete pipelines the ASSETS_DIR
is
defined on a common file included by all the pipelines and the project type is
defined on the .gitlab-ci.yml
file of each project.
If you review the code you will see that the file processed by the tmpl
command is named releaserc.json.tmpl
, its contents are shown here:
{
"plugins": [
{{- if .sr_local_plugin }}
"{{ .sr_local_plugin }}",
{{- end }}
[
"@semantic-release/commit-analyzer",
{
"preset": "conventionalcommits",
"releaseRules": [
{ "breaking": true, "release": "major" },
{ "revert": true, "release": "patch" },
{ "type": "feat", "release": "minor" },
{ "type": "fix", "release": "patch" },
{ "type": "perf", "release": "patch" }
]
}
],
{{- if .replacements }}
[
"semantic-release-replace-plugin",
{ "replacements": {{ .replacements | toJson }} }
],
{{- end }}
"@semantic-release/release-notes-generator",
{{- if eq .branch "main" }}
[
"@semantic-release/changelog",
{ "changelogFile": "CHANGELOG.md", "changelogTitle": "# Changelog" }
],
{{- end }}
[
"@semantic-release/git",
{
"assets": {{ if .assets }}{{ .assets | toJson }}{{ else }}[]{{ end }},
"message": "ci(release): v${nextRelease.version}\n\n${nextRelease.notes}"
}
],
[
"@semantic-release/gitlab",
{ "gitlabUrl": "{{ .gitlab_url }}", "successComment": false }
]
],
"branches": [
{ "name": "develop", "prerelease": "SNAPSHOT" },
{{- if .rc_branch }}
{ "name": "{{ .rc_branch }}", "prerelease": "rc" },
{{- end }}
"main"
]
}
The values used to process the template are defined on a file built on the fly
(releaserc_values.yaml
) that includes the following keys and values:
branch
: the name of the current branchgitlab_url
: the URL of the gitlab server (the value is taken from theCI_SERVER_URL
variable)rc_branch
: the name of the current rc branch; we only set the value if we are processing one becausesemantic-release
only allows one branch to match therc
prefix and if we use a wildcard (i.e.release/*
) but the users keep more than onerelease/#.#.#
branch open at the same time the calls tosemantic-release
will fail for sure.sr_local_plugin
: the path to the local plugin we use (shown later)
The template also uses a values_${project_type}_project.yaml
file that
includes settings specific to the project type, the one for nodejs
is as
follows:
replacements:
- files:
- "package.json"
from: "\"version\": \".*\""
to: "\"version\": \"${nextRelease.version}\""
assets:
- "CHANGELOG.md"
- "package.json"
The replacements
section is used to update the version
field on the relevant
files of the project (in our case the package.json
file) and the assets
section includes the files that will be committed to the repository when the
release is published (looking at the template you can see that the
CHANGELOG.md
is only updated for the main
branch, we do it this way because
if we update the file on other branches it creates a merge nightmare and we are
only interested on it for released versions anyway).
The local plugin adds code to rename the package.json_disabled
file to
package.json
if present and prints the last and next versions on the logs with
a format that can be easily parsed using sed
:
The git_push_setup
function
The code for the git_push_setup
function is the following:
git_push_setup()
{
# Update global credentials to allow git clone & push for all the group repos
git config --global credential.helper store
cat >"$HOME/.git-credentials" <<EOF
https://fake-user:${GITLAB_REPOSITORY_TOKEN}@gitlab.com
EOF
# Define user name, mail and signing key for semantic-release
user_name="$SR_USER_NAME"
user_email="$SR_USER_EMAIL"
ssh_signing_key="$SSH_SIGNING_KEY"
# Export git user variables
export GIT_AUTHOR_NAME="$user_name"
export GIT_AUTHOR_EMAIL="$user_email"
export GIT_COMMITTER_NAME="$user_name"
export GIT_COMMITTER_EMAIL="$user_email"
# Sign commits with ssh if there is a SSH_SIGNING_KEY variable
if [ "$ssh_signing_key" ]; then
echo "Configuring GIT to sign commits with SSH"
ssh_keyfile="/tmp/.ssh-id"
: >"$ssh_keyfile"
chmod 0400 "$ssh_keyfile"
echo "$ssh_signing_key" | tr -d '\r' >"$ssh_keyfile"
git config gpg.format ssh
git config user.signingkey "$ssh_keyfile"
git config commit.gpgsign true
fi
}
The function assumes that the GITLAB_REPOSITORY_TOKEN
variable (set on the
CI/CD variables section of the project or group we want) contains a token with
read_repository
and write_repository
permissions on all the projects we are
going to use this function.
The SR_USER_NAME
and SR_USER_EMAIL
variables can be defined on a common file
or the CI/CD variables section of the project or group we want to work with and
the script assumes that the optional SSH_SIGNING_KEY
is exported as a CI/CD
default value of type variable (that is why the keyfile is created on the fly)
and git
is configured to use it if the variable is not empty.
Warning:
Keep in mind that the variables GITLAB_REPOSITORY_TOKEN
and SSH_SIGNING_KEY
contain secrets, so probably is a good idea to make them protected
(if you do
that you have to make the develop
, main
and release/*
branches protected
too).
Warning:
The semantic-release
user has to be able to push to all the projects on those
protected branches, it is a good idea to create a dedicated user and add it as a
MAINTAINER
for the projects we want (the MAINTAINERS
need to be able to push
to the branches), or, if you are using a Gitlab with a Premium license you can
use the
api
to allow the semantic-release
user to push to the protected branches without
allowing it for any other user.
The semantic-release
command
Once we have the .releaserc
file and the git
configuration ready we run the
semantic-release
command.
If the branch we are working with has one or more commits that will increment the version, the tool does the following (note that the steps are described are the ones executed if we use the configuration we have generated):
- It detects the commits that will increment the version and calculates the next version number.
- Generates the release notes for the version.
- Applies the replacements defined on the configuration (in our example updates
the
version
field on thepackage.json
file). - Updates the
CHANGELOG.md
file adding the release notes if we are going to publish the file (when we are on themain
branch). - Creates a commit if all or some of the files listed on the
assets
key have changed and uses the commit message we have defined, replacing the variables for their current values. - Creates a tag with the new version number and the release notes.
- As we are using the
gitlab
plugin after tagging it also creates a release on the project with the tag name and the release notes.
Notes about the git
workflows and merges between branches
It is very important to remember that semantic-release
looks at the commits of
a given branch when calculating the next version to publish, that has two
important implications:
- On pre release branches we need to have the commit that includes the tag with the released version, if we don’t have it the next version is not calculated correctly.
- It is a bad idea to squash commits when merging a branch to another one, if
we do that we will lose the information
semantic-release
needs to calculate the next version and even if we use the right prefix for the squashed commit (fix
,feat
, …) we miss all the messages that would otherwise go to theCHANGELOG.md
file.
To make sure that we have the right commits on the pre release branches we
should merge the main
branch changes into the develop
one after each release
tag is created; in my pipelines the fist job that processes a release tag
creates a branch from the tag and an MR to merge it to develop
.
The important thing about that MR is that is must not be squashed, if we do that the tag commit will probably be lost, so we need to be careful.
To merge the changes directly we can run the following code:
# Set the SR_TAG variable to the tag you want to process
SR_TAG="v1.3.2"
# Fetch all the changes
git fetch --all --prune
# Switch to the main branch
git switch main
# Pull all the changes
git pull
# Switch to the development branch
git switch develop
# Pull all the changes
git pull
# Create followup branch from tag
git switch -c "followup/$SR_TAG" "$SR_TAG"
# Change files manually & commit the changed files
git commit -a --untracked-files=no -m "ci(followup): $SR_TAG to develop"
# Switch to the development branch
git switch develop
# Merge the followup branch into the development one using the --no-ff option
git merge --no-ff "followup/$SR_TAG"
# Remove the followup branch
git branch -d "followup/$SR_TAG"
# Push the changes
git push
If we can’t push directly to develop
we can create a MR pushing the followup
branch after committing the changes, but we have to make sure that we don’t
squash the commits when merging or it will not work as we want.