Retrieving Vault Secrets Using GitLab Tokens

January 20, 2022

written by:

One of the lesser known features of GitLab is the generation of Java Web Tokens (JWT) during job build time. These tokens are generated in the community edition of GitLab, too. They present a sufficiently secure proof of origin so they are ideal to gather and use secrets from other sources... Enter Vault!

flavor wheel

The JWT GitLab generates is stored in the environment variable ${CI_JOB_JWT}. It is treated as a masked variable. Thus, simply echoing it to the log will be prevented by GitLab.

Inspecting a GitLab web-token reveals which useful properties which are stored in the token. As an example, here is the claims section of a decoded json web token:

{
  "namespace_id": "34",
  "namespace_path": "test",
  "project_id": "231",
  "project_path": "test/playground",
  "user_id": "334",
  "user_login": "<USERNAME>",
  "user_email": "<USER E-MAIL>",
  "pipeline_id": "3322",
  "pipeline_source": "web",
  "job_id": "40643",
  "ref": "main",
  "ref_type": "branch",
  "ref_protected": "true",
  "jti": "6ce50bd0-cc7a-4438-a0e9-833f12c38048",
  "iss": "<GITLAB HOST>",
  "iat": 1642087778,
  "nbf": 1642087773,
  "exp": 1642091378,
  "sub": "job_40643"
}

The most noticeable attributes in the token claims are probably the information about the user, who started the build job (user_login, user_email), the project (project_id, project_path), the namespace (namespace_id, namespace_path), and the branch (ref, ref_type, ref_protected). Check out the corresponding GitLab documentation.

As you may know, JWTs are cryptographically signed. A third party cannot modify this token unnoticed. The only exception is, if the third party was able to aquire the secret signing key. So, let us assume GitLabs signing key is secret - and stays that way.

Let us assume further, that the branch is protected against malicious code entering it. In terms of GitLab, this may be realized by employing Guarded Commits or Code-Reviews upon Merging (Pull-Requests, Merge-Requests). This way, sneaking in code, for stealing the token is prevented.

Now, we have a sufficiently secure proof of who (User, who was authenticated by GitLab) is trying what (Project / Branch) to build. The token is ideal to gather and use secrets from another source.

HashiCorp Vault JWT Authentication

In case you have not heard about HashiCorp Vault. Vault is a software to securely and tightly control access to tokens, passwords, certificates, encryption keys and other secrets.

Vault offers different ways to authenticate requests, one of them is using the JWT Auth Method (others are LDAP, OAuth, Username + Passwort, as well as others).

Auth Method Configuration

In order to use the JWT auth method, it first needs to be enabled. A path may be provided, which prevents collisions if multiple JWT auth methods are used.

vault auth enable --path /jwt/gitlab jwt

Remember the path /jwt/gitlab.

Next, the auth method needs to know, how to validate token information. GitLab has an json web key sets (JWKS) endpoint which publishes public keys to validate the signature of JWTs. The endpoint URI is https://<GITLAB HOST>/-/jwks and it is set with jwks_url.

If there was no such endpoint, it is also possible to provide public keys for JWT signature validation. For such a scenario, the field jwt_validation_pubkeys accepts an array of Strings with public keys or a String with a comma spearated list of public keys. - However, you need to pick one, either jwks_url or jwt_validation_pubkeys.

Additionally, during authentication, we can check the issuer (iss) field of the JWT claim with the auth method configuration.

vault write auth/jwt/gitlab/config \
  jwks_url="https://<GITLAB HOST>/-/jwks" \
  bound_issuer="<GITLAB HOST>"

This configuration remains the same for all projects on GitLab. However different projects, users, branches, ... may require different secrets. This is where roles come into play.

Auth Method Roles

In order to allow access control to be tightly knitted to the token claims, a role for the JWT auth method needs to be defined. You need at least one role, if you want to define finer grained access control measures, you may define many different roles.

For example, you may define roles for access to production secrets from the main branch, but access to test secrets only for any builds from other branches.

You may restrict access to secrets to any information which is mentioned in the claims section of the JWT.

Access, however, is not granted directly in the role definition. But by selecting policies which are assigned to the vault token which is created if the JWT token is authenticated. In further requests, this token needs to be provided and the policies assigned to that token determine which actions may or may not be invoked.

vault write auth/jwt/gitlab/role/project-demo-auth - <<EOF
{
  "role_type": "jwt",
  "policies": ["gitlab/prepare/project-demo-policy"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims_type": "string",
  "bound_claims": {
    "project_id": "231",
    "ref_protected": "true",
    "ref_type": "branch",
    "ref": "main"
  }
}
EOF

Please notice the path in line 1: /jwt/gitlab refers to the path defined in the setup of the auth method configuration. project-demo-auth is the name of the role we are just defining.

As explained aboive, Vault uses policies which define access restrictions. Policies can be reused and combined in different roles - and also other auth methods. The field policies refers to a list of policies. In this case, we will define a policy project-demo-policy which will allow access to the secrets the demo project requires. Vault does not check, whether the policy already exists. If it does not exist, when a request with a JWT comes in, the JWT token may be authenticated and a Vault token is returned - but the Vault token will not be authorized to do anything, because there is no policy granting anything.

The field user_claim specifies which field of the JWT contains (unique) user information. In our case we set it to the user_email field of the GitLab token. This information will be present in the Vault token as well, and will be logged.

bound_claims_type may either be string or glob where string enforces strict checking of the bound claims values, whereas glob allows to use wildcard * characters to match any characters.

The most interesting part is the definition of the map in the bound_claims fields. Each key in that map refers to a claims field in the GitLab JWT token, and each assigned value needs to be matched by the JWT to be authenticated. In the case above, access is limited to the project (id = 231), for which we have presented the JWT at the beginning of the article. Further, access will be limited to token which are created for a protected main branch, only. This way it is very easy, to control access based on the user or the project namespace, too.

Vault generates tokens, upon authentication. These tokens are used for all further requests to Vault. Tokens may be renewed and expire. The value for token_explicit_max_ttl specifies how long a token is valid the longest (including renewals). Set this value to a small but reasonable value. If a token gets - for whatever reason - stolen, the time-span in which the token may be exploited, is limited by this value.

Additionally, it is possible to add a token_bound_cidrs field, whose value is a JSON array of Strings, or a comma separated String of IP addresses. This enables a functionality which restricts access to blocks of IP addresses. This is a neat feature, because not only will the authentication of the JWT token be restricted to the given IPs, but also further requests with the granted Vault token will be checked against the same addresses.

Beyond GitLab

The Vault JWT Auth Method is a versatile tool to grant access to secrets stored or created in / by Vault. By creating several roles for an auth method, granulartiy of access can be varied. Also, it is rather simple to adapt this method to other applications, which generate JWTs.