Developer’s Guide to Writing a Good Helm Chart

Hugues Alary

2022/02/05

This post will guide you through the process of creating a good Helm Chart.

A good Helm chart is one that makes the components of your application intuitive, consistent and discoverable. When writing your chart, carefuly considering each one of these attributes will make debugging your application much simpler, prevent bugs and simplify the maintenance of your chart.

Refresher

Before creating a Helm chart, we must understand the relationship between Kubernetes and Helm, explain what they are and how they interact with each other.

You should have a minimum understanding of the core concepts of Kubernetes. Writing a Helm chart without a basic knowledge of Kubernetes would be akin to coding a React application without knowing Javascript.

Important

If you aren’t familiar with Kubernetes, you should head over to https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ before continuing to read this guide.

You should at a minimum be familiar with the most common Kubernetes resources:

Kubernetes

The important thing to understand about Kubernetes is that the only thing it understands is manifest files. In order to create Resources (Deployments, Services, HorizontalPodAutoscalers, etc) on a Kubernetes cluster, the end-user, you, creates manifest files that contains the Resource declaration.

Kubernetes itself has no knowledge of Helm.

Note

Manifest files can be written in YAML or JSON. However, YAML tends to be preferred and I would discourage using JSON. As such, from now on this guide will only mention YAML.

The only thing Kubernetes understands is plain YAML (or JSON) manifest files

Helm

Helm defines itself as the "Package Manager for Kubernetes".

It is, however, more than just a package manager; and the important thing to understand about Helm is that, at its core, it is a templating tool.

Helm allows you to generate plain YAML manifest files from a set of templates.

When creating a Helm chart, you are writing a re-usable and configurable template (the input) that will be converted to a Kubernetes manifest file (the output).

This output can then be sent to Kubernetes which will read and create the resources defined in this manifest.

At its core Helm is a templating tool

Summary

At the end of the day, the relationship between Helm and Kubernetes is the following:

Helm outputs a YAML manifest from a set of configurable templates, for consumption by a Kubernetes cluster.

Developer’s Guide

Now that we understand the relationship between Helm and Kubernetes, let’s dive into the actual implementation of a Helm chart.

The default Helm Chart Structure

Helm provides a command to automatically create the basic structure of a Helm chart: helm create NAME.

The folder structure provided by this command makes even medium sized Helm charts hard to manage.

Default structure

The default structure looks like the following:

.
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   ├── serviceaccount.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

We can see that all the templates are stored under the templates/ folder. Should one need to add a second Deployment, they would either create a new .yaml file under templates/ or add this new resource in the existing deployment.yaml.

Although this flat structure might seem fine for a small application, in practice even the smallest application grows to a size that requires a better structure.

Because the default structure created by helm create doesn’t scale, I recommend that you use the structure in the next section instead.

Recommended structure

I talk about the reasonning behind the structure below in details in a previous post, but here’s a TL;DR.

.
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployments
│   │   └── nginx.yaml
│   ├── hpas
│   │   └── nginx.yaml
│   ├── ingresses
│   │   └── nginx.yaml
│   ├── serviceaccounts
│   │   └── nginx.yaml
│   └── services
│       └── nginx.yaml
└── values.yaml

In the strutucture above, the Kubernetes resources have been grouped in folders named after the Kind of resource.

  • Deployment resources are under the deployments/ folder.

  • Service resources are under the services/ folder.

  • HorizontalPodAutoscaler resources are under the hpas/ folder.

  • Ingress resources are under the ingresses/ folder.

  • ServiceAccount resources are under the serviceaccounts/ folder.

  • Etc

With this structure, it becomes much simpler to reason about the design of an application made of many manifest files.

Note

Your Helm chart will not necessarily have some of the resource Kinds listed in this example. Only create folders for the resource Kinds you need.

Summary

The summary below is a straight copy/paste of the conclusion of my other post on the subject.

  • Group manifest files in directories named after the Kind of object: deployments, configmaps, services, etc. Note the directory name lower cased and pluralized.

    .
└── my-kubernetes-app
    ├── configmaps
    ├── crons
    ├── deployments
    ├── hpas
    ├── pdbs
    ├── podpriorities
    ├── pvcs
    ├── services
    ├── statefulsets
    └── ...

  • Shorten very long names with commonly used abbreviations, for example, hpas/ instead of horizontalpodautoscalers/, pdbs/ instead of poddisruptionbudgets/. Following the kubectl nomenclature is probably a good idea.

  • Use consistent manifest filenames between kinds/ directories. For example, if your app is named funnygifs-slackbot:

    • the Deployment should be named funnygifs-slackbot.yaml

    • the Service funnygifs-slackbot.yaml

    • the ConfigMap funnygifs-slackbot.yaml

    • the HPA funnygifs-slackbot.yaml

    • the PVC funnygifs-slackbot.yaml

    • etc.

  • Avoid stutter: do not call a deployment manifest funnygifs-slackbot-deploy.yaml or a service funnygif-slackbot-service.yaml, etc.

  • Use resource names consistent with the manifest filename:

    deployments/funnygifs-slackbot.yaml
    ---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: funnygifs-slackbot (1)

    1. This is consistent with the filename

    services/funnygifs-slackbot.yaml
    ---
apiVersion: v1
kind: Service
metadata:
  name: funnygifs-slackbot (1)

    1. This is consistent with the filename

    configmaps/funnygifs-slackbot.yaml
    ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: funnygifs-slackbot (1)

    1. This is consistent with the filename

      • etc.

Best practices

Below is a list of best practices to follow when writing a Helm chart.

However, there are situation where best practices might not be implementable or apply to your particular context. Use your best judgement.

Resources and Manifest Files

Do not put multiple resources in one manifest file.

For example, do not create a template file containing 2 Deployment or 1 Deployment and its corresponding Service.

Don’t do this:

nginx.yaml
---
apiVersion: apps/v1
kind: Deployment (1)
metadata:
  name: nginx
spec:
[...]
---
apiVersion: apps/v1
kind: Service (2)
metadata:
  name:  nginx
spec:
[...]
There are more than one resource in this file

  1. Deployment

  2. Service

Do this:

deployments/nginx.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
[...]
services/nginx.yaml
---
apiVersion: apps/v1
kind: Service
metadata:
  name: nginx
spec:
[...]

2 manifest files, each with their own Resource.

Resource Naming

Avoid stutter when naming your resources.

Don’t do this:

---
apiVersion: apps/v1
kind: Deployment (1)
metadata:
  name: nginx-deployment (2)
spec:
[...]

  1. Kind is Deployment

  2. -deployment in the resource name is a stutter. We already know this resource is a Deployment.

Do this:

---
apiVersion: apps/v1
kind: Deployment (1)
metadata:
  name: nginx (2)
spec:
[...]

  1. Kind is Deployment

  2. The resource name does not stutter. When we list deployments with the kubectl command we will see nginx and not nginx-deployment.

Templatized resource names

Your resource names should be templatized. That is, the metadata.name of your Resource should be a function of the {{ .Release.Name }} and a static descriptive name of your resource.

The default Helm scaffold provides a template function {{ include "my-chart.fullname" }}. Use this function as a prefix to your resource’s metadata.name.

Don’t do this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx (1)
spec:
[...]

  1. This resource name is entirely static and will make it impossible to deploy your Helm chart multiple times in the same namespace.

Do this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "my-chart.fullname" }}-nginx (1)
spec:
[...]

  1. This resource name is prefixed with the release name. It allows deploying your chart multiple times in the same namespace without conflicts.

Templatized "sub-components" names

On the contrary, "sub-components", "non-resources" names should not be templatized.

Don’t do this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "my-chart.fullname" }}-nginx
spec:
[...]
template:
    spec:
      containers:
        - name: {{ include "my-chart.fullname" }}-nginx (1)

  1. The container name: is templatized, this is bad

Do do this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "my-chart.fullname" }}-nginx
spec:
[...]
template:
    spec:
      containers:
        - name: nginx (1)

  1. The container name is static, and does not vary between Helm releases, this is good

Note
 the example uses a Deployment resource, but this is true for other resources

Template functions {{ define }} and {{ include }}

It can be tempting to try and make your chart as DRY as possible, and as such try {{ defining }} as many re-usable bits of templates as possible.

However, Helm is not a programming language; it is a templating tool and the {{ define }} and {{ include }} "functions" are far from being functions like you would expect in a programming language.

While in theory it might seem like a good idea to define many reusable templates, in practice this does not scale at all with Helm.

The {{ define }} and {{ include }} directive end up creating layers upon layers of indirection which can be very tricky to understand for anyone unfamiliar with your chart. Poor IDE support for this kind of indirection, coupled with subtle implementation details in Helm, make these directives a recipe for extremely hard to debug issues.

In summary, you should use {{ define }} and {{ include }} directives with extreme caution and carefully consider whether you really need to use them.

The values.yaml file(s)

fullnameOverride:

The default Helm scaffolding supports a variable called fullnameOverride. Setting this value will override the release name when installing a release.

For example, if you run helm install foo carta/foo, and fullnameOverride is set to fullnameOverride: bar, your Helm release will be automatically named bar.

This behavior will surprise your users since there’s no way to know in advance, aside from reading the Helm chart code, that the release name will be overridden.

This behavior will also prevent your users from installing your Helm chart multiple times in the same namespace, because two releases can not have the same name.

For these reasons, do not use fullnameOverride.

Hyphenated variable names

Helm does not handle hyphenated variables well and using them will cause obscure failures.

Don’t use hyphenated variables.

Don’t do this:

chicken: true
chicken-noodle-soup: true

Do this:

chicken: true
chickenNoodleSoup: true
Quoting string values

Always use the quote function if you need a value to be a string. YAML will implictly convert "number strings" into a number, "boolean strings" into a bool, etc.

Particular attention should be paid to values that are subsequently used in ConfigMap. ConfigMap resources expect all values to be strings.

Don’t do this:

values.yaml
webserverMaximumNumberOfWorkers: 10 (1)

  1. This is a number.

configmaps/webserver.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "my-chart.fullname" }}-webserver
data:
  WEBSERVER_MAXIMUM_NUMBER_OF_WORKERS: {{ .Values.webserverMaximumNumberOfWorkers }} (1)

  1. This will error out; the value is a number, ConfigMap resources expect string values.

Don’t do this:

values.yaml
webserverMaximumNumberOfWorkers: "10" (1)

  1. This is a string.

configmaps/webserver.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "my-chart.fullname" }}-webserver
data:
  WEBSERVER_MAXIMUM_NUMBER_OF_WORKERS: {{ .Values.webserverMaximumNumberOfWorkers }} (1)

  1. This will error out; the value in the values.yaml file is a string, but Helm will convert it to a number, and the ConfigMap resource requires a string.

Do this:

values.yaml
webserverMaximumNumberOfWorkers: 10 (1)

  1. This is a number.

configmaps/webserver.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "my-chart.fullname" }}-webserver
data:
  WEBSERVER_MAXIMUM_NUMBER_OF_WORKERS: {{ .Values.webserverMaximumNumberOfWorkers | quote }} (1)

  1. This will work thanks to the quote function

Feature gates

When creating feature gates, don’t re-use the same feature gate for multiple unrelated features

For example, imagine you have a configuration variable secretManagerEnabled: true|false.

Don’t do this:

{{ if eq .Values.secretManagerEnabled }} (1)
---
apiVersion: apps/v1
kind: Service (2)
metadata:
  name: redis (3)
spec:
[...]
{{ end }}

  1. This feature gate implies it controls the secretManager

  2. but it creates an unrelated Service

  3. called redis

env:, environment: and the likes

Don’t couple the "environment" to your Helm Chart.

Your chart might declare a variable called env: or environment: set to an arbitrary value corresponding to the name of the "environment" it will be deployed in. This configuration variable can be useful to configure components that need to report information about their environment; for example, a monitoring system needing to tag traces to differentiate between production traces and lower environment traces.

However, this value should not be used as a feature gate.

For example, imagine your Helm chart provides a "debugging-tools" Deployment that you only want to be enabled in a testing environment, or non-production environment.

Don’t use the environment: configuration variable as a "feature gate":

{{ if eq .Values.environment "test" }} (1)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: debugging-tools
spec:
[...]
{{ end }}

  1. {{ if eq .Values.environment "test" }} implicitly enables "debugging tools" when the environment is "test"; and prevents us from doing so in other environments.

Instead, create a configuration variable called enableDebuggingTools

Do this:

{{ if .Values.enableDebuggingTools }} (1)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: debugging-tools
spec:
[...]
{{ end }}

  1. {{ if .Values.enableDebuggingTools }} very explicitly allows us to enable debugging tools, regardless of the environment.

This effectively makes your Helm chart agnostic of the environment it runs in; allowing your debugging tools to be enabled regardless of the environment.

Non-Secret configuration values

As much as possible you should avoid storing non-secret configuration values inside a Secret.

For example, imagine you need your workload to access a database and you have 3 environment variables DATABASE_HOST, DATABASE_USER and DATABASE_PASSWORD.

Don’t store the DATABASE_HOST value in a Secret.

Store the DATABASE_HOST value in the values.yaml file (and ultimately in a ConfigMap).

Because the management of secret values is done out-of-band outside of source control, storing a non-secret configuration value like DATABASE_HOST in a Secret surfaces multiple issues:

  • It hides an important configuration value: the only way to know this value exists in the first place is by looking at the Secret.

  • It makes it possible to change DATABASE_HOST silently by modifying the Secret.

  • There’s no record in version control about this change.

This can lead to hard to debug situations whereby your application suddenly starts failing because someone changed the configuration stored in the secret, but a configuration change is the last thing suspected since there is no record of it.

If your workload uses configuration values that mix secret and non-secret values like a DATABASE_DSN, consider splitting DATABASE_DSN into multiple components (DATABASE_HOST, DATABASE_USER and DATABASE_PASSWORD) and removing DATABASE_DSN.

Note

What about DATABASE_USER? Depending on how you see it, one might think DATABASE_USER should be stored in ConfigMap since it is not really secret.

I think that, although DATABASE_USER isn’t secret, it also isn’t really a configuration value per-se and should be stored alongside its associated password in a Secret. The rationale being that should DATABASE_USER be changed by mistake, the error generated will generally be quite obvious: error: invalid username or password.

Note

This section uses DATABASE_* as an example, but this applies to any similar environment variable.

Whitespace Chomping {{- …​ -}}

Pay close attention to whitespace chomping operators {{- and -}}.

Whitespace chomping will sometime end up generating invalid YAML, and trigger errors similar to:

Error: YAML parse error on path/to/template/file.yaml: error converting YAML to JSON: yaml: line 25: mapping values are not allowed in this context

Use --debug flag to render out invalid YAML

You can prevent such error by not using chomping operators at all. Extra newlines do not generally cause issues.

Use helm template --debug to debug these issues. See the Tips section for more information.

Tips

helm template [--debug]

Helm provides a command helm template to render your chart templates locally and display the output.

This command can be very handy to verify that your chart’s generated manifest are what you expect them to be and debug issues.

The command’s argument are almost identical to the helm install arguments.

Use the --debug to force Helm render invalid YAML, which can be very handy when puzzling errors happen.

Use helm template --help for more information.

Comparing different versions of a Helm chart

Prior to Upgrading a release

If you are upgrading a running release, you can check the differences between the YAML manifests of the current release and the YAML manifests of the new release using a Helm plugin called helm diff.

Here’s how:

  1. Install Helm diff if you haven’t already: helm plugin install https://github.com/databus23/helm-diff

  2. Run helm diff upgrade [release-name] [helm-package] --reuse-values -f path/to/values/file.yaml

Tip
 Learn more about helm diff by running helm diff upgrade --help or reading the official documentation
During Development of a chart

When modifying/refactoring an existing Helm chart, you can check the differences between the YAML manifests generated by the previous Helm chart and the new manifests.

Here’s how:

  1. Checkout the main branch

  2. Run helm template my-release-name path/to/helm/chart -f path/to/values/file.yaml > old.yaml

  3. Checkout the new-chart branch

  4. Run helm template my-release-name path/to/helm/chart -f path/to/values/file.yaml > new.yaml

  5. Use your favorite diff tool to visualize the differences between old.yaml and new.yaml

Tip
 You can use git 's default diffing tool with untracked files by running git diff --no-index ./old.yaml new.yaml

General Advice

  • Write your code like you will release it in the wild as an open source project (even if you won’t).

    Framing your code this way will force you to decouple your application and your helm chart and make things more modular.

  • Make your chart agnostic of the environment it runs in

  • Emphasize consistency

    Consistency allows to easily recognize patterns in your chart, which makes it more intuitive to the next developer. It also allows major refactors via search/replace.

  • Avoid clever code and tricks

    Clever code is unintuitive and will be misunderstood by the next developer.

  • Avoid stutters

    Stutters happen when you repeat information already available. For example, calling a Deployment resource nginx-deployment is a stutter. Stutters do not provide any valuable information and make refactoring hard.

  • Don’t overuse {{ define }} {{ include }} just to make your code DRY

  • Read the official Helm best practices

Lastly, best practices are just that. Depending on the context, best practices can’t be followed, or result in more complex code; use your best judgement.