<BlogImage
 src="/gitops/flipt-gitops.png"
 alt="Flipt for GitOps"
 width={2400}
 height={1350}
>
 
 Image generated by <a href="https://www.midjourney.com/">MidJourney</a>
 
</BlogImage>

Today we’re excited to announce that we’re exploring a new type of backend for Flipt!

The team has been discussing and exploring ways Flipt could be better positioned to support GitOps-style workflows. For those unfamiliar with GitOps practices, we recommend the WeaveWorks [Guide To GitOps](https://www.weave.works/technologies/gitops/).

Our new [filesystem backends proposal](https://github.com/flipt-io/flipt/discussions/1652) adds two ways to configure Flipt to serve flag state content encoded as yaml stored in files, `local` and `git`.

We’re keen to understand if this idea resonates with you and your use case, this will help us shape the implementation, which is currently underway.

## Problem

We feel that many feature flags offerings (including Flipt today) work at odds with GitOps practices. GitOps prescribes that you store all your configuration in one or more Git repositories. This gives your entire system the benefits of version control. Feature flag configuration is just like any other configuration in that it adjusts the behaviour of your applications at runtime. However, we find that it is rarely stored and tracked within Git itself.

If you’re using GitOps practices for your wider configuration needs, but your feature flag solution is completely isolated, then you’re subverting the benefits and traceability of Git. When the alerts start firing you'll have to correlate your configuration changes in Git with the auditing features of your flagging solution.

Alternatively, some existing feature flag solutions support GitOps by bolting a declarative syntax onto their existing imperative API. For example, some offerings provide a [Terraform](https://www.terraform.io/) provider which is an entirely legitimate way to create a GitOps-style contribution experience. In fact, we’re still considering having one for Flipt and our existing relational backends. However, this approach does have drawbacks, primarily, there are now two sources of truth.

Reconciling from your Git repository into your online feature flag service also has to be performed. This process is often performed during some kind of continuous delivery step (e.g., via a GitHub action), which likely pushes changes into your feature flag system. You may have to manually intervene when this process fails for intermittent reasons. Your environment must also have credentials and access to your application repository and your feature flag solution. Ideally, your feature flag solution provides tooling to ease the extra work required to set up this CI/CD process and make it resilient.

In this scenario, given you’re using a self-hosted solution (such as Flipt), you now have the following set of applications to run and configurations to manage:

- The feature flag service itself (e.g. the `flipt` binary)
- Backing database for storing flag state (e.g. SQLite, PostgreSQL etc)
- Flag state configuration (e.g. in Terraform HCL or via some other process such as `flipt import`)
- CI processes for reconciliation (plus monitoring)
- For solutions such as Terraform, storage of reconciliation state (`tfstate` files)

This is all achievable, but care must be taken when building the reconciliation process to ensure a predictable end state.

Ultimately, feature flag offerings have to build a host of features you’re already getting from your SCM (e.g. GitHub or Gitlab). Version control, peer review, access control and backup to name a few.

## How We Intend to Serve GitOps

We intend to have Flipt pull feature flag state directly from Git at runtime.

We want Flipt to be able to drop its dependency on a database for flag state and go directly to the source. This removes the need for a reconciliation process altogether. By adopting a pull model, we embrace eventual consistency and allow Flipt to recover automatically during transitive errors without intervention.

You'll simply be able to configure Flipt with access to your repository and with some additional configuration, you will be able to control how and where to locate your flag state files. Your repository could be used solely for Flipt, it could be your existing configuration repository, or it could be stored within a monorepo.

<BlogImage
 src="/gitops/gitops-scm.png"
 alt="Flipt for GitOps - SCM"
 width={800}
 height={600}
/>

Flipt’s evaluation and read APIs won't change, meaning your existing interactions with the Flipt SDKs can remain the same. The existing [`flipt export`](https://www.flipt.io/docs/configuration/storage#export) command produces configuration files valid for committing straight to your repositories for ease of transition.

Flipt will be configured to serve a particular branch for a target repository. This branch doesn’t always have to be `main`, meaning you can deploy Flipt in multiple ways to support validating flag state changes in arbitrary Git branches or tags.

You'll also be able to run Flipt over a local directory. In this configuration, Flipt will simply serve your flag configuration without the need to go through Git. Updates to the local directory will update Flipt automatically. There is no need to access an external running instance of Flipt to replicate or alter your production feature flag state in your _local_ environment. Simply check out your repository and run `flipt` to begin experimenting.

<BlogImage
 src="/gitops/gitops-local.png"
 alt="Flipt for GitOps - Local"
 width={800}
 height={600}
/>

## Flipt in CI

Alongside building a new core backend for Flipt, we’re also developing a suite of tools for validating your use of Flipt during your existing continuous integration workflows.

Before Flipt can serve your configuration files, it is important to validate they are fit for consumption. Failure to do so could lead to runtime errors or unexpected behaviour. `flipt validate` is a new subcommand currently being added to Flipt. This will be useful for local validation or running directly in a CI pipeline.

We’re also developing a GitHub action which simplifies running `flipt validate` which will make it trivial to add alongside your existing GitHub workflows.

## Just the Beginning

We’re just getting started turning our proof-of-concept into something ready for production Flipt usage. While we have a good feeling that the experience will resonate with users, there are still a number of interesting technical challenges we expect to have along the way. This is why we wanted to get this plan out early in order to hear your feedback.

Help come shape the future of this solution on [GitHub](https://github.com/flipt-io/flipt/discussions/1652), or in our [Discord](https://www.flipt.io/discord).

Also, check out our new landing page for more of what's possible when combining Flipt + Git: [flipt.io/gitops](https://www.flipt.io/gitops).

How's about Git?

<BlogImage
 src="/audit-events/audit-events.png"
 alt="Events for Flipt"
 width={2400}
 height={1350}
>
 
 Image generated by <a href="https://www.midjourney.com/">MidJourney</a>
 
</BlogImage>

Events and logging are vital for many organizations to either keep track of history within the application/platform or to actually take action on things once they happen. Two brief examples include:

1. An organization keeping track of payment events that have happened on their platform
2. A platform triggering a job to purge data somewhere if an event is of a particular nature

In Flipt's case it may prove essential for some organizations to know when feature flags are mutated in some fashion as they can affect the real-time behavior of the consumer’s application.

It's also important to know who made the change and when it was made, especially when it comes to trying to determine the root cause of an issue in the consuming application. This is where audit events come in.

We're happy to announce that support for these type of audit events are now implemented as of [v1.21](https://github.com/flipt-io/flipt/releases/tag/v1.21.0) of Flipt! 🎉

## Ideal Implementation

Before diving into how the feature was implemented, it is worth discussing what an ideal scenario looks like for Flipt and audit events. Many providers, whether SaaS or open source, leave it strictly up to the consumer to deal with events as they please. In most cases, this is done through a webhook URL that you provide to the platform which sends a request to that URL when an event occurs.

An application that is listening to the endpoint of the webhook URL can then do whatever they want with the event. For instance, they can choose to send the message to a Kafka broker, do some other sort of pub/sub, or send that audit event to Slack in a specialized channel, so this is more of the laissez-faire approach.

What we decided on for an ideal implementation is to have this concept of first-class support for ‘native sinks’. In this case, Flipt will house the logic necessary to interact directly with the sinks that the user provides configuration for. The idea is that messages can go straight to a Kafka broker, pub/sub system, Slack channel, etc, as a direct form of communication. This allows users to avoid having to write their own backend to communicate with these systems and avoid the extra network hop.

The implementations for communicating with these native sinks would ideally have an abstraction point so that users can make easy sense of it to contribute to Flipt and write their own implementations for native sinks. One of the downsides of this native approach, however, is that as more and more sinks become implemented, it can lead to version churn for the Flipt application. To combat this potential for churn and to make contributions more accessible, we are currently looking into a way to make the sink implementations pluggable in their own repositories, potentially using a plugin system like Hashicorp's [go-plugin](https://github.com/hashicorp/go-plugin).

## Implementation

Implementing any event-driven system comes with many considerations and trade-offs. On the producer side of the system, an ‘event’ usually represents things in the system that have already happened to then notify the consumer. The benefit for the consumer(s) is that once the event is received on its end it knows the event is complete, and therefore does not have to poll and waste resources getting the complete event from the source.

While the definition of an event-driven system seems simple enough, there are many considerations to keep in mind. In this case particularly, what happens to the producer or consumer during high-traffic scenarios? Or what if the consumer becomes unavailable for a portion of the time? These represent just a few of many considerations, and we sought to address those in our implementation.

<BlogImage
 src="/audit-events/event-diagram.png"
 alt="OTEL Event Diagram"
 width={300}
 height={261}
 caption="The two basic components of an event system. One has to consider where to implement the functionality to deal with the considerations described above."
/>

Our first stab at an implementation was a basic homegrown publisher which contains abilities to batch events and send batches asynchronously to different sinks that are configured. This is done so the consumer is not burdened with a whole bunch of writes, and it frees up the publisher to do other work without worry of communication failures with the consumer (addressing some of those considerations mentioned above).

In addition to a batch size, there should also be configuration for a flush period to avoid messages laying around in memory for an indefinite amount of time. Considering all of this, it was a pretty straightforward implementation, but halfway through we realized that this problem has been solved several times for a lot of different use cases, leading us to pause the publisher implementation and seek out a more tried and true solution.

### Enter OpenTelemetry

<BlogImage
 src="/audit-events/otel.png"
 alt="OpenTelemetry"
 width={1000}
 height={524}
/>

For the sake of brevity, [OpenTelemetry](https://opentelemetry.io/) will not be discussed in this article, but simply how it was used in the Flipt application to achieve desired functionality for audit events. For trace data, OpenTelemetry (OTEL) not only provides configurable batching, and flushing abilities but also allows users to provide implementations of [`Exporters`](https://opentelemetry.io/docs/concepts/glossary/#exporter) which OTEL uses to send trace data to whatever the implementation specifies. In simpler words... OTEL does everything already implemented in the homegrown publisher implementation, and it already exists as a dependency in the Flipt source code for exporting trace data to [Jaeger](https://www.jaegertracing.io/), [Zipkin](https://zipkin.io/), or [OTLP](https://opentelemetry.io/docs/reference/specification/protocol/).

There might be some lack of context as to what a `Trace` or `Span` here is and how it ties into audit events for Flipt. Essentially, a `Trace` represents a unique transaction of a collection of operations throughout an application. A `Span` represents a single unit within that `Trace` or a single operation. Spans allow for additions of [Events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events) which are viewable through any tracing provider client. The addition of `Event`s is the bit that we leveraged for audit events.

Specifically in the code, we’ve implemented a middleware that will intercept the request, and determine if the event is auditable, if it is, the code will add the details of that audit event to the Span via the `Event` described above. When a batch size or flush period is reached for the `SpanProcessor`, events are pulled off of the `Span`(s) and converted into a data structure housed in our code to then send it off to the native sinks that the user has configured on their Flipt instance. In addition to viewing the audit events in their sink implementation, an added benefit of using OTEL is the span events will show up in the tracing provider client for users to view.

<BlogImage
 src="/audit-events/jaeger-ui.png"
 alt="Jaeger UI with Event"
 width={3024}
 height={1542}
 caption="The audit event added to the flipt.Flipt/CreateConstraint span. The flipt* properties represent the audit event."
 className="shadow-md"
/>

## What's Released

With this initial release, there is only functionality for audit event logging to a file on disk. Users can specify the name of that log file, and Flipt will write the audit events to it in a JSON-encoded format.

If you were to `tail` the logs of your audit event log file for orthodox Flipt usage you might see something like the following:

```json
{"version":"0.1","type":"flag","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"description":"your favorite NBA team","enabled":true,"key":"scenter","name":"scenter","namespace_key":"default"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"variant","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"attachment":"","description":"","flag_key":"scenter","id":"41e84846-bfbd-426f-afa9-3358d0d40875","key":"lakers","name":"","namespace_key":"default"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"segment","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"constraints":[],"description":"Segments for evaluation NBA teams","key":"postclavicula","match_type":"ANY_MATCH_TYPE","name":"postclavicula","namespace_key":"default"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"constraint","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"id":"a7ccde21-870e-4e36-953c-4d5849d96a77","namespace_key":"default","operator":"eq","property":"championship","segment_key":"postclavicula","type":"STRING_COMPARISON_TYPE","value":"most"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"flag","action":"updated","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"description":"your favorite NBA team","enabled":false,"key":"eyepoint","name":"eyepoint","namespace_key":"default"},"timestamp":"2023-05-01T14:01:37-05:00"}
{"version":"0.1","type":"token","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"io.flipt.auth.token.description":"hello","io.flipt.auth.token.name":"worldh"},"timestamp":"2023-05-01T14:02:45-05:00"}
```

Currently, the auditable events are all CRUD operations (sans Read) upon the following entities:

- [Flag](https://www.flipt.io/docs/concepts#flags)
- [Variant](https://www.flipt.io/docs/concepts#variants)
- [Distribution](https://www.flipt.io/docs/concepts#distributions)
- [Segment](https://www.flipt.io/docs/concepts#segments)
- [Constraint](https://www.flipt.io/docs/concepts#constraints)
- [Rule](https://www.flipt.io/docs/concepts#rules)
- [Token](https://www.flipt.io/docs/authentication/methods#static-token)
- [Namespace](https://www.flipt.io/docs/concepts#namespaces)

The event payload which is logged has the following fields for each event:

- `type`: the type of entity changing
- `action`: the action taken upon the entity
- `metadata`: extra information about the event such as identity of the subject who initiated the event
- `payload`: the actual payload of the request on Flipt
- `timestamp`: the timestamp of when the event was created

Any combination of actions on those entities will result in an event with the above fields being logged out to the file you specify via configuration:

```yaml
audit:
 sinks:
 log:
 enabled: true
 file: /tmp/flipt/audit.log
```

We also have an example of how you could potentially collect, parse, visualize, and query these events using Grafana's [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) + [Loki](https://grafana.com/docs/loki/latest/) in our [examples](https://github.com/flipt-io/flipt/tree/main/examples/audit) on GitHub. This would allow you to query the audit events in a similar fashion to how you would query logs in a traditional logging system, even when Flipt is running in a distributed fashion.

<BlogImage
 src="/audit-events/loki.png"
 alt="Grafana Loki Query"
 width={2880}
 height={1800}
 caption="Querying Flipt audit events using Grafana Loki"
 className="shadow-md"
/>

## What's Next

We are excited for this release to see just how users will interact with this feature, and also excited for community discussion on potential event sinks that have not been added yet.

[Contributions](https://github.com/flipt-io/flipt#contributing) are always appreciated! This [document](https://github.com/flipt-io/flipt/tree/643ccc1d541e6a1e79cfeeb548df6b53a09e9914/internal/server/audit) describes how you can implement your own audit event sink.

We hope you found this post useful. As always, you can find us on [GitHub](https://github.com/flipt-io), [Discord](https://www.flipt.io/discord), [Twitter](https://twitter.com/flipt_io), or [Mastodon](https://hachyderm.io/@flipt).

Audit Events for Flipt

<BlogImage
 src="/dagger/embracing-dagger.png"
 alt="Flipt loves Dagger."
 width={2400}
 height={1350}
>
 
 Background by [Paul Volkmer](https://unsplash.com/@laup) on Unsplash
 
</BlogImage>

In a [previous blog post](/blog/generating-the-flipt-go-sdk), I briefly mentioned our adoption of Dagger.
I wanted to take a moment to introduce Dagger and explain how we’re slowly but surely adapting our build, test (and one-day release) pipeline to leverage it.

> Dagger is a programmable CI/CD engine that runs your pipelines in containers.

[Dagger](https://dagger.io) lets you describe your build, test and release pipelines as code, often alongside the application you’re developing it around. They support an expressive set of SDKs in multiple programming languages. At Flipt we’re using Dagger's (you guessed it) [Go SDK](https://github.com/dagger/dagger/tree/main/sdk/go).

Perhaps unsurprisingly, in Dagger, almost everything is a container. I say unsurprisingly since a lot of the team also gave us Docker, the technology which put Linux containers truly on the map. Each of the Dagger SDKs exposes a builder pattern interface with the primary function of describing a graph of instructions. These instructions are remarkably similar to those found in a Dockerfile, which is not a coincidence. These instructions also produce layers in resulting OCI images, which can then be published or exported from your Dagger pipeline.

I don’t know about you, but I do love a DAG and so does Dagger. The DAG (Directed Acyclic Graph) is intrinsic to what makes it such a neat solution. No points for guessing where they got their name.

<BlogImage
 src="/dagger/dag.svg"
 alt="A DAG"
 width={509}
 height={144}
 caption="A DAG"
/>

A DAG is a set of vertices (the boxes) and edges (the arrows). The edges express dependencies from one vertex to the next. Before I can copy my Flipt binary into an appropriate location for execution, I first need to build it from the source. In order to run `go build` I first need to install Go.

DAGs allow us to express the dependent order in which operations must occur. Where this differs from a plain old sequential list, is that each vertex (entry) can have more than one (potentially) independent inputs. When you describe your workflow in this way, an execution engine (such as Dagger) can leverage the fact that fully _independent_ operations can be executed concurrently. If you’re running your workflow on a multicore machine you may even see these operations run in parallel.

In the rough illustration above, our Go and Node components can be built independently from one another. At this stage there are no interdependencies, so they can be run concurrently. Once both operations have been completed, we bring the results together.

### How does it work?

Dagger itself is comprised of a server and client architecture. The server (known as the engine) runs in a container on a target Docker host. Consumers (like Flipt) use one of the Dagger-generated and managed SDKs in one of the supported languages. These SDKs managed to initialize the Dagger engine container for you, as well making all the GraphQL API calls. Dagger has chosen to embrace GraphQL, which is a particularly good fit when you’re talking in directed acyclic graphs.

Having a GraphQL API means consumers are not strictly bound to only the SDKs developed by the Dagger team. Anyone can take the GraphQL docs and bake their own client in the language of their choice. This is similar to how we at Flipt use protobuf and gRPC and generate our managed clients from these definitions.

<StarCTA />

Under the bonnet Dagger leverages [Buildkit](https://github.com/moby/buildkit/). It was originally created by [Tõnis Tiigi](https://github.com/tonistiigi) and now lives under the Moby (Docker) brand. It powers most of `docker buildx`, where Dockerfile’s are actually transpiled into `llb` instructions (A Buildkit structure for describing DAGs) and then _solved_ (evaluated) by a Buildkit daemon.

If you’re familiar with the concept of `git` and its _plumbing_ vs _porcelain_ commands, I would liken the same to Dagger compared with Buildkit. Buildkit is the plumbing and Dagger is the porcelain. Buildkit exposes a fine-grained API for solving DAGs of instructions, which produce OCI images and layers. Dagger is a fast-evolving, developer-friendly layer on top. A layer focussed on solving the higher-level problem of expressing build, test and deployment pipelines in code.

I could talk about these pieces of tech all day, but for now, here is how we’re leveraging them at Flipt to build and ship quickly with confidence.

## Flipt’s Build, Test, Release Pipeline

<BlogImage
 src="/dagger/build-test-deploy.png"
 alt="The classic build, test, and release pipeline"
 width={2570}
 height={694}
 caption="Build, Test, and Release"
/>

Using Dagger we’re developing the classic build, test, and release sequence.

### Build Phase

The Build phase involves sourcing all the raw materials and putting the puzzle together. Effectively we perform the following five-step process:

1. Establish a base image containing all the build and test tools necessary.
2. Fetch all of our source code from GitHub.
3. Download all the Go and Javascript libraries we need.
4. Build the thing(s).
5. Copy the resulting Flipt binary into its own image with only the necessary _runtime_ dependencies.

If you’ve had experience writing a `Dockerfile` in the past, you may be familiar with some tricks folks employ to make better use of the Docker build cache. For example, with Go you might first add just your `go.mod` and `go.sum` files to your container before the rest of your application. Next, you will likely run `go mod download` in order to populate the module cache. Once this is complete then you add the rest of the application source and perform the build. You might also employ Docker's newer `RUN` cache features and mount a cache volume.

```docker
FROM golang:1.20

WORKDIR /src

ADD go.mod .
ADD go.sum .

RUN --mount=type=cache,target=/go/pkg/mod go mod download

ADD . .

RUN --mount=type=cache,target=/go/pkg/mod go build ./...
```

The same can be achieved with Dagger and the experience is familiar. The following is an illustration of using the Go SDK in Dagger to layer our graph to make the best use of caching for Go applications.

```go
func GoApp(ctx context.Context, client *dagger.Client) (*dagger.Container) {
	deps := client.Host().Directory(".", dagger.HostDirectoryOpts{
		Includes: []string{
			"./go.mod",
			"./go.sum",
		},
	})

	return client.Container().
		From("golang:1.20").
		WithWorkDir("/src").
		WithMountedDirectory("/src", deps).
		WithMountedCache("/go/pkg/mod", client.CacheVolume("go-mod-cache")).
		WithExec([]string{"go", "mod", "download"}).
		WithMountedDirectory("/src", client.Host().Directory("."))
		WithExec([]string{"go", "build", "."})
}
```

Our pipeline for building Flipt ultimately leverages this same process, which is to use layers effectively and take advantage of the engine's layer caching, and mounted cache volumes for finer control ensuring fetched or computed assets are preserved between builds. As with Go, the same applies for Node and its dependencies. First add the dependency manifests (`package.json`), install the dependencies and then add the source code.

Beyond this, we have some other details which are a little more specific to how we build Flipt:

1. The Flipt repo is a Go multi-module repository and uses a [Go workspace](https://go.dev/doc/tutorial/workspaces) to tie it all together. We read the contents of the `go.work` file to identify the multiple sets of `go.mod` and `go.sum` file pairs through the repository and ensure they’re all added before downloading dependencies.
2. `client.CacheVolume(...)` in Dagger takes a string to identify the cache volumes key. We build this key based on a hash of the `go.sum` contents. This invalidates our cache each time our sum file changes. The idea is to keep our cache fresh over time and stop it from growing indefinitely.
3. We initially exclude adding the `ui` directory containing our frontend code from the Go base. Instead, we mount this directory into its own container based on a `node` image and produce a static directory containing all of our frontend assets. This resulting static assets directory is copied into the base Go container and we use Go’s embedding to bake this into the binary.

This process is defined in the function we call `Base(...)` and the result is a `*dagger.Container`. Once all the dependencies have been built and added appropriately, we build the `flipt` binary into a well-known path inside the last layer of the container.

There is another function called `Flipt(...)` which takes the resulting container from `Base(...)` and copies the `flipt` binary into a fresh, empty `alpine` Linux image. This image is prepared with just the runtime dependencies needed for Flipt and gives us a thin distributable image for release.

### Test Phase

There are two important artifacts out of the Build phase which become inputs into our Test phase. The first is the base image (the result of steps 1, 2 and 3) with all the build and test tooling and source code. The second is the distributable Flipt OCI image (the thinner distributable image based on `alpine`).

In the testing phase, we first use the base image again to run our suite of Go unit tests. We run our unit test suite against each of the databases Flipt supports (SQLite, Postgres, MySQL and CockroachDB). The unit test suite requires Go to be installed, along with access to all the source code, which is why we use the base image.

The release of Dagger `v0.4.0` came with support for service containers. If you’re familiar with `docker-compose` and how it supports multiple containers being networked together, then this feature will be familiar. It supports the ability to run containers which are used as dependencies for your workflows as network addressable services. For example, a backing database, or your very own service that you want to test over the network. This feature was crucial for connecting the various databases Flipt supports during unit testing. It also allowed us to further build out our integration testing pipeline.

In a [previous blog post](/blog/generating-the-flipt-go-sdk), we discussed building our new Go SDK. In order to gain confidence in Flipt, its API and our SDK, we leverage the full power Dagger gives us with defining our pipelines in Go.

We first defined a suite of test cases using the Go SDK, which exercises a common set of Flipt operations: create a namespace, create a flag, define its variants and segments, evaluate it, etc. The core functional requirements of Flipt as a feature flagging system.

Given how we built our SDK to abstract away details such as the transport used and authentication, we could then easily extract those (non-functional requirement) details as arguments for running our test suite. This means that we could run our suite against the HTTP or gRPC API with or without authentication enabled. With the introduction of namespaces in our [latest release](https://github.com/flipt-io/flipt/releases/tag/v1.20.0), we further enhanced the suite to run across a configurable namespace. Now that our test suite can be run in a number of configurations, we have an n-dimensional matrix of possibilities.

<BlogImage
 src="/dagger/test-matrix.png"
 alt="Our test matrix"
 width={444}
 height={394}
 caption="Our test matrix"
/>

By simply using a set of nested loops in Go, we have a tidy way of expressing the multitude of possible ways Flipt could be run, giving us confidence that we’re not breaking something each time we add or change some functionality. Couple this with the ability to execute each possible configuration combination in a goroutine and we have a fast, concurrent way of executing the many permutations.

```go
func TestAll(base, flipt *dagger.Container) error {
	var g errgroup.Group
	for _, auth := range []bool{true, false} {
		for _, namespace := range []string{"", "some-namespace"} {
			for _, transport := range []string{"http", "grpc"} {
				g.Go(testFlipt(base, flipt, auth, namespace, transport))
			}
		}
	}
	return g.Wait()
}
```

Our pipeline is now fairly large and relatively complex. Lots of layers and concurrent operations in a web of directed edges and vertices are used to describe both build and test functionality. As with all big and complex CI/CD pipelines, they can start to slow down the more you add. This is where the advanced caching capabilities of Buildkit and Dagger come into play.

The beauty of the DAG in the Buildkit world is that each computed vertex (node) in the graph and its outputs are stored in content-addressable storage. Buildkit takes advantage of when vertices whose definitions and their inputs do not change and avoids recomputing any of these outputs. Instead, it retrieves previously computed results and uses those instead.

Locally, we have seen a full 10-minute end-to-end Flipt build pipeline with subsequent integration suite go from 10 minutes with an empty cache to **17 seconds** on a subsequent run. Flipt currently references 3.2GB **(!)** of Go module dependencies. When you compile Flipt with an empty build cache, a fresh run will produce 1GB of build-cache entries. All this takes a considerable amount of time to fetch and compile. Dagger makes a lot of this work trivial to compute once and reuse on subsequent runs.

Buildkit has the functionality to export its cache storage to a number of remote backends so that you can persist all the hard work done computing layers and cache volumes in CI/CD environments and reuse them between separate triggers of your pipeline. Dagger today (`v0.6.0`) does not expose a stable API for this feature. However, the team are hard at work sweating the details so we won't have to. They’re working hard to find a DX that works for many. At Flipt we’re looking forward to when this lands, as we expect the gains in CI will be very noticeable.

### Release Stage

Once Flipt is packaged and tested sufficiently, it is ready for release. Today we still use the excellent [GoReleaser](https://goreleaser.com/) project to package and ship the final Docker images and binaries to their various destinations. We do suspect though, that this will move to be more Dagger driven as time goes on, ultimately publishing the artifacts produced by the pipelines that we have developed using Dagger.

All the code to define our Dagger-based pipeline can be found in the [build](https://github.com/flipt-io/flipt/tree/main/build) directory of the Flipt repository. We use [mage](https://magefile.org/) both in the root of the Flipt repository and in our Dagger pipeline code as the CLI wrapper. We’re big fans of `mage` and so is Dagger. For those wanting a `make`-like interface which is defined in Go, it is a nice intersection of the two. It also marries nicely with all of our pipeline code being defined using the Dagger Go SDK.

### Dagger All The Way

Dagger is shaping up to be the solution I always wanted for CI/CD. Call me strange, but I am actually excited to develop and maintain these pipelines and I am hopeful it is going to be a sharp tool in our belts here at Flipt.

If you’re interested in Dagger then I highly recommend checking out their [site](https://dagger.io/), [GitHub project](https://github.com/dagger/dagger) and in particular [Discord](https://discord.gg/dagger-io).

Equally, we really want to understand your needs in building and shipping software here at [Flipt](/). So please do come and stop by our [Discord](https://www.flipt.io/discord) and say hi too.

Building the future of Feature Flags

Deploy With Confidence

Works with your existing stack

Features

Ship Safely

Context Targeting

Self Hosted

Secure

Fast

Easy Integration

Integrations

From the Blog

How's about Git?

Audit Events for Flipt

Dagger, a ❤️ story

Building the future of Feature Flags

Deploy With Confidence

Works with your existing stack

Features

Ship Safely

Context Targeting

Self Hosted

Secure

Fast

Easy Integration

Integrations

Keep up with what we're building

From the Blog

How's about Git?

Audit Events for Flipt

Dagger, a ❤️ story