GitOps Is a Means, Where Is the End?

In this post, I will introduce you to a project called Cup that we have been experimenting with. But before we delve into Cup, let me take you through the journey we have embarked on to reach this point.

For those in a rush, here are some quick takeaways:

• We need better APIs for reading and writing configuration to Git
• You don’t have time to sweat the Git details, as you should be focusing on your user's needs
• Our new project Cup is attempting to solve the above problems

Gitting Going with GitOps

Back in May of this year (2023) we proposed our new filesystem backend for Flipt. These backend types aimed to enable a native GitOps workflow for feature flags and users of Flipt. Now Git can be configured as the source of truth for your feature flags.

We then proceeded to build and deliver this capability over the coming months. Fast forward to today, Flipt can be deployed in front of your local directory, a Git repository, or even AWS S3 thanks to one of our amazing contributors.

The main downside to using Flipt with a filesystem backend in its current form is that the UI enters into a read-only mode. With these backends, the feature flag state management happens through files committed to git, written to directories, or pushed to object storage. This process happens outside of our current UI’s purview.

Our UI is designed to work over Flipt’s existing management APIs. These APIs operate imperatively and are built around storing state in a relational database backend. While there are ways we could engineer these endpoints to write to storage files, add and commit them to Git, and push these changes, the end experience would still likely leave a lot to be desired.

An ideal solution we have imagined would be for the UI to support both Git and proposal mechanisms such as GitHub’s pull-requests, or GitLabs merge-requests. Changes in the UI could be composed into proposals and then made onto these Git providers. The UI would also then present and track the proposals made back to the user.

Going in Circles

While spiking on some of the pieces required to make a process like this work, I ended up reflecting on a lot of previous components and glue projects I had built with my past team working on a GitOps CD pipeline. I was reminded of all the pain and frustration many engineers came to us with around our configuration repositories.

• Where should the configuration for X live?
• How do I configure an environment variable for service Y?
• What’s needed to define an entirely new service?
• Where’s my recent application change deployed to right now?

To my team, reconfiguring a service's environment variables was a walk in the park. We did it every single day. But for newcomers, it was a real headache learning the code base. So instead we built tools and automation to read and write the configuration for them. Some of the things we ended up building to help with these problems were:

• Automation to update image versions in our repository, as and when new versions become available (continuous delivery all the way from merging an application change).
• Automation to promote image versions from one deployment environment (staging, production, etc.) to the next (progressive delivery).
• Graphical UIs to explain what versions of things were deployed in which environments.
• Templates for bootstrapping new services and new entire clusters.

At the end of the day, our configuration repository was a means to an end. No internal developer or platform team sets out with GitOps as their goal. GitOps is a process and set of tools to enable the end goal. The end goal is to build a platform that allows application developers to self-serve the infrastructure they need to deploy and manage their applications and to track those changes in something versioned and immutable.

The automation we built was designed to either make our configuration and Git more accessible, or hide the need to interact with it altogether. This usually involved adding APIs over the top of these repositories to materialize more meaningful insight, or automation to generate pull requests over GitHub with the desired change for the logical request (new service, updated image version, and so on).

Git is ubiquitous these days, and it has a lot of desirable properties out of the box (versioned and immutable). This is why GitOps is so appealing, but is Git the best interface we have to offer? Let’s face it, it is prohibitive to non-developer or less technical audiences. In my experience, failing to provide more meaningful abstractions around these problems to our engineers led to a couple of undesirable outcomes:

1. Engineers work around you to get what they want
2. Or, they fill your inbox with requests for you to do it for them

I want internal developer and platform teams to have to focus on their developer’s needs and for these manual processes to be automated and for these stakeholders to serve themselves.

How to Bridge the Gap?

We believe that we need to be building more meaningful APIs over Git. APIs that are designed to serve our stakeholder's needs:

• New service creation
• Getting a database
• Adding a feature flag
• Continuous deployment

Ideally, if you’re in one of these teams, you use a tool that helps you focus on these problems and not the necessary but distracting details of Git.

You should only need to focus on the business logic of parsing the current state, and transforming your repository contents from one revision into another, based on some new desired state. The tool should handle branching, adding, committing, pushing, and opening pull or merge requests for you.

You’re likely going to have all sorts of configuration formats and languages written into your repositories. This means the tool needs to be very extensible. There is no way a single project can support all the possible formats out there, given the nature of what can and is currently being stored in Git configuration repositories (YAML, JSON, Terraform/HCL, Jsonnet, CUE to name a few).

Beyond just automating away fiddly Git details, we’re going to want to consume these APIs with command-line tools and graphical user interfaces. This is how we reach those users that need a lower barrier to entry. They don’t want to interact with just a raw API as much as they don’t want to interact with configuration in a Git repo. But who has time to build these things? Most smaller to medium organizations don’t have the luxury of dedicating engineering time to these kinds of tasks.

Ideally, there will exist some off-the-shelf command line tools and user interfaces that can interact with any project built with our theoretical Git hiding framework or tool.

For this to work, the APIs should ideally present a declarative contract to consumers. This will greatly simplify the complexity of more generically applicable clients. Equally, the contract should be consistent across the extensible surface area that the tool can support. Finally, the API needs some capacity to be self-describing for generic tooling to present a meaningful interface over its extensible contents.

Does any of this seem familiar? The Kubernetes API server does this very well already.

Shoutout to Marko Mikulicic for helping to clarify this vision. I brought an early prototype to him and he suggested that the Kubernetes API server is a perfect fit for this problem.

Building Cup

Introducing Cup.

A few months back we set out to build a Kubernetes-like API server that facilitates reading from and proposing changes back to Git repositories. A tool with a declarative API for retrieving well-structured and consistent API responses. The API reflects the shape of the data you’ve configured inside of it, such that more general-purpose tools can be developed to access its state. Eventually, Cup should get you a lot of the way without needing to build too much yourself, while also giving you the escape hatch to build your solutions around it.

Consistent

The API is consistent from one resource kind to the next.

Retrieving, searching, updating, and removing should be the same for a service as it is for a logical feature or database. Retrieval, update, and removal require instances of each kind to be uniquely addressable (group, version, kind, namespace, and name). Search and cataloging require a taxonomy on which to organize and facet discovery (groups, namespaces, and labels).

Declarative

The API expresses resources through a logical representation of what is contained in the repository, as opposed to how it is stored. The how is defined inside controllers, where you can extend Cup to suit your layout and format requirements.

Controllers combine the state of the Git repository at a particular reference, with a desired state of a resource to calculate a difference to be proposed via a pull or merge request on a target SCM (GitHub, GitLab, Gitea, etc.).

Self-describing

Each resource has a structured schema, which can be requested via the API. This schema can be used by external tools to present meaningful interfaces or enable further automation to be built.

Where Do We Go from Here?

Cup is very much an experiment and very much in its infancy. We’re looking to see if it excites people, and what people want to start making with it.

Status Quo

Out of the box Cup comes with a simple configuration-driven controller for organizing the Kubernetes-like API resources as JSON in your repositories. This has limited scope for customizability. However, it works quite well for raw Kubernetes-style configuration.

The following video does a tour of Cup and the template controller configuring an application deployed with ArgoCD:

Additionally, Cup comes with a very extensible WASM-type controller thanks to the awesome folks building Wazero for Go. This controller takes programs compiled to WASM and WASI snapshot preview 1, with a documented command line protocol, and turns them into a controller engine for whatever you can dream up. All you have to do is read and write files to a local directory, Cup (and Wazero) will take care of the rest.

If you’re interested in seeing this in action, we developed a controller you can explore for Flipt feature flag configuration. This is a stepping stone we envision for creating a UI experience for Flipt over Git. The video on the Cup GitHub project README demonstrates the Flipt controller for Cup.

Future Vision

We’ve thrown around a lot of potential projects that could be built on top of Cup and Git in the future. Including but not limited to:

• Internal developer platforms
• Continuous delivery automation and observability
• Automatic dependency updates
• Progressive delivery
• End consumer self-service of infrastructure (let your customer's actions invoke configuration change)
• General purpose configuration API for your applications (why not read from Cup directly in your applications)

To get there we’re going to need to build some more features. Our current roadmap of ideas includes:

• Proposal status directly in the Cup API (see what desired state has been requested)
• Authentication and authorization (fortunately we have a consistent API to hang policies on)
• First-class clients in different programming languages
• Automatic replay of desired state changes as track branches progress (allowing for proposal to stay accurate as Git commits are merged ahead of your desired changes)
• Better management and packaging for controller implementations (support pulling controllers directly from OCI registries)

There is a lot to do to prove (or disprove) the idea, but the project is out there and we’re committed to exploring it. We could really do with your expertise in evolving this idea. Is there something you could automate with the help of Cup? We want to hear about it and to make Cup more appropriate for you.

Come check out the project at https://github.com/flipt-io/cup and open an issue or discussion. Alternatively, come speak to us in realtime on our Discord.