Chris Maier
Jun 2, 2022

Automating our Infrastructure with new Pulumi Providers

As a product, Grapl puts code as a front and center capability. By allowing our users to write Python for their queries we allow them to leverage constructs like classes, functions, comments, as well as libraries that can be shared with others. Using real programming languages unlocks a lot of power, and we take that approach where we can.

Another area where we use code is to automate our infrastructure and deployment tasks using Pulumi. When we started using Pulumi over a year ago, it was just to set up the infrastructure needed to run the Grapl service. The more we've used Pulumi, however, the more we've used Pulumi. We now use it to manage our Github repositories, our common Vault server cluster, developer "workstations in the cloud", and even our continuous integration pipelines. The broad selection of Pulumi providers available makes this easy to do; chances are there's something there to automate whatever you need to automate.

But sometimes, there's not a Pulumi provider available. This has happened to us a handful of times on our "Pulumi journey". In such cases as this, you can try to write your own provider from scratch, or take advantage of Pulumi's code generation libraries to bootstrap a provider based on an existing Terraform provider. This allows you to take advantage of the extensive engineering efforts that have gone into the creation of these Terraform providers, and expose it to Pulumi consumers in a natural way. Given the popularity and longer history of Terraform there’s a huge ecosystem available to tap into.

Several months ago we took this ‘Terraform provider’ approach to create the beginnings of the official Nomad Pulumi Provider, which we subsequently donated to Pulumi. This worked well (and we now use this provider daily in our own code), but was ultimately not a scalable approach; after all, if the Pulumi organization has to be the gatekeeper for all Pulumi providers, they quickly become a bottleneck, which is not good for them or for the broader Pulumi community.

This changed in October 2021 with the announcement of the Pulumi Registry. This registry collects official Pulumi providers, as well as community-provided providers, in a way that removes the Pulumi organization as a bottleneck. We reached out to our contacts at Pulumi to become "guinea pigs" for their self-serve provider publication process. After some hand-holding, we were able to publish two new providers to the registry: the Hashicorp Cloud Platform provider and the Buildkite provider. Both providers are based on their respective official Terraform providers (hcp and buildkite).

The process to create and publish the providers is relatively straightforward (and getting simpler, as Pulumi works to streamline things). The fact that we're able to build and publish the providers on our own, without having to be dependent on Pulumi engineers to shepherd everything through, is very freeing, allowing us to work at our own pace, while making it easy for others to take advantage of our efforts.

We're happy to provide these resources to the broader Pulumi community, and welcome any contributions and feedback from users.

Thanks to the authors of the original Terraform providers we based our Pulumi providers on, as well as to the wonderful engineers at Pulumi, particularly Paul Stack, whose help was pivotal in getting all this set up!

Interested in our product? Check out our Github. Reach out for a demo!

Connect with us on the Discord and Slack - we'd love to answer any questions you may have about our product.