Tailoring the Key Vault Secret Manager
In a microservice based system we've been working on, we needed a way to manage and secure sensitive configuration details for the various parts of the system in a way that provided enough flexibility to accommodate both shared configuration items amongst services as well as specific configuration on a case by case basis.
For example, many of the microservices share the same database connection string in our dev testing environment, but we needed to be able to provide a dedicated database for specific microservices in order to segregate data and/or provide a higher performance database.
The Microsoft documentation for the Azure Key Vault Configuration Provider has an excellent example of using an implementation of IKeyVaultSecretManager here that we'll use as the basis for our solution today. This solution will work on ASP.Net Core 2.0 and up.
In our example we have a microservice based solution hosted in Kubernetes on Azure. Let's say we have the following 3 services:
Now let's say we have the NotesApi and VersionApi services use the same connection string since they share the same database. But the Identity service needs to use a separate database for security reasons. Because connection strings contain sensitive information (username and password access to the database server), we store them in Azure Key Vault and use the Configuration Provider to access them.
To manage this, we're going to build a custom IKeyVaultSecretManager - this is designed to take advantage of using configuration key prefixes to allow a service to pass in a specific prefix (e.g. identity-)so that it can retrieve configuration keys that only that service can use. Optionally, a service can also provide a version prefix as well (e.g. identity-100) - for instance, version 1.0.0 of a service may require a configuration item value in a particular format, but refactoring in version 1.2.0 requires a different format for the same configuration item. In this case, we could have two secrets in the Key Vault - one named identity-connectionstring and the other named identity-120-connectionstring. Version 1.2.0 of the service should pick up the latter while any other versions will use the former.
In addition, we can optionally enable a global prefix (g-) for keys that are intended to be accessible by all services. For more information on how this works, take a look at Use a key name prefix on the Microsoft documentation site.
Our implementation also needs to take into account the fact that because a microservice can access both global configuration items (using the g- prefix) as well as service-specific configuration items (e.g. using the identity- prefix) the possibility exists for an attempt to map the same configuration item twice. This will cause an exception as configuration keys need to be unique, so we need to take this into account.
Our PrefixKeyVaultSecretManager implementation has the following constructor:
We're passing in the KeyVaultClient instance as well as the url to our key vault so that we can access all the secrets in advance and allow us to choose the appropriate one. This is done in a private method called FindRelatedKeys which is responsible for loading secrets and returning any that are related to the secret that is passed in. A related secret is one that shares the same key with but with a different prefix to the one passed in.
You'll see here that we're calling the KeyVault api to retrieve all the secrets in pages, iterating through until we don't receive any additional pages of secrets. The reason we need to load all the keys here is so that we can make an informed decision about which one to load when the Load method is called by external code.
Once all the secrets have been retrieved, we iterate through them calling a method called GetBaseKey that takes the key for each secret and creates some metadata around it so we can later determine which secret to load based on the key and type (based on their prefix) - we'll use that information to compare against the requested secret later on in the Load method.
To determine which configuration secret can be loaded, we implement the Load method as follows, looking for related keys and determining which one to load based on a hierarchy - Version, then Service, and finally Global:
Once all that's in place, it's fairly trivial to retrieve the actual configuration value by implementing the GetKey method in a more or less standard fashion:
And there you have it - a hierarchical method of retrieving configuration secrets from Azure Key Vault for multiple services or apps, each with their own particular needs. You can find the full implementation along with supporting code and additional implementation details over at this public gist: https://gist.github.com/robertjf/7d9eb123cc238d818c4324d2c7704d13.
If you find it helpful or have some suggestions for improvement, drop us a line - we're always up for a discussion 😎