_
_
Datadog has 600+ out-of-the-box integrations that cover a variety of technologies from web servers to databases, to 3rd party SaaS services. For many of these integrations, there are agent configuration files that require storing credentials for the technology. The larger issue is around how to store those credentials. Many security-minded engineers would prefer to not store those secrets in plaintext in case of unauthorized access to their servers or shared access to Datadog configuration. What not everyone knows is that the Datadog agent has a mechanism to run an executable at agent startup in order to reach out to a secrets management tool of your choosing and decrypt those secrets, storing them in memory for use by the agent.
If you want to utilize Datadog’s secrets management capabilities, there is a specific notation that is recognized by the agent. Let’s take the Datadog MySQL integration as an example. While the integration by default only collects information from performance-related tables, you might grant additional access to other database tables in order to ingest more business specific metrics into the platform via custom queries. This may require additional permissions to more sensitive data, so you might want to make sure that credential is not stored in plaintext in the integration configuration. To do this, Datadog utilizes the ENC[] notation in configuration files to tell the agent that the credential is one that should be decrypted from a secret backend. Below is a sample of what that might look like:
Now the question becomes, “how does the agent know to decrypt this secret?”
Within the datadog.yaml file there is a configuration parameter called secret_backend_command. This parameter is the path to the executable file that the agent uses to perform the calls to the secret backend to decrypt the secret and store it in memory. You can also use the secret_backend_arguments parameter if the executable requires arguments to be passed to it. The agent also requires specific permissions and attributes that must be assigned to the executable to ensure additional security:
Linux
Windows
The agent also has an expected output in order to determine the values of the secret(s) such as mysql-prod-password in the example above. This output would look like that below:
If the “error” field is anything but null, the integration configuration that uses this secret is dropped and will not function. For some, writing a custom executable may be out of their skillset or you may just not have time to do so. Luckily, the engineers at RapDev have developed a utility to help with the Datadog secrets management problem.
RapDev has developed an open-source utility that we appropriately call datadog-secret-backend. This utility is a simple Go executable that supports a few different secret backend tools to help lighten the burden of writing a custom executable for use within your Datadog agents. It currently supports the following backends:
The datadog-secret-backend utility is quite extensible, and even supports multiple secret backends in a single configuration. This way, users can decrypt multiple secrets stored in different backends within a single configuration file. Below we’ll use AWS Secrets Manager and Azure KeyVault to show some example configuration.
The executable expects the configuration file above to be stored in the same directory as itself, but this can be overridden with a config parameter passed to it. While there are technically secrets stored within this configuration file as well, it is important to note that the contents of this file should only be readable by administrators of the secret backends. This ultimately helps hide secrets from those that have access to the Datadog integration configuration files, as the permissions on those are often more open.
With the above configuration, the agent is now ready to decrypt secrets, with just a minor change to the MySQL integration configuration. In order for the executable to know which secret backend to utilize to decrypt this secret, we prefix the secret_id with the ID of the backend as follows:
After adding the necessary secret_backend_command configuration to the datadog.yaml file, restart your Datadog agent. You can verify that the secrets are being decrypted successfully by running the agent’s built in secret command.
Once you confirm that the secrets are being decrypted appropriately, you’re ready to go! If a secret backend your organization utilizes is not supported or you have issues with any of the current secret backends the utility supports, feel free to open a GitHub Issue and one of our engineers will take a look. You can find all documentation related to each of the supported secret backends in the repository’s README. We hope this helps you keep your secrets hidden and eases some of the burden of custom tool development for your team.
