_
_
As organizations grow, so do their demands, and while sometimes you have the tools necessary to embark on a task, quite frequently you might find yourself having to customize your own toolset.
Datadog Monitors make it easier to have observability into your infrastructure by watching metrics and generating custom alerts that respond to your threshold specifications through notifications, an essential component of the monitoring process.
Datadog provides its convenient API tool to manage monitor notifications during planned outages and maintenance. But sometimes you would want to dynamically suppress unnecessary alerts during specific timeframes. For this, we could create a stand-alone Python script that can pull parameters from a spreadsheet file and that enables you to globally exclude scopes from alerting. Even better: the file can have multiple rows and create as many downtime calls as necessary.
First off, you will need to create a file that you can store locally or inside an Amazon S3 bucket, for example. After that, we will proceed to generate columns that correspond with the parameters set for the Schedule Downtimes API call, eg:
Datadog has excellent documentation on how to structure the script for the api call and the parameters for this in their Downtime documentation page. (Keep in mind that building and using the API client library requires Python 3.6+. )
In the CLI, install the API client library using `pip install datadog-api-client`. Then, in your code import relevant libraries and its dependencies to hit the selected endpoint (monitors_downtime):
Now, proceed to set up the configuration for the Datadog API. A great idea would be to keep a .csv file with the output from the generated downtimes. You can specify so by importing the Pandas library and providing a path for the output file to be saved at. You will also need to generate an empty list that contains the final output dictionary so you can easily convert this into a Python DataFrame:
Subsequently, you will have to iterate through each index and row in the provided file. If the type of recurrence equals to weeks, then `week_days=[“Mon”,”Tue”,”Wed”,”Thu”, “Fri”]`; else, the type of recurrence will correspond with that of the specified type on your file. Make sure that the time frames are formatted in UNIX timestamp.
Set the api instance response to return a dictionary and assign the body of the Downtime as the one you previously defined in your code: `response = api_instance.create_downtime(body=body)`. From your output dictionary, generate a new dictionary containing the parameters you would like to see on the .csv file and convert it into a Python DataFrame:
Finally, you will need API authentication by providing both your Datadog API Keys and Application Keys. Provide the following command when running your custom script on the CLI:
`DD_SITE=“datadoghq.com” DD_API_KEY=“<DD_API_KEY>” DD_APP_KEY=“<DD_APP_KEY>” python3 “your_script.py”`
You can verify that your script works accurately, logging into your Datadog account in the Datadog UI and navigating to Monitors > Manage Downtimes
So there you have it: a convenient and efficient way to dynamically leverage and adjust Datadog API tooling to our customers needs. If you are interested in implementing Datadog, get in touch: chat@rapedv.io
