Generating Datadog Events Using Vector

Don’t miss critical events in your logs by using Vector to send them to Datadog as Events

2

min read

|

by

Aaron Rhodes

September 16, 2024

Vector is a popular observability pipeline tool that can be used to transform data as it traverses your ecosystem. Commonly we use it to filter logs or convert those logs to metrics, but recently we have started using it to turn logs into Datadog Events.

Compared to logs, Datadog Events have a more rigid object structure which enables Datadog to use them for features like Watchdog and Event Correlation. They are best suited for sparse events, like when an application is started or stopped, a new configuration is loaded, or an unexpected fault that results in application crash and/or restart. Datadog includes 500 events per host per month as part of its Pro plan and 1,000 events per host per month as part of its Enterprise plan.

Critical events like these are often buried in log files that are filled with other verbose messages. With Vector, you can watch for specific log events and generate a Datadog Event with the relevant Title, Status, and Service tag. These events can then be used for Monitors, Dashboard Overlays, or Triggering Workflows.

‍

Demo

Here is an example where we use Vector to watch our logs for three specific messages and publish an event into Datadog with varying status levels.


sources:
  demo_logs:
    type: demo_logs
    format: syslog

transforms:
  logs_to_events:
    type: remap
    inputs:
      - demo_logs
    source: |
      msg = to_string(.message) ?? ""
      if contains(msg, "We're gonna need a bigger boat") {
        . = {
          "title": "Demo service is starting",
          "text": .message,
          "tags": ["service:" + .service ?? ""],
          "source_type_name": .source_type,
          "host": "demo-host",
          "alert_type": "info",
        }
      } else if contains(msg, "what just happened") {
        . = {
          "title": "Demo service has unexpectedly restarted",
          "text": .message,
          "tags": ["service:" + .service ?? ""],
          "source_type_name": .source_type,
          "host": "demo-host",
          "alert_type": "warning",
        }
      } else if contains(msg, "There's a breach in the warp core") {
        . = {
          "title": "Demo service has crashed",
          "text": .message,
          "tags": ["service:" + .service ?? ""],
          "source_type_name": .source_type,
          "host": "demo-host",
          "alert_type": "error",
        }
      } else {
        # Returning an empty array output will drop all other log events
        . = []
      }

sinks:
  print:
    type: console
    inputs:
      - logs_to_events
    encoding:
      codec: json
  datadog_events:
    type: datadog_events
    inputs:
      - logs_to_events
    default_api_key: ${DD_API_KEY}

With Vector's powerful transformation capabilities, you can seamlessly convert critical log data into actionable Datadog Events. Try it yourself and see how it can enhance your monitoring and response efficiency.

Have questions or want to learn more? Reach out to our team at chat@rapdev.io to learn more.

Written by

Aaron Rhodes

Boston, USA

A Boston-based expert in building and monitoring large-scale applications, with extensive experience utilizing Datadog for performance optimization and system reliability.

More by

Aaron

Advanced Logs to Metrics Using Vector

Oct 2024