Generating metrics from Logs is a common task in any observability platform. We commonly use Vector’s Log to Metric Transform to accomplish basic metric generation, such as counting error logs or generating a gauge from http access logs. While easy to configure, this transform has some limitations that can be cumbersome to work with.
Since Vector v0.35.0, there is a new option in the Log to Metric transform called all_metrics, which allows you to convert structured log events into metrics. This enables you to use VRL (Vector Remap Language) to write custom code to generate the metrics. More information on the structure of a metric event can be found in the Data Model.
Example 1 - Logs to Many Metrics
When dealing with a stream of unstructured logs that we want to convert to metrics, we usually want to incorporate some level of parsing as well as conditional cases where we want to generate metrics for some logs, but not others. Traditionally this would be a mix of VRL to parse the logs, a Router transform to split the logs up, and at least one Logs to Metric transform to generate the metric. But now we can use VRL to do all of this and more.
Below is an example that will take in Apache common logs and generate a new metric for the size of bytes per request with a tag of status, in addition to a counter for any requests for a specific url path matching.
sources:
# Generate some demo Apache Logs
demo_logs:
type: demo_logs
format: apache_common
transforms:
generate_metrics:
type: remap
inputs:
- demo_logs
source: |
# Use a parser to turn the log into an object
parsed = parse_apache_log!(.message, format: "common")
# Initialize a list we can append to, which will be returned as the output
metrics = []
# Add bytes gauge for requests by status
metrics = append(metrics, [
{
"timestamp": .timestamp,
"namespace": "rapdev.demo",
"name": "count_by_status",
"tags": {
"status": parsed.status
},
"kind": "absolute",
"gauge": {
"value": to_float(parsed.size)
}
}
])
# Count of wp-admin requests by status
http_path = string!(parsed.path)
if downcase(http_path) == "/wp-admin" || starts_with(http_path, "/wp-admin/", case_sensitive: false) {
metrics = append(metrics, [
{
"timestamp": .timestamp,
"namespace": "rapdev.demo",
"name": "wp_admin_by_status",
"tags": {
"status": parsed.status
},
"kind": "incremental",
"counter": {
"value": 1.0
}
}
])
}
# Return the list as the output, which could have one or two events in it
. = metrics
# Convert our new Log events into Vector Metric events
convert_metrics:
type: log_to_metric
inputs:
- generate_metrics
all_metrics: true
metrics: []
sinks:
print:
type: console
inputs:
- convert_metrics
encoding:
codec: json
Example 2 - Conditional and Default Tags
When adding tags to metrics using the Log to Metric transform, we are limited to only simple templating for the values. And in cases where the template uses a property that doesn’t exist, a WARN is thrown and the tag is not added. Using the VRL method of generating metrics, we can handle these cases by adding default values or checking for the existence of the property before adding the tag.
sources:
# Generate some demo Apache Logs
demo_logs:
type: demo_logs
format: shuffle
lines:
- <87>2 2024-08-13T11:44:31.168-04:00 some.chase meln1ks 5612 ID243 - We're gonna need a bigger boat
- <158>1 2024-08-13T11:44:33.168-04:00 some.circle benefritz 6753 ID278 - Maybe we just shouldn't use computers
- <53>2 2024-08-13T11:44:07.990-04:00 we.gl - 8376 ID782 - There's a breach in the warp core, captain
transforms:
# If we want to use log_to_metric we must first parse the log
ex1_parse_logs:
type: remap
inputs:
- demo_logs
source: |
# Use a parser to turn the log into an object
.parsed = parse_syslog!(.message)
# This transform will generate a WARN when appname is not set
ex1_generate_metrics:
type: log_to_metric
inputs:
- ex1_parse_logs
metrics:
- type: counter
field: service
namespace: rapdev.demo
name: log_count1
tags:
service: "{{.parsed.appname}}"
severity: "{{.parsed.severity}}"
# Instead we can parse and generate the metric with VRL
ex2_parse_logs:
type: remap
inputs:
- demo_logs
source: |
# Use a parser to turn the log into an object
parsed = parse_syslog!(.message)
# Build the metric event
metric = {
"timestamp": .timestamp,
"namespace": "rapdev.demo",
"name": "log_count2",
"tags": {
# Add appname value as service tag and default to unknown when set
"service": string(parsed.appname) ?? "unknown"
},
"kind": "incremental",
"counter": {
"value": 1.0
}
}
# Optionally add the severity tag if it exists
if exists(parsed.severity) {
metric.tags.severity = string!(parsed.severity)
}
. = metric
ex2_generate_metrics:
type: log_to_metric
inputs:
- ex2_parse_logs
all_metrics: true
metrics: []
sinks:
print:
type: console
inputs:
- ex1_generate_metrics
- ex2_generate_metrics
encoding:
codec: json
Interested to learn more or need some help troubleshooting? Reach out to us at chat@rapdev.io
