Learn how to write events() queries.

Summary

events(<filterName>="<filterValue>” [,|and|or [not] <filter2Name>="<filter2Value>"] ...)

Controls how events display in a chart. Can be combined with event set operators and event functions that let you fine tune your events() query.

Parameters

ParameterDescription
filterName An event filter that lets you limit which events to display. You can optionally specify multiple event filters combined with Boolean operators.
filterValue Value accepted by a specified filterName.

Description

The events() function returns the set of stored events that match one or more event filters. You can use events() to display events in time-series charts, for example, to inform other users about reasons for a change in a time series. The result set of an events() query depends on the timing of the query.

You can describe the set of events to be returned using event filters:

  • Match events in general by filtering on name, source, type, severity, and event tags.
  • Match events that are associated with alerts by filtering on alert tags, targets, and alert ID.

You can combine multiple event filters with Boolean operators. For example:

  • Return events of any type that start with Request and have severity that is either severe or warn:
    • events(name="Request*" and (severity="severe" or severity="warn"))
    • events(name="Request*", severity="warn" or severity="severe")
  • Return events of any type that either have severity warn or are generated by the source app-1:
    • events(severity="warn" or source="app-1")

You can use events() as an eventsExpression that you specify to another event function or to an event set operator. For example:

  • Filter a set of events and return only the events that have ended: closed(events(name="Request*"))
  • Examine a set of events and return the single event with the earliest start time: first(events(name="Request*"))
  • Use a set of events as the basis for creating synthetic ongoing events: since(events(name="Request*"))
  • Count the number of events at each point in time and return a time series: count(events(name="Request*"))
  • Return all ongoing events: events(name="Request*") - closed(events(name="Request*"))

Event Filters

Event filters allow you to limit which events are returned from events() queries.

FilterDescriptionExample
alertId The ID of the alert that created the event. events(alertId=1411189741192)
alertTag A tag associated with the alert that generated the event. events(alertTag="ops")
eventTag A tag associated with the event. events(eventTag="codepushes")
name The name of the event. Manually created events have a unique name, while events generated by an alert have the same name as the associated alert. The name filter requires quotes if spaces exist in the name. events(name="Request Latency too high")
severity The classification of the user event or the severity of alert that generated the event. User event classification levels are severe, warn, info, and unclassified. Although an event can be left as unclassified, the severity filter does not accept the string “unclassified” as a valid value. events(severity="info")
source or tag The source or source tag associated with the alert that generated the event. The source filter allows you to display events generated by an alert based on a single source or set of sources. The tag filter works the same way, but allows you to specify a source tag instead of a source name. events(source="app-*" or tag="dc2")
subtype The subtype of event of type alert-detail: failing, recovered. events(subtype="failing")
target The target of the alert that generated the event. The list of targets associated with an alert are considered a single string. If you want to identify a single target within that string, then you must use wildcards. For example, if the notification field contains: john.doe@example.com, jane.doe@example.com, pd:fbw21c9ee0219473w2179r4t23f8c34, to use jane.doe@example.com as the target filter, specify the email as \*jane.doe@example.com\*. events(target="*jane.doe@example.com*")
type The type of an event. The system-generated event types are:
  • alert
  • maintenanceWindow
  • alert-detail
  • credentials-error
  • alert-created
  • alert-updated
  • alert-deleted
  • alert-undeleted
  • alert-snoozed
  • alert-unsnoozed
  • dashboard-deleted
  • maintenancewindow-created
  • maintenancewindow-updated
  • maintenancewindow-deleted
You can optionally assign a type to a user event. The value requires quotes if it contains spaces or starts with a wildcard.
events(type="Code push")
events(type="*-created")

Event Set Operators

An event set operator combines or compares two sets of events, and returns a set of events as the result.

Event Set Combination Operators

Combination Operator Definition
<e1> union <e2> Returns all events that exist in either of the event sets. For example, these queries return events of type maintenanceWindow (with any name), and events named test (of any type):
events(type="maintenanceWindow") union events(name="test")

events(type="maintenanceWindow" or name="test")   // Equivalent alternative 
<e1> intersect <e2> Returns all events that exist in both of the event sets. For example, these queries return events of type maintenanceWindow that are also called test:
events(type="maintenanceWindow") intersect events(name="test")

events(type="maintenanceWindow" and name="test")  // Equivalent alternative 
<e1> - <e2> Returns the difference between two event sets. For example, this query returns all ongoing events:
events() - closed(events()) 

Event Set Comparison Operators

You can use Allen’s interval algebra operators to compare two event sets, based on the time intervals (start time and end time) of the events in each set.

Event Set Operator Definition
<e1> < <e2>
<e1> > <e2>
Returns the events in e1 that occurred entirely before (or entirely after) the events in e2.
<e1> m <e2>
<e1> mi <e2>
Returns the events in e1 that finished just as events in e2 started.
<e1> o <e2>
<e1> oi <e2>
Returns the events in e1 that overlap with events in e2.
<e1> s <e2>
<e1> si <e2>
Returns the events in e1 that start at the same time as events in e2.
<e1> d <e2>
<e1> di <e2>
Returns the events in e1 that occurred during events in e2.
<e1> f <e2>
<e1> fi <e2>
Returns the events in e1 that finish at the same time as events in e2.
<e1> = <e2> Returns the events in e1 that start and finish at the same time as events in e2.

Examples:

  • events(severity="severe") d since(1d) returns all events with severity severe that occurred in the last day.
  • events(severity="severe") - (events(severity="severe") d since(1d)) returns all events with severity severe that are older than one day.

When Does an Event Query Return Events?

Where an event happens in relation to the query start time and query end time determines whether a query returns an event or not. Returning an event means showing the event in the UI, or, if you use the API, returning the event itself. The following diagram illustrates the behavior:

when events return

Here are the details:

Event Number Event start Event end Returned?
Event 1 Before query start time time Before query start time No
Event 2 After query start time Before query end time Yes
Event 3 Before query start time After query start time Yes
Event 4 Before query start time After query end time No
Event 5 Before query start time N.A. (ongoing event) Yes
Event 6 After query start time After query end time Yes
Event 7 After query end time N.A. (ongoing event) No
Event 8 After query start time N.A. (ongoing event) Yes
Event 9 After query end time After query end time No