Find actionable usage information and learn how to improve PPS.

How much your company pays for using Tanzu Observability by Wavefront depends on PPS (point per second) consumption. What you pay is based on data throughput, primarily:

  • Data ingestion. When the Wavefront service ingests data, those data count toward your PPS allocation.
  • Data analysis. When you run a query, either as part of an alert or when you look at a dashboard, the backend has to process the data, and those data points count toward your PPS allocation.

This page helps you get value from your PPS and avoid overage. Overage is an extra fee that customers with certain contracts pay if they consume more data than they paid for.

Where Do I Start?

Each customer has a contract with VMware that allows them to send a predetermined amount of data, measured in points per second (PPS), to their Wavefront instance.

If the customer uses more than the contracted rate, VMware bills for those additional data. Because VMware has to pay the cloud provider for data consumed by each Wavefront instances, we have to ensure that customers pay for the data they consume. But we want for you to get the largest amount of useful information possible from your data. This page has some tips.

How Do Find Usage Information?

Your Wavefront instance includes out-of-the-box dashboards and charts that help you determine how close you are to your contracted rate and allow you to explore remediation options.

If you're a SuperAdmin user and you want to know how close you are to exceeding your committed rate and whether you will be billed for overages, you can use the Usage Summary dashboard.
  1. Click the gear icon in the top right corner and select Usage Portal
  2. Examine the charts on the dashboard.
screenshot of usage summary dashboard
If you want to drill down into usage and pricing information, you can examine the dashboards in the Wavefront Usage integration. Your contract with VMware determines which dashboard has the information you need:
  • Billable: Most users need the information on the Committed Rate vs Monthly Usage (PPS P95) for Billable dashboard.
  • Burndown: A few customers have a burndown billing contract. For those customers, the Usage (PPS) vs Remaining Balance (PPS P95) for Burndown has the information they need.
screenshot of the 5 Wavefront usage dashboards

How Can I Make Improvements?

You can find information and improve PPS in several ways. The following table gives a summary.

Action ItemDescriptionMore Info
Find areas with high PPS and take action If your Wavefront instance ingests high-cardinality data or if you scan a lot of data during query processing, you'll get high PPS usage. Query processing happens:
  • When a dashboard renders data.
  • When an alert is checked.
How can I optimize my ingestion rate?

How can I optimize my scan rate?
Ingest only data that you use Datapoints that are sent in contribute to the PPS, so it makes sense to ingest only what's used, to reduce granularity, etc. It might also be necessary to find the teams who send data and don't use them. Ensure that someone benefits from all data that's sent in. How can I optimize my ingestion rate?

Who's Responsible for Ingested Data?
Remedy high PPS To remedy high PPS consumption, you can optimize the ingestion rate and the data shape.
  • Be smart about queries by using filters and time windows.
  • Ensure optimal dashboard performance, for example, by limiting the data each chart displays.
Optimizing the Data Shape to Improve Performance

Optimize Query Performance

Ensure Optimal Dashboard Performance
Find teams with high PPS If several teams use the Wavefront service, explore which team sends in most data. Who's Responsible for Ingested Data?

How Can I Optimize My Ingestion Rate?

Billing for Tanzu Observability is based primarily on the ingestion rate. Ingestion is so important because data is ingested again and again. Tanzu Observability supports ingestion every second for metrics. Here’s how you can look for ways to optimize and reduce your ingestion rate.

Examine Ingestion with the Namespace Usage Explorer Dashboard

The Wavefront Namespace Usage Explorer dashboard can help you pinpoint where data are coming from.

  1. Log in to your Wavefront instance.
  2. In the toolbar, select Integrations, and click the Wavefront Usage integration.
  3. Select the Dashboards tab and click the Wavefront Namespace Usage Explorer dashboard.

This dashboard provides not only a current view but also a historical view. Start at the level 1 namespace to identify the top level. Then dive into levels 2 and 3 for finer-grained information to answer these questions:

  • How many different namespaces do I have at each level?
  • What are my top namespaces?
  • Which are the top 10 metric namespaces and trends over time at each level?

The screenshot below shows an example from our demo server. The data is prefixed with the data source. In the pie chart, you can see that the top namespaces include pcf and kubernetes.

Metrics namespace dashboard screenshot

The Namespace Usage Explorer dashboard is especially useful if your metrics use hierarchical name spaces of up to 3 levels that identify who sends which metrics. For example, some of our customers use namespaces that show the Business Unit (Level 1), team (Level 2), and data source. For example, you might have monitoring.dev.kubernetes and monitoring.sales.kubernetes for Kubernetes data coming from the dev and sales teams in the monitoring Business Unit.

(Optional) Clone Namespace Explorer and Create Custom Charts

If you don’t see the information you need, for example if you need a chart that shows histogram ingestion, clone the Namespace Usage Explorer dashboard and modify existing charts or create custom charts. You can use cs() queries to retrieve information about counters, histograms, and spans. For example, the default dashboard examines ~metric information, but you can also examine counter information using the following format:

cs(~<data_type>.global.namespace.<namespace>.pps, source=<depth_number>)

Here’s an example query that returns the top 10 Level 1 metrics:

rawsum(align(1m, taggify(cs("~metric.global.namespace.*.ppm", source="depth_1"), metric, Name, 3)), Name) / 60

Drill Down with Wavefront Top and Wavefront Spy API

If you need more than 3 levels of namespaces or if the dashboard doesn’t answer your questions for other reasons, Wavefront Top and the Spy API show in detail what’s happening right now.

  • Wavefront Top lets you examine which ingested metrics were accessed during the last lookback period.
  • The Wavefront Spy API also supports delta counters, histograms, spans, and span logs.

With Wavefront Top you can:

  • Use the PPS column to examine ingestion rates.
  • Use the %Acc. column to examine the percentages of the ingested rates that are accessed by queries.
  • Dive into deeper levels of the namespace than with the Namespace Usage Explorer dashboard.
  • View ingestion rate by source, point tag, or ingestion source.
  • See what percentage of currently ingested data within a namespace is actually accessed in queries over X days. The number of days defaults to 7 and is configurable.
  • See what range of values is sent in for a namespace.
  • See the data lag for a namespace.

The Wavefront Spy API gives even more detail, but the information in Wavefront Top is usually sufficient.

Consider Sending Data Less Frequently

Even though Tanzu Observability supports per second granularity for metric data points, data rarely needs to be that granular.

  • For data that does not need to be that granular, increase the reporting interval for significant savings. For example, switching from a 1-second interval to a 1-minute interval results in a 60x reduction in the ingestion rate for that set of data.

  • Values that do not change often (constants) are great candidates for increasing reporting intervals.

    You can use Wavefront Top to uncover constant values. The Range column shows the range of the reported values (the maximum value minus the minimum value) for each namespace.

    • If the range is 0, then this data set is most likely reporting constant values.

    • If the range does not change, it is also possible that only a few fixed values are reported. The data set is a candidate for higher reporting intervals.

How Can I Optimize My Scan Rate?

Each time a query is executed, the points that the query engine looks at are counted toward your PPS.

  • If you examine a dashboard, all queries on the dashboard are executed. What precisely that means for your PPS depends in part on the time window the dashboard uses.
  • When an alert is checked, the alert queries are executed.

If you ensure that your environment doesn’t include unused dashboards or alerts, you can significantly improve your scan rate.

Examine the Data Scan Rate with the Wavefront Usage Integration

Data points are usually scanned because an alert is checked or a dashboard is displayed or updated. You can get a high-level view in the Wavefront Usage integration.

  1. Log in to your Wavefront instance and select Integrations on the toolbar.
  2. Select the Wavefront Usage integration.
  3. On the Dashboards tab, select the Wavefront Service and Proxy dashboard.
  4. Examine the Data Scan Rate and Data Scan Rate by User chart.
  5. Click a chart to temporarily change the chart type (for example to topK) or the query. To permanently modify the chart, clone the dashboard.
Data scan by user.

More Info

Examine Points Scanned in the Alerts Browser

Even if an alert has no recipients, the alert query is executed at the predefined Alert Checking Frequency (1 minute by default). To significantly reduce the scan rate:

  • Delete or snooze all alerts that aren’t being used.
  • Examine the alert query (or queries) for alerts with high point rates and optimize query performance with filters and other strategies.
  1. Log in to your Wavefront instance and select Alerts >All Alerts
  2. To order the display by points scanned, select Points Scanned from the menu in the top right.
  3. The alerts are ordered, and each alert shows Points below the query.
  4. For alerts with a high number of points scanned, examine the query and checking frequency to find ways to improve performance.
Menu option starts with Default, shows Points Scanned.

Optimize Query Performance

You can significantly reduce the load on the query engine by changing your query. For example, use filters to minimize the search space, include a time window with certain operations, and more.

More Info

Consider Histograms to Improve PPS

Histograms store data as distributions rather than as individual data points. For billing purposes, the rate of distributions ingested is converted to a rate of points ingested through a conversion factor, which is 7 by default.

More Info

See How Histograms Can Improve PPS for an example.

Use the Wavefront API to Compare Ingested and Accessed Metrics

Create a script that uses the Access API category to check how often an entity has been accessed. Supported entities are metrics, histograms, and spans. The default lookback period is 7 days but you can configure it up to 60 days.

Start with metric namespaces that contribute the most to the overall ingestion rate.

  1. Create a script to determine all of the metric names within those namespaces.
  2. Feed each of those metric names to the Access API. Focus on specific namespaces one at a time to avoid overload and get actionable information.

Tip: There is an underlying (undocumented) API that the Metrics Browser uses. To take advantage of that API, use your browser’s developer tools to see the underlying API calls.

More Info

Access API category

Who’s Responsible for Ingested Data?

You can find out which teams are using the most data with ingestion policies or from the Namespace Usage Explorer dashboard.

Action ItemDetailsLinks & Examples
1. Use Ingestion Policies. Create an ingestion policy for each team and monitor the usage per team. Finding Ingestion and Query Problems
2. Examine Ingestion with the Namespace Usage Explorer If many teams in your company use Wavefront, it’s good practice to have a policy that includes the team name in each ingested data point. You can then easily determine which teams ingest a lot of data. Examine Usage with Ingestion Policies

FAQs About PPS

You want to know...You can...For example
What are the usage trends for my instance? Use the charts on the Wavefront Ingestion Policy Explorer dashboard or the dashboards in the Wavefront Usage integration. data ingestion points
Who is responsible for high usage? Use the Wavefront Ingestion Policy Explorer dashboard in the Wavefront Usage integration to drill down into accounts and find the accounts responsible for the high ingestion.

Use the Wavefront Namespace Usage Explorer dashboard in the Wavefront Usage integration to drill down into namespaces and find where the ingested data is coming from.
usage by account
Top 10 Level 1 Metric Namspaces
What are current usage details? Start with the dashboards in the Wavefront Usage integration. The dashboards allow you to look at both current usage and usage over time. If you need to drill down more, use the Wavefront Top GUI for a detailed view of current spikes. Data Scan by User, with hover showing users
Why do I have cardinality problems? If you know there's a spike in data (for example, an ID burst), you can use Wavefront Top or the Wavefront Spy utility during the spike to drill down into individual data points and tag values. Those two tools aren't as useful after the spike -- use the Wavefront Usage dashboards instead. example screenshot from wftop Github page
How much data is coming from my source? You have several options:
  1. Use the Source Browser for a first look.
  2. Look at the last section in the Wavefront Service and Proxy Data dashboard to see the top sources listed by PPS.
  3. Use the Wavefront Top GUI to filter by source and examine what's coming from the selected source.
Sources rate screenshot