Support Usage Mode in new Trace Explorer to make it easier to track Span Consumption

[ankit-prajapati-26]

Environment

SaaS (https://sentry.io/)

What are you trying to accomplish?

I have following two questions

Need clarity on span calculation and how is counted for costing

I am trying to evaluate the overall span usage by different category and operation for specific time period,
Here I am checking for the last 30 days.

As you can see on the traces pages over overall span count look going >9M (by doing sum of all span categories) while when I see it on the Stats dashboard it shows Span usage as 3.5M.

I don't understand how span is calculated here, which span are getting calculated in costing.

As you can see there are multiple span are getting created with mark, measure, ui.long-animation-frame Is this getting used in any metric evaluation of insights and can I discard this with tracingSampler to save a span usage ?

How are you getting stuck?

Yes, We are planning to introduce Tracing and Performance monitoring on all product lines, but we are not able to calculate the total costing and usage due to this discrepancy

Where in the product are you?

Performance

Link

No response

DSN

No response

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-performance for triage ⏲️

[Dhrumil-Sentry]

Hi @ankit-prajapati-26 - the Explore->Traces UX you are using is currently available to early adopters only and it shows extrapolated metrics for your spans that account for client side sampling. Please see these docs

The stats page is the source of truth for tracking what data you are sending us and being charged for

In the future we plan to allow a setting to turn off extrapolation so you can use use the Trace Explorer to unextrapolated metrics - but we're likely a few months away from that

[ankit-prajapati-26]

@Dhrumil-Sentry

• I was unable to find any method to display span usage per individual product within the custom dashboard.
• This metric is essential to:
  • Monitor span consumption per product.
  • Determine whether any proactive measures are required, such as adjusting the sampling rate.
• Additionally, we would like to set up alerts for span usage limit breaches at the project level.
• Requesting your guidance on:
  • How to achieve this via the custom dashboard.
  • Whether any specific parameter or workaround exists, as I could not locate any relevant option.

Your assistance will be appreciated.

[Dhrumil-Sentry]

Hi - If product is a project then you can group the count of spans by project, if it's a custom attribute you need to ensure its attached to all your spans and then you can group the count of spans by that attribute

[bcoe]

@ankit-prajapati-26 just following up regarding your other question:

As you can see there are multiple span are getting created with mark, measure, ui.long-animation-frame Is this getting used in any metric evaluation of insights and can I discard this with tracingSampler to save a span usage ?

We do not currently use mark, measure, or ui.long-animation-frame in the insights modules, so you should feel free to filter these out for the time being.

You should be able to filter these out with beforeSendTransaction, like so:

  beforeSendTransaction: event => {
    event.spans = event?.spans?.filter((span) => {
      const skippedOps = [
        'mark',
        'measure',
        'ui.long-animation-frame'
      ];
      return !skippedOps.includes(`${span.op}`);
    });
    return event;
  }

Let us know if @Dhrumil-Sentry and my help has unblocked you 👏

[ankit-prajapati-26]

@Dhrumil-Sentry @bcoe
There is a difference in span usage count in the custom dashboard I am creating and on the stats page.
As I have mentioned here, We wanted to track span usage at project level in custom dashboard, also we wanted to setup alerts at project level for same.

Why this numbers are different ? how can I show the actual span usage in custom dashboard and how can I use the same in setting up alerts ?

[Dhrumil-Sentry]

Hi, As I mentioned in my previous reply #89266 (comment) the Explore->Traces UX shows aggregates that are extrapolated based on your sampling rates.

The only workaround right now would be to create alerts based on extrapolated volumes - The extrapolated aggregations are a true representation of your tracing traffic

There is no way to turn off extrapolation today and do what you want which is mainly quota volume tracking - This is something in our backlog that we plan to enable in the future