Measuring Success
Context
We have developed a solution to address a common challenge for all our customers: mitigating risks around their pipelines. However, customers may perceive the quality of our analytics differently.
Several factors can influence this perception, including the varied geographical environments that require fine-tuning of our machine learning model and QA processes—for example, the contrast between arid desert landscapes in the Middle East and lush, forested regions in Northern Europe.
Geographies
The level of interest can vary based on the type of detected change. For instance, one customer might prioritize insights on agricultural or plowing activities differently than another.
We need your feedback!
We have showed you in the Insight review section that you could evaluate the quality of an Insight:
This is essential to fine-tune the analytics so you can prioritize the risks that matter most to you, minimize false positives that create operational overhead, and assess whether SurfaceScout could become a valuable information source (we think it is! But we need your data to confirm it ;)).
We typically track 3 key metrics:
Precision: The percentage of actual changes detected out of all predictions delivered.
Relevance: The percentage of relevant predictions out of all predictions delivered.
Uniqueness of Information: The number of unique detections not captured by other methods (like helicopters), compared to everything we’ve detected.
Uniqueness & Relevance combined: The number of unique and relevant detections compared to everything we've detected
With your feedback, these metrics tend to improve significantly—often reaching over 80% relevancy, for example after a few deliveries. Without your input and domain expertise, however, results are less reliable (see the orange section for details).
For Precision, which measures the rate of false alerts, we typically begin around 90%.
For Relevance, we usually start lower, around 40%. This is where your feedback makes a significant impact: after 4-6 deliveries, we often achieve scores above 80%!
For some customers, we can directly correlate detected Insights with tangible risk reduction in financial terms (€ or $). Recently, with a customer, we were able to reduce risk exposure by 4M€ annualy for every 1,000 km of pipeline!
Last updated