This may sound familiar: you have been working on a new feature for weeks or months and excitedly release it to your users! Sadly, in the weeks following the release you notice a disturbing trend. A core metric that tracks user interest in your service is steadily decreasing!
User engagement with your service has decreased since your feature release. What to do? Do you roll back the feature update?
Since releasing the update, your product has changed through other deployments. Your users have changed. The world has changed. Can you be confident that the initial feature update is the root cause? Unless you release your feature update through an A/B test it may be impossible to determine if the update caused the metric change.