Chapter 10: Digital Analytics Methodologies and Problem-Solving

Recommendation validation checklist

A rigorous recommendation doesn't just present numbers: it specifies expected benefits, risks, hypotheses, and confidence limits.

Verification of 10 robustness criteria

Use sliders to assign a score of 1 to 5 for each validation criterion.

01. Alignment with a business objective
Is the business objective explicitly named and quantified (e.g., increase profitability, loyalty) rather than a simple technical indicator increase?
1 - Not addressed / Not discussed 3 - Mentioned briefly 5 - Completely aligned and quantified
Level 3
02. Clear presentation of evidence (Data)
Are raw data used for diagnosis presented with their collection and trust sources?
1 - Absent 3 - Partial data 5 - Complete data and cited sources
Level 3
03. Transparency on data uncertainty
Are measurement limits and uncertainties (tag blockers, 10% conversion discrepancies, identifier loss) mentioned transparently?
1 - Ignored (False precision) 3 - Briefly mentioned 5 - Documented with error margin
Level 3
04. Statistical rigor (Significance)
If the conclusion is based on performance difference (e.g. A/B test), have we verified and shared statistical significance and confidence interval?
1 - Not tested / Possible randomness 3 - Tested but not detailed 5 - Proven statistical rigor (p < 0.05)
Level 3
05. Facts vs Inferences Separation
Do we clearly distinguish what is an observable fact (e.g. 15% abandonment) from an inference (e.g. the user fled due to price) and a recommendation?
1 - Everything confused 3 - Suggested separation 5 - Strict logical structure
Level 3
06. Operational and technical feasibility
Has feasibility been validated with technical or development teams, avoiding proposing unrealistic changes?
1 - Not validated 3 - Informal exchanges 5 - Validated with dev team
Level 3
07. Effort/Impact report
Does the recommendation offer an estimate of required effort (time, resources, cost) versus expected business benefits?
1 - No comparison 3 - Rough estimate 5 - Rigorous ROI / Effort analysis
Level 3
08. Collateral risk analysis
Have we analyzed cannibalization risks (e.g. moving organic traffic to paid) or potential undesirable effects on other indicators?
1 - Ignored 3 - Summarily mentioned 5 - Evaluated with attenuation measures
Level 3
09. Formulation of concrete actions
Is the recommendation formulated as actionable advice (what to do, when, and how) rather than a vague statement?
1 - Passive observation 3 - Suggested action without roles 5 - Strict action plan (Who, what, when)
Level 3
10. Tracking and control plan
Does the recommendation integrate an indicator or post-implementation tracking method to verify if the expected impact materialized?
1 - No scheduled tracking 3 - Follow-up suggestion 5 - Structured post-test measurement protocol
Level 3

Robustness index

60 On 100
Fragile Recommendation

Weak points to optimize:

References and academic sources