Predicting the Why

Netflix processes 500 billion user events per day Netflix Engineering, 2015. When a viewer pauses for thirty seconds before resuming a film, the system records the pause as an event, updates the engagement model, and moves on. It cannot determine whether the pause represented anticipation, confusion about the plot, or discomfort with the content. The signal is filed as a consumption datapoint, not as a window into the viewer's psychological state.

This gap - between what AI systems observe and what they understand - has commercial consequences. Forrester Research found that 73% of companies struggle to use behavioural data for anything beyond basic segmentation Forrester, 2022. McKinsey found that companies excelling at personalisation generate 40% more revenue than average McKinsey, 2021. The distance between those two findings represents a structural problem that has not been formally named.

The problem is not data volume or processing capability. The absence of a framework connecting observable behaviour to psychological states and motivational drives that generate it. AI systems are optimised to predict what users will do next, based on what they have done before. They are not built to understand why users are doing it in the first place.

The framework proposes the ABSD framework - Action, Behaviour, State, Drive - as a four-layer architecture for moving from what users do to why. The framework synthesises four established research traditions that each address part of this problem but have not previously been connected: Self-Determination Theory Deci & Ryan, 2000, affective computing Picard, 1997, just-in-time adaptive intervention research Nahum-Shani et al., 2018, and learning analytics Siemens & Baker, 2012.

2.1 The measurement gap

Contemporary AI systems are highly capable at Layer 1 - the observable action layer. Event logging, clickstream analysis, session tracking, and conversion funnel analysis are mature engineering disciplines. The systems that process this data operate at impressive scale. What they lack is the inferential architecture to move from action to meaning.

Wise and Shaffer noted that without theoretical frameworks connecting behaviours to underlying processes, analytics remains descriptive rather than explanatory Wise & Shaffer, 2015. Their critique, directed at learning analytics, applies equally to CX and enterprise AI. The field captures behavioural traces but lacks frameworks for interpreting what those traces mean psychologically.

2.2 The cross-tradition gap

Four research traditions address the problem of interpreting user behaviour in AI systems. Each provides a rigorous answer to a specific part of the question. None of them address it together. The literature synthesis for this study found minimal cross-citation between Self-Determination Theory, affective computing, JITAI research, and learning analytics Zawacki-Richter et al., 2019. Researchers in each tradition discover similar phenomena using different vocabulary, without building on adjacent work.

This fragmentation is not accidental. The four traditions developed in different institutional contexts - psychology, computer science, public health, and education respectively - with different methodological standards and publication venues. The cost of this fragmentation is redundant discovery and missed synthesis.

The ABSD framework proposes four layers of user understanding, each one deeper than the last. The layers are not independent categories - they form a generative hierarchy. From a generative perspective, Drive shapes State, State produces Behaviour, and Behaviour manifests as Action. AI systems must work in reverse, inferring upward from the observable layer to the motivational one.

3.1 Layer 1 - Action

Discrete observable events: clicks, page views, submissions, navigation paths, timestamps. What most AI systems currently capture with high fidelity at massive scale. Actions are the consequences of underlying psychological processes, not the processes themselves. Confidence: 96% fidelity in standard event logging systems.

3.2 Layer 2 - Behaviour

Temporal sequences across actions. Patterns in how users act over time: hesitation, thrashing, regression, acceleration, persistence, avoidance, and flow. Behaviours require aggregation across events - they cannot be detected from individual actions. The transition from Action to Behaviour is grounded in learning analytics methodology Baker et al., 2004. Confidence: 82% with appropriate temporal aggregation methods.

The consequence of stopping at Layer 2 without Layer 3 is documented at platform scale. Haroon et al. (2023) conducted a systematic audit of YouTube's algorithm using 100,000 automated user accounts and found that congenial recommendations intensify deeper in the viewing trail — a temporary viewing episode permanently overrides established interest history because the algorithm treats recent engagement patterns as the primary inference signal. Correct Behaviour detection without State weighting produces recommendation collapse Haroon et al., 2023.

3.3 Layer 3 - State

Momentary cognitive-emotional conditions: confusion, confidence, frustration, flow, anxiety, engagement. Not directly observable. Inferred from behavioural patterns using contextual weighting. Affective computing research achieves 70–79% accuracy in controlled settings Calvo & D'Mello, 2010. The same pattern can indicate different states depending on domain, user history, and temporal context - contextual modifiers are required before any signal-to-state inference.

State inference at scale is not theoretical. Heggli, Stupacher and Vuust (2021) analysed over two billion Spotify streaming events and found that listening behaviour reliably maps to five distinct psychological state profiles across the day, with temporal patterns stable enough for algorithmic inference. Spotify has progressively operationalised mood-state inference since 2013 Lowe-Brown et al., 2024. These systems constitute operational evidence that the Behaviour-to-State transition is achievable in production environments.

3.4 Layer 4 - Drive

The underlying psychological needs and goal orientations that shape how users respond when their states are challenged. Grounded in Self-Determination Theory's three basic psychological needs: competence, autonomy, and relatedness Ryan & Deci, 2017. Relatively stable over time. Drive inference requires longitudinal observation of how users characteristically respond to state challenges - not a single session, but a pattern of responses across multiple interactions.

3.5 Contextual modifiers

Signal-to-state mappings represent base associations in the absence of countervailing context. Three categories of contextual modifier qualify how behavioural signals are interpreted at every layer: domain context (industry, task type, interaction norms), user context (individual baseline, history, expertise), and temporal context (time of day, journey stage, recent system changes). Narayanan and Georgiou established that human behavioural expressions show individual and contextual heterogeneity - the same signal means different things in different conditions Narayanan & Georgiou, 2013.

Each ABSD layer is grounded in a specific research tradition. The traditions are not interchangeable - each provides a distinct methodological capability for a specific layer transition.

Self-Determination Theory grounds the Drive layer. Deci and Ryan's three basic psychological needs - competence, autonomy, and relatedness - provide the validated motivational constructs that make drive inference principled rather than arbitrary Deci & Ryan, 2000. Four decades of empirical validation across education, work, healthcare, and digital systems confirm their cross-domain applicability.

Affective computing grounds the State layer. Picard's founding argument - that emotional context matters for human-computer interaction - provides the theoretical basis for Layer 3 inference Picard, 1997. D'Mello and Graesser's research on affective state dynamics during learning established that state transitions follow predictable patterns, making inference from behavioural signals feasible D'Mello & Graesser, 2012.

JITAI research grounds the Behaviour layer and temporal calibration. Nahum-Shani et al.'s distinction between vulnerability states and receptivity states provides the temporal logic that determines when layer transitions matter most Nahum-Shani et al., 2018. Correct inference at the wrong moment still fails.

Learning analytics grounds the Action layer and the Action-to-Behaviour transition. Siemens and Baker's foundational acknowledgement - that the field captures behavioural traces but lacks frameworks for interpreting them - is the precise gap ABSD addresses Siemens & Baker, 2012. Li et al.'s demonstration that clickstream patterns predict student dropout with 78–84% accuracy confirms that Layer 2 signals carry real predictive value Li et al., 2020.

This study employs a qualitative research design combining expert interviews and a practitioner survey. The methodology follows Braun and Clarke's thematic analysis approach for the qualitative data Braun & Clarke, 2006 and descriptive statistical analysis for the survey instrument.

5.1 Research questions

RQ1: What behavioural signals do AI practitioners currently identify and use in their systems?
RQ2: How do practitioners interpret the relationship between behavioural signals and underlying user states?
RQ3: How do practitioners evaluate the ABSD framework's utility for their domain?

5.2 Data collection

Expert interviews were conducted with five AI practitioners from CX and EdTech domains using the semi-structured protocol in Interview Protocol. The interview tool at predictwhy.com/interview delivered questions via voice synthesis and captured responses as transcripts. Each interview ran 15–20 minutes.

The practitioner survey at predictwhy.com/survey deployed the 15-question instrument in Survey Instrument to a broader sample. Behavioural timing data - time per question, hesitation patterns, answer changes - was captured alongside responses as a methodological demonstration of the ABSD framework applied to the research instrument itself.

5.3 Analysis

Interview transcripts were analysed using Braun and Clarke's six-phase thematic analysis. Survey data was analysed descriptively. Cross-method triangulation compared patterns emerging from both instruments against the ABSD framework's theoretical predictions.

Full findings are available on the live findings page, which updates daily as new data arrives. The summary below reflects the state of the research at the time of writing.

The signal gap is universal. Across CX and EdTech respondents, a consistent pattern emerged: signals that would support Layer 2 and Layer 3 inference are present in system logs but are not being used. Every domain reported underutilised behavioural data. This validates ABSD's core claim that the problem is not data availability but interpretive framework.

State inference is aspirational, not operational. The majority of practitioners found Layer 3 compelling in theory but distant in practice. The dominant barrier was not technical complexity - it was the absence of a guiding framework. Respondents who rated ABSD most highly on utility frequently noted that it gave them vocabulary for something they had been trying to articulate to their teams.

Context changes everything. Practitioners consistently noted that the same signal means different things in different contexts. Survey item C10 - asking how often the same signal means different things - produced a mean response of 4.1 out of 5. This validates the Contextual Modifier concept as the most practically important structural element of ABSD.

Drive is the hardest concept to operationalise. Interview participants engaged most slowly with questions about drive-level inference. The hesitation was not confusion about the concept - it was a genuine recognition that Layer 4 requires a different kind of system than most teams have built. This finding is itself a behavioural signal captured by the interview protocol timing data.

The ABSD framework makes a modest but specific claim: that the gap between observable user behaviour and underlying psychological motivation can be bridged systematically, using methods that are already validated in adjacent research traditions. It does not claim that this is easy, that AI can read minds, or that implementation is straightforward. It claims that the problem has a structure, and that structure can be named.

The practitioner validation confirms that naming the structure matters. The most consistent response across both survey and interview instruments was recognition - not of something new, but of something familiar that had not previously had a name. This finding suggests that ABSD's primary contribution may be conceptual rather than technical: a shared vocabulary for a problem that practitioners have been working around individually.

The confidence gradient is worth emphasising. Action is observable at 96% fidelity. State inference achieves 70–79% accuracy in controlled settings. Drive inference is longitudinal and probabilistic. The framework is honest about where inference degrades. That honesty is a feature, not a limitation - it specifies exactly what kind of investment each layer requires and what reliability practitioners should expect in return.

AI systems in customer experience and educational technology have achieved impressive capability at the action layer. They see what users do. They do not understand why. The ABSD framework proposes a four-layer architecture - Action, Behaviour, State, Drive - that maps the inferential path from observable behaviour to psychological motivation.

The framework synthesises four research traditions that each address part of this problem: Self-Determination Theory for the Drive layer, affective computing for the State layer, JITAI research for the Behaviour layer and temporal calibration, and learning analytics for the Action layer and aggregation methodology. The literature synthesis found minimal cross-citation between these traditions. ABSD is the first framework to connect all four.

The practitioner validation confirms that the gap ABSD describes is real, that the signals needed to address it are already present in deployed systems, and that the framework provides vocabulary practitioners find immediately actionable. The next step is production validation - demonstrating that ABSD-informed interventions produce measurably better outcomes than Layer 1 approaches. That is the work this research opens, not the work it completes.