Dynamic Topic Modelling for Consumer Segmentation

The importance of meaningful segmentation of consumers from a retail perspective cannot be overstated. Current approaches, however, oversimplifies the situation, most notably by aggregating away time, conflating behaviour based on temporal differences – despite the fact that time, in for example the form of seasons and holidays, has a well known influence on retail behaviour. Addressing this a novel technique has been developed. The approach has since been evaluated and applied internationally within two retail companies, considering both consumer and store segmentation and description.

Project Description

The importance of meaningful segmentation of consumers from a retail perspective cannot be overstated. Many artefacts, such as product lines and communication campaigns, are still aimed at groups and cognitively manageable insights into the consumer based continue to provide invaluable information for strategic management.

A similar argument applies to the meaningful segmentation of stores, where such segmentation can drive stocking decisions, store design and designation or even the variant image projection of stores in certain locations.

While segmentation is commonplace, it typically oversimplifies the situation, most notably by aggregating away time, conflating behaviour based on temporal differences – despite the fact that time, in for example the form of seasons and holidays, has a well known influence on retail behaviour.

Outcomes and Impact

This project focuses the development of novel mathematical approaches and their subsequent interpretation to leverage the abundant transactional records of consumer behaviour (either from a consumer or store perspective) routinely captured by many retailers’ loyalty programs or point-of-sales records. While numerous techniques have been proposed in the past, traditional customer segmentation suffers from at least one of two key potential weaknesses:

  • Segments tend to assume that an individual’s behaviour is stationary – i.e. it is static over time.
  • Segmentation traditionally assumes that each individual can be adequately summarized by placing them in a single behavioural group or assigning a probability of membership to multiple groups.

Both of these assumptions are not only tenuous, but have the potential to lead to non-optimal business decisions. In the former, assuming that an individual’s behaviour is stationary neglects the well-known behavioural influences of seasonality, holiday periods and life stage changes. Under such an assumption all these differing behaviours are conflated to a single aggregate, temporally invariant, model – a potentially unrepresentative temporally averaged model which is unlikely to reflect an individual’s actual behaviour at any given, short term, time frame. In the latter, discriminative and/or insightful non-dominate sub-behavioural patterns are potentially obfuscated or lost. Moreover, while techniques exist to emerge underlying temporal topics/trends/motivations within alternative fields such as text mining, these approaches are not suitable for understanding retail data, either due to the way they model time or due to assumptions they invoke which are clearly violated in transactional retail data.

Addressing this a novel technique that uses a variant of dynamic topic modelling based on non-negative tensor factorization has been developed. The approach has since been evaluated and applied internationally within two retail companies, considering both consumer and store segmentation and description.

Dissemination

The work was presented at the AMA Winter Marketing Academic Conference. The extended abstract is available as a pre-print or in the AMA Proceedings.

A Journal paper is currently in preparation.

Posted in Projects.