Modelling in uncertain times

EMMA MURRAY, IBM GLOBAL BUSINESS SERVICES

On a dreary morning in mid-November, the Heads of OR and Analytics Forum (HORAF) held its third quarterly meeting since the start of the pandemic. While our previous meetings had focused on the challenges that come with leading remote teams and adapting to entirely new ways of working, our panel discussion now centred on whether the actual nature of the work we do had changed as a consequence of the pandemic.

Over the last decade, we have seen an exponential increase in the development and deployment of machine learning models. But most of these models work because they are predicated on the simple assumption that the behaviours and patterns observed in historical data will continue into the future. What happens when history ceases to provide a reliable baseline for what we are observing around us?

For myself at least, many of the use cases I’ve worked on over the last year have moved away from reliance on pure ML modelling, with a marked shift towards the application of soft OR techniques and systems thinking. With the advent of lockdowns, policy interventions, and other externalities, it’s not enough to rely purely on the data. Scenario and sensitivity modelling, along with other soft OR techniques, have come back to the fore. Data analysis continues to be an important part of the job – but the reliability of the insights has become less robust as the pandemic continues to drive fluctuations in behaviour. The application of soft OR techniques has helped mitigate some of these challenges and provided a useful construct for modelling the impact of behaviours not previously observed in historical data sets.

Tom Dewar, Hartley McMaster, shared his view that “particularly now, with the massive shifting world we live in, it’s precisely those types of tools and techniques which would add more value at this time. OR equals data plus logic to help people make better decisions.”

Ruth Kelly, NAO, suggested that OR has benefited from using greater volumes of data, but that in a more uncertain environment, there is always a place for the softer side to better calibrate the assumptions that are used and to consult with the broad range of stakeholders needed to input and understand those assumptions.

Sandra Weddell, TfL, shared some of the challenges that come with modelling demand for transport in these times and the need to revisit old models. “The challenge is the rate of bounce back and behavioural change… Will people feel old levels of crowding are acceptable? How do we forecast the numbers, and how will that feel for our customers? That’s an additional consideration.” After a very rich discussion, it was clear that the pandemic has disrupted how many organisations approach building models – but it has also presented an opportunity to apply more traditional, Soft OR techniques that have fallen out of use.

Following this discussion, John Hopes and John Ranyard undertook an investigation to better understand the current landscape of where and how Soft OR is currently being applied. Their investigation found that the Soft OR community is thriving. There is a wealth of resources available to those who want to see how these techniques can help improve modelling in these uncertain times (see this month’s Leader by John Hopes).