How do you make decisions under uncertainty? Jonathan Sykes is studying uncertainty and how it relates to sustainable built environments. In January 2019 the School of Mathematics and Statistics and the University of Sheffield hosted the SHELF User Forum. SHELF (SHeffield ELicitation Framework) is used by at least one major player in pharmaceutics and a big European safety agency. In this blog Johnathan will explain what it is, and how it works, and how it helps make decisions under uncertainty. He’ll also look at how SHELF can be used in sustainability research.
The trouble with making decisions is that sometimes you don’t know everything you’d like to know. For example, you don’t know everything about the house you might buy (do you need that more expensive survey?) but you still have to make a decision one way or another. Well, it is the same for a lot of policy decisions – private or public.
SHELF is an established standard for making decisions under uncertainty – when information is difficult to collate and needs to be evidenced. It is a package of documents, templates and software to carry out elicitation of probability distributions for uncertain quantities from a group of experts. SHELF provides a better framework for decisions made under uncertainty, and calculations informed by informal chats.
The basic premise of SHELF is to assemble a group experts of a model. An example might be a model of flu spread in the elderly in Britain. The group might include a GP and a NHS manager brought together to find out the likely exposure of the elderly each day compared to 18-35 year olds. The goal is to better inform the assumptions for updating the model with more realistic estimates of input parameters after a recent outbreak. The GP would have more anecdotal information (but also more detailed information) about how often the elderly interact with other groups. The NHS manager would have information on what policy could be changed to try and address the flu risk.
The SHELF facilitator would introduce the model to the group and explain the context. They would then provide some practice examples as to how to provide probability assessments. Then they move on to assessing the likely exposure. This might well include tasks such as probability distributions with the assistance of the facilitator and discussing the impact on the model.
Most statisticians incorporate expert opinion where possible when trying to calculate the key values. Statisticians need statements of the likelihood of one thing against another, which is a counter intuitive thought process for most people, including experts. The SHELF framework provides extensive guidance about how to convene a meeting to handle this process. One issue that might arise is that panel members could present very different evidence as a result of different subfields. Back to our flu example, doctors may encounter a disease differently in different patient groups. SHELF guidance and training is designed to assist in these scenarios.
In the field of climate science the IPCC has a well developed framework for dealing with uncertainty, but other groups will also have to develop methods to handle uncertainty.
For example, extreme weather events and rising populations are damaging international food supply chains. Jeremy Grantham (founder of the Grantham Centre) highlighs that soils are degrading and the global population has tripled since he was born. This means decisions need to be made now about how to invest money in more sustainable and more productive agriculture. However such decisions will be subject to uncertainty about where shortages will be. Further, there is uncertainty about where and how big populations will be in the future. Climate change and responses to it mean uncertainty has to be managed.
I myself have built on other work to provide a framework for handling climate change in the context of building energy simulation. I produced a statistical model of the building energy use of the Information Commons in Sheffield. Statistical models can be used to produce uncertain estimates for the energy use of the building. I combined this model with uncertain weather future from the MET office, which then enabled forecasting of future energy use.
We need to be aware of how uncertain the world around us. And we need to learn how to cope with our lack of knowledge. The world generates vast amounts of information, no one can know everything they ‘need’ to. Einstein summed it up this way: ‘the more I learn, the more I realise how much I don’t know.’ As we delve into the future we will realise more things are subject to uncertainty and need to be handled using frameworks such as SHELF.
Read more of Jonathan’s blogs on his project page