Applying team design patterns to achieve meaningful human control over AI-based systems | Dr. Jurriaan van Diggelen (NLD Civ), TNO Soesterberg
Jurriaan van Diggelen is a senior research scientist at TNO, the Netherlands Organisation for Applied Scientific Research. He has a PhD in artificial intelligence and human factors. His main focus area is the design of intelligent technology for civil and military maritime applications. Covered research topics include human-agent teaming, explainable artificial intelligence, machine learning, situational awareness support, multi-modal user interfaces and systems engineering. He is co-chair of the NATO STO group on meaningful human control.
The exact definition of proper control for artificial intelligence systems has been the subject of intensive debate over the past decades. We identify three different positions on this question. The human-centrics argue that humans should always be in control because humans have the greater intelligence and are capable of moral reasoning. The techno-centrics argue that control should – to a large extent – be handed over to computers because these have a significantly greater potential for acting intelligently due to their information processing speed, almost infinite amount of memory, and immunity to stress. The collective perspective stresses that there will never be a single point of control, but that control is always distributed between humans and machines within the system. Therefore, to achieve what is frequently referred to as meaningful human control, we must develop AI systems that behave like team members within larger systems.
Teaming is something people do every day. Children learn it at an early age and can quickly and easily adapt their teaming skills to novel situations with different people. Given people’s intuitive ability to team in varying circumstances, one might expect that coding such common sense into a machine would be straightforward, but common sense has proved an elusive goal in more areas than teamwork. Currently, most machines lack even the most basic teaming skills.
Together with the Institute for Human & Machine Cognition in Florida, TNO has developed the concept of team design patterns to assist in the understanding and design of human-machine systems. Design patterns are reusable solutions to recurring problems. The patterns try to capture the common invariant properties of the problem and the essential relationships needed to solve the problem. These patterns can be extended to meet varying teaming needs across a variety of teaming contexts.
Additionally, this approach captures two critical aspect of teaming that are missing in current approaches and often overlooked in design: nesting and time. Nesting refers to the recursive and compositional nature of activity. When a human collaborates with a machine, the work is embedded in larger organizational and procedural structures and can often be decomposed into simpler structures. Connecting these levels of design from individual AI systems to whole human-AI societies can be regarded as one of the great research challenges for the coming decades. Additionally, joint activity is a process, extended in space and time. One of the main advantages of teams is their flexibility to adapt, which means they will change patterns over time. The team design pattern language provides a means to capture both nesting and time.
Future work involves applying these patterns to even larger systems consisting of multiple teams which in turn consist of many AI systems and humans working together.
Key take-away
Meaningful human control does not reside in one place but is distributed over the collective human-machine team.
Meaningful human control does not occur at a single moment in time. It results from multiple human-machine interactions, some of which have occurred long ago, some of which are instantaneous.