the Annual Report

Transcription

the Annual Report

ORCHID
Developing the Science of
Human-Agent Collectives
FOURTH ANNUAL REPORT
www.orchid.ac.uk
Table of Contents
INTRODUCTION .........................................................1
RESEARCH ACHIEVEMENTS ...................................4
Flexible Autonomy .......................................................................... 5
Agile Teaming ................................................................................ 8
Incentive Engineering ................................................................... 10
Accountable Information Architecture .......................................... 13
HAC Applications ......................................................................... 16
KNOWLEDGE TRANSFER ACHIEVEMENTS .........19
Progress against Objectives......................................................... 20
Key Aims ...................................................................................... 22
APPENDICES ...........................................................24
A – Publications from 2014 .......................................................... 24
B – HAC Applications ................................................................... 29
C – Deliverables from 2014 .......................................................... 30
D – Statements of Work for 2014 ................................................. 31
E – Measures of Success............................................................. 32
F – Project Management Data ..................................................... 40
Introduction
The computer has come a long way from its initial role as a scientific tool in the research
lab. We live in a world where a host of computer systems, distributed throughout our
physical and information environments, are increasingly implicated in our everyday
actions. Computer technologies impact all aspects of our lives and our relationship with the
digital has fundamentally altered as computers have moved out of the workplace and
away from the desktop. Networked computers, tablets, phones and personal devices are
now commonplace, as are an increasingly diverse set of digital devices built into the world
around us. Data and information is generated at unprecedented speeds and volumes from
an increasingly diverse range of sources and via ever more sensor types. It is then
combined in unforeseen ways, limited only by human imagination. People’s activities and
collaborations are becoming ever more dependent upon, and intertwined with, this
ubiquitous information substrate.
As these trends continue apace, it is becoming apparent that many endeavours involve
the symbiotic interleaving of humans and computers. Moreover, the emergence of these
close-knit partnerships is inducing profound change. The ability of computer systems to
sense and respond to our on-going activities in the real-world is transforming our daily
lives and shaping the emergence of a new digital society for the 21st century. More
specifically, rather than issuing instructions to passive machines that wait until they are
asked before doing anything, we are working in tandem with highly inter-connected
computational components that act autonomously and intelligently (aka agents). This shift
is needed to cope with the volume, variety, and pace of the information and services that
are available. It is simply infeasible to expect individuals to be aware of the full range of
potentially relevant possibilities and be able to pull them together manually. Computers
need to more proactively guide users’ interactions based on their preferences and
constraints. In so doing, greater attention needs to be given to the balance of control
between people and machines. In many situations, humans are in charge and agents
predominantly act in a supporting role, providing advice and suggesting options. In other
cases, however, agents are in control and humans play the supporting role (e.g.,
automatic parking systems on cars and algorithmic trading on stock markets). Moreover,
these relationships may change during the course of an activity (e.g. a human may be
interrupted by a more pressing request and so take a less hands-on approach to the
current task or an agent may encounter an unexpected situation and have to ask for
human assistance for a task it was planning to complete autonomously).
In ORCHID, we term this emerging class of systems human-agent collectives (HACs) to
reflect the close partnership and the flexible social interactions between the humans and
the agents. As well as exhibiting increased autonomy, such systems are inherently open
and social. This openness means participants need to continually and flexibly establish
and manage a range of social relationships. Thus, depending on the task at hand, different
constellations of people, resources, and information need to come together, operate in a
coordinated fashion, and then disband. The openness and presence of many distinct
stakeholders, each with their own resources and objectives, means participation is
motivated by a broad range of incentives ̶ extrinsic (e.g., money or tax-benefit), social or
image motivation (e.g., public accreditation or leader-board position) or intrinsic (e.g.,
personal interest in a social cause) ̶ rather than diktat.
1
HAC systems exhibit a number of distinctive features that make them particularly
challenging to engineer, control and predict. Their open nature means control and
information is widely dispersed among a large number of people and agents with different
aims and objectives. The various system elements exhibit a range of availabilities, some
are persistent others are transient. The independent actors need to coordinate flexibly with
people and agents that are themselves adapting their behaviour and actions to the
prevailing circumstances to best achieve their goals. The real-world context means
uncertainty, ambiguity and bias are endemic and so the agents need to handle information
of varying quality, trustworthiness and provenance. Thus, techniques are required to
provide an auditable information trail from the point of capture (a sensor or a human
participant), through the fusion and decision processes, to the point of action and the
agents have to reason about the trust and reputation of their collaborators to take the best
course of action. Finally, in many cases, it is important that the collective action of the
volitionally participating actors results in acceptable social outcomes (such as fairness,
efficiency or stability). These features of HACs require us to:
 understand how to provide flexible autonomy that allows agents to sometimes take
actions in a completely autonomous way without reference to humans, while at other
times being guided by much closer human involvement.
 discover the means by which groups of agents and humans can exhibit agile teaming
and come together on an ad hoc basis to achieve joint goals and then disband once the
cooperative action has been successful.
 elaborate the principles of incentive engineering in which rewards are designed so
that participants are encouraged to take actions that generate socially desirable
outcomes.
 design an accountable information infrastructure that allows the veracity and
accuracy of seamlessly-blended human and agent decisions, sensor data and crowd
generated content to be confirmed and audited.
The ORCHID team all share the ethos of undertaking fundamental research in real-world
contexts (“in the wild”) and are working closely with industrial collaborators to demonstrate
and evaluate this fundamental work in the application domains of energy systems, citizen
science and disaster response. To this end, a number of prototypical HACs (see appendix
B for more details) have been built these include:

AtomicORCHID which illustrates agent-based task co-ordination in disaster response
situations.
 AgentSwitch a fully-automated energy tariff advice system using real-time monitoring
and prediction algorithms. It has been deployed both in homes directly and to support
the Centre for Sustainable Energy in advising thousands of householders on their
energy tariffs.
 TariffAgent, a prototype smart grid system tested in trials with 20 households, has been
developed to explore issues of how users might live with differential tariffs.
 A smart phone app to help locate the New Forest cicada – this has been downloaded
by over 5,000 members of the public. A gamified version has been used in conjunction
2
with electronic cicadas (which emit a sound similar to the real cicada) at a number of
public engagement events including the British Science Festival.
 The MyJoulo home heating advice system was deployed (2,245 users to date) as part
of a DECC study into the requirements of smart heating controls, and won the British
Gas Connected Homes start-up competition. During 2014, trials were conducted with
three UK energy retailers and in 2015 the associated spin out was acquired by Quby.
 An on-line game, Apocalypse of MOP, developed to explore understandings of
provenance. To date it has attracted over 850 users resulting in over 100,000 logged
provenance activities.
 The FigureEnergy system for better understanding and analysis of home energy data,
used by over 30 homes in a trial in collaboration with Kingston University, London.
 The CrowdScanner system that was developed for, and won, the US State
Department’s TAG challenge for social mobilisation and rapid information gathering.
This involved five individuals roaming the streets of New York, Washington DC,
Bratislava, Stockholm, and London on a single day and the challenge was to locate
them based only on a single photo of each individual released at 8am.
 The Bayesian Classifier Combination algorithm for the Shared Task Challenge,
developed in collaboration with a Microsoft Research/Bing team, which won the Shared
Task Challenge in the Crowdsourcing at Scale workshop.
 An automated fantasy football manager that has outperformed 2.5M players in the
online Fantasy Premier league. This has been developed into the RateMyTeam app
which currently has 1,000 views per day.
 The OutrunCancer platform, launched to evaluate networked incentive schemes in
collaboration with Cancer Research UK. This attracted over 1,500 participants.
 Veri.ly a web application that tasks volunteers with verifying reports from the ground or
on-line during major disasters. A recent pilot over 2 days attracted 70 participants and
hundreds of visitors.
ORCHID technologies have also resulted in several software releases in 2014:

Latent force software to support an article in the Journal of Machine Learning
(http://jmlr.csail.mit.edu/papers/v15/)

InferNet module, developed in conjunction with Microsoft Research, which
automatically generates large scale inference code for Bayesian graphical models
(http://blogs.msdn.com/b/infernet_team_blog/archive/2014/06/25/community-based-bayesian-classifiercombination.aspx).

Prov Python library, an implementation of the Provenance Data Model by the World
Wide Web Consortium (https://pypi.python.org/pypi/prov).

ProvStore which is a public online repository for provenance documents with over
13,000 documents (https://provenance.ecs.soton.ac.uk/store/).

Non-intrusive Load Monitoring Toolkit which is an open-source toolkit for evaluating
energy disaggregation research (https://github.com/nilmtk/nilmtk).

Literatin which is a plug-in that allows users to explore the complexity of any text found
within a webpage (https://chrome.google.com/webstore/detail/literatin/igpbgncfceidjgcjddcdamjgbp
dcmbnl?hl=en-GB).
3
Research Achievements
The project team involves a number of leading University groups: the University of
Nottingham (Tom Rodden), the University of Oxford (Steve Roberts and Michael Osborne),
and the University of Southampton (Nick Jennings, Luc Moreau, Alex Rogers and Gopal
Ramchurn). To date, it has directly employed 17 Research Fellows and 24 PhD students,
and a further 6 researchers have
been associated with the project. It
has the following organisations as
industrial collaborators: BAE
Systems, Secure Meters Ltd, the
Australian Centre for Field Robotics
(ACFR) and Rescue Global. It also
employs a Knowledge Transfer
Officer, David Nicholson, who is
jointly funded by EPSRC and BAE
Systems, and a project administrator
(Angela Westley). The project
receives advice from an Independent
Scientific Panel whose members are
Brian Collins (University College London), Yolanda Gil (University Southern California),
Thore Graepel (Microsoft Research), David Parkes (Harvard University) and Alan Winfield
(University of West of England) and an Industrial Advisory Group whose members are
Antony Waldock, Henry Tse and Robert Johnston (BAE), Kaushik Ghosh (Secure), Salah
Sukkarieh (ACFR) and David Jones (Rescue Global).
The research within ORCHID is split into five work areas (WAs), each of which is described
in more detail in the remainder of this section. Each section describes the progress to date
in that area and then outlines the high-level plans for the coming year. More details of the
work carried out are given in the Appendices: C lists all the deliverables produced and D
the statements of work that were active in the reporting year.
This year the project has produced 62 publications in journals and conferences/workshops
(see Appendix A for the full listing). To date, the project has produced 228 publications, 57
in journals/books and 171 in conferences and workshops. Of these, 34 are between more
than one of the partners involved in the project, 9 involve an industrial co-author, 93 involve
an international co-author and 8 were either shortlisted for or won a best paper award. The
project web-site is a key resource and has attracted some 15,000 users and 51,000 page
views since 2012.
In addition to those listed in this section, a number of research achievements relate to
knowledge transfer activities, including the award of 6 patents and participation in Angel
Thunder (one of the largest search and rescue exercises in the world). These are detailed
in the section on Knowledge Transfer Achievements.
4
Flexible Autonomy (WA1)
Progress against Objectives
This Work Area is led by Rodden and its initial objectives, as identified in the proposal,
were:
We will focus on the establishment of the key interactional principles between human and software agents
that are needed to allow a sense of flexible autonomy to be established, monitored and amended in a context
sensitive manner. These principles will be built into key mechanisms and reflected in the ways we represent
and reason about these systems. Key objectives in establishing these principles are as follows. (i) The
development of new presentation and interaction techniques to allow users to understand the actions of
large collections of independent systems as they reason and act on behalf of users. (ii) The development of
new control models and representations that allow dynamic and fluid interactive arrangements of agents
and users. For example, allowing authority relationship between users and agents to change in different
contexts and agents to work under varying levels of supervision. (iii) The development of techniques to
recognise human activity to allow agents to reason about human interaction so they might exploit this
understanding to augment and support the actions of users. (iv) An elaboration of the effectiveness of
different styles of interaction and their role in HACs. For example, when should users give direct orders
that agents must obey? When might users express their broad desire and allow these to be re-interpreted by
agents? Under what circumstances should agents instruct users what to do and when should users be able to
question them?
The principal aim was to continue with the real world deployment and understanding of
HACs and to supplement these with the development of more conceptual work on theories
and frameworks. In more detail:
(i) Our work on presentation and interaction techniques will focus on the following.
First, the development of new interaction techniques focusing on the programme’s
applications. More emphasis will be placed on the situational awareness and disaster
response, which provide particular issues in terms of multimodal interaction and the
visceral nature of the environment. Second, we will complement this exploration by a
closer exploration of physiological interaction where we may exploit the monitoring of
user physiological data as part of interaction. Third, the development and elaboration of
more generic frameworks of the presentation and interaction techniques that generalise
across our different investigations.
(ii) Work on new models of control and representation will focus on the analysis and
understanding of the data collected from the various deployments of ArgusUAV and
AtomicORCHID and the emerging fieldwork from our studies of disaster response
training. These studies will shape the articulation of a range of models and frameworks
that may be used to inform the future design of these systems.
(iii) We will continue to exploit a range of activity recognition techniques to explore the
extent to which these can best be applied. This will be guided by the project’s key
applications with a particular focus on disaster response. This will be complemented by
a consideration of activity recognition based on human physiology gained from a range
of experiences and the analysis of network traffic information captured from domestic
networks and how these might be exploited alongside energy and other captured
domestic information.
(iv) Work on the assessment and evaluation of different styles of interaction will be
undertaken as novel interactive techniques emerge during the year. The aim is to
5
consolidate this work and shape a set of techniques and approaches to promote lowcost rapid prototyping of HACs to allow the assessment of techniques before they are
fully developed. This will build upon our focus group work and animated sketching and
explore the development of approaches based on “Wizard of Oz” tools to mimic agent
response and involvement.
The progress against the aims is as follows:
(i) We continued our work on the development of new presentation techniques with an
increasing focus on articulating the underlying conceptual interactive frameworks of
HACs. This included an exploration of the role of trust within agent - user interactions
and the impacts of different forms of discourse between agents and users. This included

Consolidation of the AtomicORCHID into a full HAC demo presented at the All
Hands Meeting and published at AAMAS 2015.

Engagement and fieldwork with Rescue Global to outline appropriate means of
presenting HAC information in disaster response settings. Resulting in a full paper to appear at CHI 2015.

A third workshop on Human Agent Interaction Design and Models was hosted at
AAMAS and a special issue of the Journal of Autonomous Agents and MultiAgent Systems will be published in 2015.
(ii) We explored issues of control and representation by consolidating our work on novel
methods and techniques for HACs to consider how we might envision future HACs and
allow users to comment on these through a combination of in-situ prototyping and
games-based investigation. Highlights have included

Studies of planning and orienting in AtomicORCHID resulting in a series of publication in both the HCI and agent-based communities, e.g. COOP204 and
CTS2014.

Continued studies of the use of language in the giving of instruction as part of an
on-going PhD studentship.

Design deployment and analysis of a provenance based game, which led to a
publication at CHI2015.
(iii) Our work on capturing and representing human activity continued to focus on the
significant ethical challenges that need to be addressed within HAC systems. In particular, this focused on a reconsideration of consent and ethics for HAC systems. This built
upon our existing explorations of consent within energy systems to consider how future
autonomous systems might be understood legally and ethically, as well as interactionally. This has included

The design and development of a series of Ideation cards for involving legal and
ethical considerations in the design of intelligent and agent based systems which
was published at CHI 2015.

A lab based study of multi UAV control using Functional New-Infrared Spectroscopy (fNIRS) brain study technology.
6
(iv) We elaborated the effectiveness of different styles of interaction through the development of a HAC pattern language looking at the various forms of human agent arrangement that have been used to date and the construction of a repository to capture
these arrangements. This included the design and use of a patterns card design used in
the design of game experiences including agents.
Key Aims for 2015
Next year will emphasize wide scale public release and engagement with real world settings:
(i) We will continue our work on the development of new presentation techniques focusing in particular on the development and demonstrations of the use of augmented projection of information within disaster responses settings in partnership with Rescue
Global.
(ii)
We will extend our work on
control and representation to consider how we might envision future
HACs in real world settings using insitu prototyping and deployment “in
the wild” focusing particularly on
control room settings within disaster
response. This will be complemented by an in-depth study of the use of
fNIRS brain scanning technology as
a means of capturing human activity
in controlling UAVs.
(iii) Our work on capturing and representing human activity will continue to focus on the
significant ethical challenges that need to be addressed within HAC systems. This will
include extending and rolling out our design ideation approaches.
(iv) We consolidate our work on the development
of a HAC pattern language by exploring how
these may be presented to designers through
a series of design prototyping exercises.
7
Agile Teaming (WA2)
This Work Area is led by Rogers and its initial objectives, as identified in the proposal,
were:
We will focus on the development of key mechanisms and formalisms to deliver the agile teaming described
above and how these are made available to users. Key objectives in establishing these mechanisms and
formalisms include the following. (i) The development of new metaphors and design guidelines that
enable effective teams to be formed by interleaving humans and agents such that they can work collectively
toward a particular task, while fully exploiting the specific advantages and constraints of both parties. (ii) The
development of mechanisms and algorithms that allow humans and agents within a group to effectively
coordinate their activities and actions to collectively maximise their utility. (iii) The integration and
demonstration of these approaches within exemplar applications within the domains of energy systems and
disaster response.
The stated aims for 2014 were:
(i) In terms of the development of new metaphors and design guidelines, the insights
from AtomicORCHID, the Cicada Hunt trials, and engagement with Rescue Global will
be incorporated into a set of HAC design patterns. The generation of these patterns will
incorporate key insights from the user trials conducted to date.
(ii) In terms of the development of mechanisms and algorithms that allow humans and
agents within a group to effectively coordinate their activities work, Maleki will focus on
the scaling of both Shapley value calculation and coalition structure generation,
exploiting both approximation schemes that provide bounds, and features of the
coalition characteristic function. Beck and Venanzi will continue work to develop
algorithms that combine principled probabilistic representations of future UAV task
arrival with decentralised coordination algorithms to efficiently allocate tasks between
UAVs, while Baker will investigate algorithms for UAV path planning. Salisbury will also
be studying how such UAVs can be coordinated with human controllers through
crowdsourcing interfaces such as the ArgusUAV app. Wu will focus on investigating
algorithms for coordination of emergency responders in uncertain environments.
(iii) In terms of the integration and demonstration of these approaches, this will focus on
further development of the AtomicORCHID platform, specifically integrating IBCC
algorithms to extract tasks from social media text feeds, incorporating data feeds from
UAVs that are autonomously coordinating on the basis of these feeds, and providing
software agents to assist the first responders in their own coordination and the
additional allocation of tasks to UAVs. Moreover, we will work on extending the
TariffAgent App to a multi-user setting to investigate human-agent interaction in
collective energy purchasing scenarios.
(i) A publication formalising and presenting HAC design patterns, based upon the real
deployments performed during the course of the ORCHID project has been submitted
for publication.
8
(ii) Fundamental research on agile teaming has continued with Maleki developing and
presenting a rich understanding of approximation algorithms for calculating the Shapley to fairly distribute the rewards of collective behaviour and Beck presenting robust
algorithms for multi-UAV coordination.
(iii) The AtomicORCHID platform was developed into a compelling disaster management
demonstration which was presented at the 2014 ORCHID All Hands Meeting.
Key Aims for 2015
(i) The agile teaming approaches demonstrated within AtomicORCHID will be further
developed in collaboration with Rescue Global as part of an ‘interactive bird-table’ system, which will form part of Rescue Global’s ongoing disaster management operations.
(ii) The coalition formation and agile teaming approaches developed and demonstrated
in simulation in the context of coordinating cooling loads in coalitions of buildings, will
be further developed within an Innovate UK funded project with KiwiPower (the UK’s
leading aggregate demand response
provider).
9
Incentive Engineering (WA3)
This Work Area is led by Roberts and its initial objectives, as identified in the proposal,
were:
We will focus on the development of mechanisms and methods of approach that provide a means of
influencing the behaviour of individual actors (humans and software agents) and groups, where there is no
direct means of controlling the internal workings of the participants. In particular, the key initial objectives for
this work are as follows. (i) Evaluate a principled value for information, action and strategy to inform an
agent’s assignment of utility. (ii) Develop methods to incentivise humans and agents to gather and utilize
information and act in uncertain, dynamic environments. (iii) Develop methods by which communities and
social networks and coalitions of humans and agents can be analysed, tracked and forecasted. (iv)
Determine how and what (weak or indirect) control needs be exerted over collectives of humans and
agents to achieve global objectives by engaging in joint actions.
(i) In terms of developing a principled value for information, action and strategy we
will refine, develop and understand the relationships between intelligent task
assignment, reinforcement learning (RL), inverse RL and preference learning. We aim
to modify such algorithms to accommodate humans within the loop more naturally and
thus, mitigate the need for costly incentives and reduce disincentives. We will
experiment, for example, on crowd-sourcing information with a limit on how often
human participants can be queried and develop approaches to infer a human's utility
function from observing preferred actions instead of constant direct questioning. We will
investigate the impact of these algorithms on the HACs ability to complete tasks
successfully through engagement with empirical demonstrators.
(ii) We will further understand the theoretical underpinning of how humans and agents
can be incentivised to gather and utilize information. We will consider incentives for
humans to operate within a HAC as not only direct incentives (money, knowledge, etc.),
but also engineering software agents to encourage humans to engage with them. We
will work to establish incentive mechanisms to encourage the human to remain truthful
and fully engaged in these situations. We will also investigate the design of budgetlimited crowdsourcing workflows in order to improve the accuracy and efficiency of
crowdsourcing where humans may not be truthful or expert at classification or content
generation tasks.
(iii) We will explore the intimate relationships and methods that exist between dynamic
social networks and coalitions of agents and humans. Investigations into the use of
group-level, rather than individual-level, incentives and weak control will be
investigated, along with further integration of provenance and inferred trust.
(vi) We will drive forwards the application domains as foundations for testing and
comparing incentive mechanisms and weak control. This will achieved by
evaluating the effect of 'nudges' and (ethical) weak control on human agents and more
extensive empirical testing via experiments on HACs using our network of
demonstrators in Citizen-science (Zooniverse, BioSound, New Forest Cicada) and
crowd-generated information (CollabMap, AtomicORCHID).
10
(i) In terms of developing a principled value for information, action and strategy we
have continued the development and exploitation of dynamic methodologies for
tracking user policies and value of information in large, real-world citizen science and
crowd-sourced environments. In particular we have:

Further developed our system for intelligent tasking, using a joint utility function
of information value (system performance) based upon the selection of tasks
and users so as to maximise overall utility. Extensions of Independent Bayesian
Classifier Combination (IBCC) have been validated this year for fusion of text
sentiment and time-series data and further showcased on financial prediction
problems, where they outperform the existing state-of-the-art (winning a best
paper award at ICICA 2014 (Levenberg et al.) and leading to commercial
internships).

Pioneered decentralised approaches for evaluating information, incentives and
optimal aggregations of knowledge from mixed human-sensor systems,
culminating in joint work with Microsoft Research (Simpson et al., 2015).

Developed DynIBCC, which can track the effect of incentives on accuracy, and
used this to determine the types of tasks that affect an individual's reliability.
This is incorporated into our intelligent tasking framework, to select tasks that
bring the most information by balancing current information needs with
incentives.
(ii) For the work on how humans and agents can be incentivised to gather and utilize
information we have developed our theory and practice around three major topics:

Financial incentives have been explored in mixed software and crowd-sourced
systems; for example, in the TREC challenge work, individual workers were paid
a fee for their assessment of documents. These assessments were then used
as 'silver labels' to enable an automated system to learn. This learning was then
used not only to make automated decisions about unseen documents, but also
to select new worker-object pairings, to hire new workers and fire workers
whose information flow to the algorithm was very low.

We have continued to use beneficial information as an incentive. For example,
work on thermal modelling and forecasting (as part of AgentSwitch and MyJoulo)
has enabled users to see the benefits of changing thermostat settings.

Our Citizen-science applications have exploited knowledge as an incentive.
Large user bases, such as Zooniverse, are maintained via knowledge feedback
and the reassurance that the value of effort given is used to maximal effect – a
core driver of our developments in Bayesian decision combination. The New
Forest Cicada work developed user engagement incentives and minimised
disincentives by incorporating best-practice design into the smartphone app and
coordination of events. The new BioSound project takes the same approach and
adds research on user-friendly path-planning to coordinate information gathering
with minimal disincentives.
11
(iii) For the relationships and methods that exist between dynamic social networks
and coalitions of agents and humans we have developed and validated social
network methods, based on probabilistic community discovery to aggregate citizen
scientists into communities based on the similarity of decision-making policies.
Originally, this was applied to a large, dynamic Zooniverse data set that enabled us to
track and quantify community evolution taking place as citizen-scientists are presented
with new objects to classify. The methodology has continued to be leveraged to
understand the importance, and value, of provenance:

In work on provenance graphs from the CollabMap application, we have shown
that perceived trust in information can be inferred from the graph (Huynh et al.,
2014).

From the AtomicORCHID provenance networks we extract meaningful summary
information of what happened to help participants as they consider
accepting/rejecting the Planning Agent's suggestions. Specifically, across
several provenance graphs, we retrospectively ran a bespoke algorithm that
was able to generate meaningful messages on an any-time basis, i.e. from
whatever provenance data had been collected up to the point of execution.
(iv) To drive forward the application domains as foundations for testing and comparing
incentive mechanisms and weak control, we have:

Analysed new data sets obtained from the New Forest cicada experiments.

Undertaken field trials of energy management systems (TariffAgent) where
users work in collaboration with an agent to choose the cheapest tariff.
Investigation has also been undertaken into interaction design to help users
minimise their cost.

Furthered the integration of ORCHID methodologies into the Zooniverse
environment and produced results indicating the approaches taken are
transferable with ease between data sets.

Developed a budget-limited crowdsourcing algorithm (Tran-Thanh et al., 2014)
for complex crowdsourcing workflows.

Developed incentive-based approaches, based on saliency and information
voracity, to underpin the engagement with Rescue Global.
Key Aims for 2015
(i) We will continue to leverage ORCHID methodologies in applications, from citizen
science, disaster response, crowd sourced sensing and home energy management.
(ii) We plan to detect the effect of different incentives through extensions of the
DynIBCC model. We aim for the complete model to learn
an individual's task preferences, how this affects their
reliability, which will therefore allow us to trade-off the most
informative tasks with those that keep a worker engaged.
We will also investigate transfer learning within
decentralised database systems that use IBCC to refine
worker confusion matrices by exploiting data from heterogeneous task domains.
12
Accountable Information Infrastructure (WA4)
This Work Area is led by Moreau and its initial objectives, as identified in the proposal,
were:
We will focus on the establishment of the key principles underpinning an accountable information
infrastructure, and on the design of its architectural foundations. Whereas provenance tends to be detailed,
providing an explicit account of how information was derived, trust and reputation tend to be measures of the
quality of such information. Given this, several approaches have been proposed to derive the latter from the
former. However, their varied nature in terms of representation and size imply different techniques to gather,
manage, reason over and query them. In particular, ORCHID aims to integrate them in a coherent manner in
the accountable information infrastructure. To this end, key objectives include the following. (i) The
specification of the information infrastructure underpinning HACs, including common data models,
representations and APIs to share and access accountable information uniformly. (ii) The definition of models
for provenance/trust/reputation of information and their source in HACs. Models and algorithms will also be
designed to support crowd generated content, uncertain information, incomplete and conflicting provenance,
and anonymity and pseudonymity for privacy concerns. (iii) The design of accountability services, for
online and offline use, based on a decentralised, large-scale architecture for capturing and reasoning over
provenance/trust/reputation. These will exploit existing cloud infrastructures. (iv) The development of
techniques to manage provenance/trust/reputation over variable timescales with an emphasis on the
computational and ethical issues that may emerge from long lived information.
(i) In terms of the specification of the information infrastructure underpinning HACs,
the focus will be on the definition of APIs allowing accountability information to be
accessed, stored, processed and transformed. The challenge here is to design
interfaces that combine ease of use, interoperability, and scalability, but also integrate
the various relevant algorithms that are being developed, in a seamless manner.
(ii) In terms of models for provenance/trust/reputation, work will focus on generalizing
the machine learning methods and demonstrating that one can learn interesting
characteristics of applications from the provenance they generate. Furthermore, a
deployment of CollabMap with online decisions about the quality of data produced and
allocation of tasks to users will seek to demonstrate the potential of this approach.
(iii) As far as the design of accountability services are concerned, the focus will be on
the engineering of scalable implementations for the APIs designed in (i). We will
continue to integrate them in ORCHID applications and, in particular, focus on
provenance patterns, which identify points of potential interest in the execution of
HACs. A promising direction of research is the relation between such provenance
patterns and the HAC design patterns being envisaged in WA1 and WA2.
(iv) In terms of the techniques to manage provenance/trust/reputation over variable
timescales, the focus will be on Cr0n and the associated data and provenance that
can be logged and analyzed. The study will focus on questions such as: can multiple
interpretations of provenance exist?, how much provenance should be exposed to
players?, how does provenance influence trust in information?, and how does
provenance logging impact on user actions? A mix of data analysis, interviews, and
lab experiments will be used in the course of this study.
13
(i) The work on specification of information infrastructure continued with the design and
implementation of an API for registering interest in some provenance events and for
sending notifications when such events occur. This was successfully integrated in the
AHM 2014 ORCHID Demonstrator, which
published notifications whenever plans
had been invalidated by new incoming
information. To support processing of
provenance documents as mathematical
graph artifacts, converters to matrix
representation were also exposed and
deployed. We have also added a new
visualisation to present provenance
information in a flow diagram, showing
flows of influences captured in
provenance information.
(ii) The provenance network analytic approach previously used for classifying the quality
of crowdsourced data in CollabMap has been formalised into a generic methodology
for analysing provenance information based on provenance network metrics. The
method was successfully demonstrated on two further applications: (1) identifying the
owners of provenance graphs deposited on ProvStore from their topological
characteristics, and (2) identifying instructions from chat messages in the
AtomicORCHID trials from their provenance.
(iii) We devised a graph analysis technique based on aggregating provenance types that
is capable of generating provenance graph summaries to facilitate the understanding
of complex provenance information. The summary captures the frequency of
provenance elements and relations in the original graph, the visualisation of which was
shown to reveal outliers hidden in the complexity of the original graph. In terms of the
accountable information infrastructure, we re-designed the data schema underpinning
the storage of provenance information. The new schema allows for queries of
provenance information to be efficiently executed across multiple documents (which
may be deposited by different users or applications).
(iv) In collaboration with Urban Angel, an artistic production company, we designed and
deployed “The Apocalypse of MoP”, an alternate reality game around provenance, to
explore issues regarding usability and comprehensibility of provenance information.
We have finished collecting data from the study, capturing interactions with
provenance of varying degrees of complexity by over 1,000 participants. A qualitative
study was carried out with the participants (41 questionnaire responses and 8 onehour interviews) to probe their understanding and perceptions about provenance in
general, the PROV standard and its graphical illustration, and the specific ways they
interacted with provenance information.
14
Key Aims for 2015
(i) We will carry out quantitative and qualitative analyses of the data of user interactions
with provenance in the Provenance game to understand common graph layout
patterns and how they relate to user understanding of provenance.
Relevant data sets will be packaged and published for the benefits of the research
community.
(ii) We will continue our work on
identifying common provenance
patterns, which will help spotting
points of potential interests in
HAC executions. Our initial
investigation showed that
existing techniques did not scale
well with the size of provenance
graphs common in ORCHID
applications. The focus for this
work will be on developing
pattern extraction techniques
tailored for provenance graphs
with the emphasis on scalability.
(iii) The matrix representation for provenance graphs has allowed us to apply matrix
analyses over provenance information. In particular, we will investigate the application
of matrix decomposition techniques over the corpus of provenance graphs from
ORCHID applications. A potential result is the identification of invariant component
matrixes from provenance graphs generated by HAC applications.
15
HAC Applications (WA5)
This Work Area is led by Ramchurn and its initial objectives, as identified in the proposal,
were:
The specific objectives are as follows. (i) Define vignettes that generate use-cases for HAC systems that
reflect the level of maturity of the developed HAC technologies. Through outreach exercises (workshops,
symposia, and other domain-specific forums) and our various advisory groups, the vignettes will be enriched
and enhanced. This may include looking outside the initial exemplar domains of energy systems and disaster
response as new opportunities present themselves. (ii) Develop a methodology for the construction of
HAC systems that can be used by researchers and practitioners that combines all the constituent
components into a coherent overarching framework. (iii) Design applications that both collect domain data
from deployments and simulate challenging domain-specific problems with high fidelity. Initially, user
applications with rapid up-time will be developed that permit the collection of sizeable datasets with regards to
user behaviour and system performance and, in the long term, act as vehicles for evaluating ORCHID
technologies ‘in the wild’. In addition, simulations will be designed to contain an extensive set of use-cases,
coupled with a benchmarking framework. (iv) As technologies mature from other work areas, they will be
evaluated in both real-world settings (through user trials) and simulation platforms, feeding back new
datasets and requirements. (v) Construct demonstrators to showcase HACs to domain experts and to
obtain feedback from the public, academics and policy makers that will help identify new requirements and
highlight the associated ethical issues.
The stated aims for 2014 were to focus on those applications already developed in the
project, rather than develop new ones. In particular, we aim to evolve more complete
versions of HAC applications that demonstrate the full range of HAC capabilities and allow
us to start establishing the foundational principles of HACs. In more detail:
(i) We will explore the notion of HAC design patterns through extensions to the
AtomicORCHID platform. Specifically, we will extend the scenario to involve notions of
agile teaming where multiple humans and multiple agents need to coordinate. We will
also introduce incentive engineering models into the platform and evaluate them in situ.
The platform will be provenance enabled and provenance will be used in the decision
making process involved in the game. Through such extensions, we will demonstrate
the full potential of HACs and further establish the design principles and interaction
mechanisms for them. We aim to involve Rescue Global in our deployments in order to
obtain real-world validation of the mechanisms we develop.
(ii) We will deepen our research into collective energy management problems. In
particular, we aim to extend AgentSwitch with collective tariff switching, through the
TariffAgent probe, and energy management features that will allow us to evaluate
coalition formation principles where humans and agents are involved. We also aim to
research new design principles for autonomous energy systems through further
deployments of our tools and platforms with energy advisors based at the Centre for
Sustainable Energy and through field trials with end users.
(iii) We are discussing MyJoulo trials for spring 2014 with EDF, British Gas and Npower.
Each trial will potentially involve 500-1000 users, and we intend to deploy a new
analysis website and new logger hardware. These deployments will generate new data
that will allow us to inform agent-based models of homes and human behaviour and
algorithms to learn these behaviours.
16
(iv) We will run the New Forest Cicada hunt again in conjunction with BugLife and the New
Forest National Park Authority using a newly updated app and develop new
mechanisms to attract more users within the Forest. This will involve attracting users
with an intrinsic interest in the cicada, but we will also look into attracting users in
serendipitous ways as they visit the Forest, thus deploying notions of incentive
engineering in situ.
(v) Apocalypse of MOP will be used to study issues around the generation, curation and
manipulation of provenance data. Moreover, in AtomicORCHID we will develop new
ways to present provenance data and use it to correct for inconsistencies in HAC action
plans.
(i) We developed AtomicORCHID version 3.0 to include novel flexible autonomy interfaces and incentive engineering mechanisms based on initial trials of the system.
AtomicORCHID was thus trialled with over 30 participants including members from
Rescue Global and results written in a number of research papers (Ramchurn et al.,
2015, Fischer et al., 2014, Jiang et al., 2014). Building upon this work, we implemented
HAC-DR, a disaster response system based on HACs. HAC-DR implements HACs to
solve challenges related to gathering situational awareness using crowds, deployments
of multiple UAVs to gather first-hand information, AtomicORCHID, and a provenance
tracking and management system. HAC-DR was demonstrated at the AHM 2014 and
led to a paper at AAMAS 2015.
(ii) We conducted a long-term field trial of TariffAgent to investigate issues around acceptance/adoption of autonomy over long periods of time. The results are being written
up for a journal paper. Furthermore, to investigate challenges related to multi-user environments, in collaboration with researchers from the University of Zurich, we implemented the SmartThermo platform that incorporates an agent that can adjust thermostat settings taking into account users’ preferences for comfort against cost. This system is to be deployed in January/February 2015. Work with the Centre for Sustainable
Energy in collaboration with the CharIoT project, has led to the construction of low-cost
temperature/humidity/presence sensors for deployment in low-income households and
trials are about to begin in 2015.
(iii) The MyJoulo home heating advice system has been deployed within a number of trials.
In January 2013 it was trialled in 750 homes as part of the initial prototype deployment.
In September 2013 the project won the British Gas Connected Homes Start-up Competition and was subsequently trailed in January 2014 in 1500 home in collaboration with
three UK energy companies. In September 2014, Joulo Ltd. was spun-out from the
University of Southampton with Prof. Alex Rogers and Dr. Reuben Wilcock as founders, and was subsequently acquired by Quby (http://quby.nl), a smart thermostat manufacturer based in the Netherlands, in January 2015.
(iv) A smart phone app to help locate the New Forest cicada – this has been downloaded
by over 5,000 members of the public, and generated over 5000 reports in the New Forest. A gamified version has been used in conjunction with electronic cicadas (which
emit a sound similar to the real cicada) at a number of public engagement events in17
cluding the British Science Festival and a seminar at the EPSRC office. The work will
be presented at the Houses of Parliament in the 2015 SET for Britain competition.
(v) Apocalypse of MOP was deployed and studied with 850 users and the results written
up for a research paper (Bachour et al., 2015).
Key Aims for 2015
A number of new research challenges have been identified in our deployments of various
prototypes as well as through ethnographic studies of user environments. While a number
of these challenges are being taken up within new projects (e.g., CharIoT and ASUR)
others will build upon the infrastructure already created by this work area (e.g.,
AtomicOrchid, Apocalypse of MOP, TariffAgent, Multi-UAV coordination systems) to
undertake deeper studies. Hence, this work area will aim to consolidate our portfolio of
applications that will form part of the legacy of ORCHID. In more detail:
(i) We will engineer AtomicORCHID to be
accessible to the wider research community.
This will involve opening up the code base of
the smart planner agent and providing support
to run initial field-trials by external parties.
(ii) We will capitalise on the work done in the HACDR system to develop new tools in
collaboration with Rescue Global. In particular,
we will
look at instantiating the functionalities of the
HAC-DR system in terms of an augmented bird
table that aims to provide enhanced capability to
disaster responders while being robust to harsh
physical environments. We aim to trial
prototypes at Angel Thunder 2015 and at the
Intelligent Disaster Management workshop in
Mexico.
(iii) The SmartThermo system will be trialled and studies on multi-user energy systems will
be carried out.
(iv) A report on HAC design patterns will be produced to capture the key lessons learnt in
developing HAC applications.
(v) We will work with the Centre for Ecology and Hydrology (CEH) to
deploy the species recording and recognition technology
developed in the New Forest Cicada project, within a new field
guide app for all UK Orthoptera. In addition, we will work with the
Royal Botanic Gardens, Kew, as part of a Google Impact Award,
to investigate the application of these techniques for the acoustic
detection of mosquitos.
18
Knowledge Transfer Achievements
The Knowledge Transfer activities, led by Nicholson, identified the following communities
as the beneficiaries for the ORCHID work: (i) academics; (ii) industry and (iii) policy makers
and the general public. The stated aims for each of these categories for 2014 were:
Academic knowledge transfer
(i) Nationally:
 ORCHID will continue to engage with EPSRC’s AISP and Digital Economy
programmes, at an individual researchers’ level, but also more widely through joint
participation in at least one workshop, conference or other relevant event, to
explore overlap and opportunities for collaboration.
 ORCHID will aim to connect with relevant UK Centres for Doctoral Training (CDT),
in particular those on Next Generation Computational Modelling, Future
Autonomous and Robotic Systems, Data Science, My Life in Data and Energy
Demand.
(ii) Internationally:
 ORCHID staff and researchers will organise the 3rd Human-Agent Interaction and
Design and Models (HAIDM) workshop at AAMAS 2014.
 ORCHID staff and researchers will organise the first workshop on Demand
Response at ACM E-Energy.
 ORCHID will continue to team and participate in new international challenges.
Industrial knowledge transfer
(i) ORCHID will host its third Industry Day.
(ii) ORCHID will continue to work to organize further short secondments and visits (both in
and out). The aim is to have 3-4 of these completed or in progress by the end of the
year.
(iii) ORCHID will run an AtomicORCHID trial ‘in the wild’ for industry, government or NGO
participants and/or observers, who will be given the opportunity to help shape future
demonstration scenarios, application, and the route to exploitation.
(iv) For ArgusUAV, the aim is to put this in the hands of end-users as early as possible to
give focus to further technology development.
(vi) On the back of the successful CDE project with BAE Systems, ORCHID will partner with
industry, government, or NGOs, to win further external funding for exploiting HAC
technology in real-world applications that could derive benefit.
(vi) ORCHID will progress its commercialization plans for MyJoulo and AgentSwitch. It will
extend the MyJoulo business to reach more large companies, so it can help them and
their consumers. ORCHID will finalize a business plan for AgentSwitch and aim to
secure funding to begin implementing the plan.
19
Policy makers and general public knowledge transfer
(i) ORCHID will formalize its collaboration with Rescue Global by working together to
define a set of relevant and realistic knowledge transfer targets and their
implementation plan. The aim is to achieve this by: workshops, tabletop exercises,
testing ORCHID’s HAC algorithms on real datasets obtained by Rescue Global on its
operations; and running trials together.
(ii) ORCHID will produce a number of articles targeted toward policy makers. These will
have a HAC application focus and will be authored with domain experts (e.g. Rescue
Global) to provide real-world examples of where HACs could benefit end-users and the
implications for future policy decisions.
(iii) ORCHID will support the British Science Festival by delivering a stand with a variety of
educational materials and hands-on demonstrations based around HAC science.
(iv) ORCHID will design, develop, and release classroom materials that will help teachers
to educate schoolchildren about HAC science in a compelling way.
(v) ORCHID will engage with the general public and invite them to play Apocalypse of
MOP.
(i) ORCHID continued to engage with other relevant EPSRC programmes, in particular the
Autonomous and Intelligent Systems Programme (AISP), the University Defence Research Collaboration in Signal Processing, and two Centres for Doctoral Training (Oxford’s Autonomous, Intelligent Machines and Systems CDT and Southampton’s Next
Generation Computational Modelling CDT).
(ii) ORCHID participated in the organization of the International Workshop on Human-Agent
Interaction Design and Models at AAMAS 2014. This was the third such event and ORCHID contributed papers, presentations, and panel discussion.
(iii) ORCHID published a HAC science piece in the Communications of ACM to promote key
HAC research challenges to a wide academic audience and encourage multidisciplinary
effort.
(i) ORCHID held its third Industry Day in July 2014 at the British
Computing Society London office. The event was attended by
25 people outside ORCHID, representing a variety of industrial
organisations (BAE Systems, Secure Meters, QinetiQ, Selex,
Envitia, Blue Bear, British Gas, IBM, and Toshiba). It also attracted attendees from national and local government (Dstl,
DECC, and Avon Fire & Rescue Service) and charities (Rescue Global).
20
(ii) ORCHID placed four researchers on secondment with external organisations: one to
British Gas Connecting Homes, two to Microsoft Research Lab, and one to BAE Systems.
(iii) ORCHID trialled its AtomicORCHID technology demonstrator with Rescue Global Pathfinders, who provided feedback based on their real-world disaster management experience and operational needs. This feedback provided valuable steer that helped to improve AtomicORCHID’s use-cases and application.
(iv) ORCHID matured the test and evaluation of its ArgusUAV technology demonstrator
through external funding from MOD’s Autonomous Systems Underpinning Research
(ASUR) programme. This resulted in a quantified evidence of the benefit of flexibility autonomy in missions involving the assignment of multiple UAVs to multiple tasks.
(v) ORCHID research was used to leverage external funding from the following sources:
MOD (for the ASUR HAC project); EPSRC (for the CharIoT project, partnering the universities of Nottingham and Southampton with the Centre for Sustainable Energy);
Google (for an Impact prize awarded to the University of Oxford to exploit HAC research
for mosquito detection); Innovate UK (for the Adaptive Demand Response project, partnering the University of Southampton with KiwiPower).
(vi) ORCHID’s spin-out company Joulo (formed in 2013)
has continued to grow, resulting in Joulo monitors
placed in 2500+ UK homes to provide home heating
advice and ongoing interest from the UK’s big energy
suppliers. In early 2015 it was acquired by Quby.
(vii) ORCHID has begun to engage with Dstl on how relevant ORCHID technologies could be transferred and demonstrated within Dstl’s C4ISR
Concepts and Solutions (CCS) programmes. A workshop was held with 12 Dstl participants and this has led to follow-up discussion around specific ORCHID technologies
and military applications.
(viii) ORCHID ran an Industry Master Class on Non-Intrusive Load Monitoring at Imperial
College. The Class was attended by representatives from AlertMe, British Gas, Green
Running, Greeniant, Navetas, ONZO and Wattgo, as well as academics from Imperial
College London and the University of Southampton.
(ix) ORCHID provided a consultancy service to Toyota Central R&D Labs to apply incentive
mechanisms for electric vehicle charging to smart grid settings with photovoltaic generation
(x) ORCHID developed a crowdsourcing module for Microsoft Research Labs and this is
being tested on an experimental platform used by Bing researchers prior to further development and application
(i) ORCHID continued to generate popular press in 2015, with new articles appearing in:
The Guardian, The Daily Telegraph, The Economist, WIRED, Financial Times, ITV
News and Sky News.
21
(ii) ORCHID researchers attended Angel Thunder 2014 at the invitation of Rescue Global.
This is a large-scale disaster management exercise conducted in the US and sponsored
by various US government agencies. ORCHID carried out an ethnographic study.
(iii) ORCHID researchers manned a stand at the British Science Festival 2014, engaging
hundreds of schoolchildren and members of the
public in a compelling game-based demonstration
of ORCHID technology and its impact in the disaster response domain.
(iv)
ORCHID participated in the First Lego
League event in Hampshire, engaging students in
the 9-16 age range with programming Lego Mindstorm robots to perform tasks that replicate those
arising in disaster settings.
(v) ORCHID researchers have partnered with the Fantasy Football First blogging site to
launch the RateMyTeam tool to football fans around the world. The tool uses HAC algorithms and concepts to help users pick their fantasy football teams.
(vi) ORCHID delivered algorithms and software for Independent Bayesian Classifier Combination into Zooniverse, a citizen science platform
(vii) ORCHID contributed algorithms and software to
Microsoft’s Infer.NET community-based
crowdsourcing application
(viii) ORCHID’s Apocalypse of MoP (Provenance
Game) attracted over 850 sign-ups and 300 active players from the general public, creating
complex provenance graphs to test and evaluate
ORCHID’s provenance research.
Key Aims for 2015
(i) ORCHID will co-organise the Intelligent Disaster Management Workshop in Mexico in
February 2015. Its goal is to serve as a springboard for long-term collaboration between
Mexican and UK universities as well as companies interested in disaster response.
(ii) ORCHID will co-organise a HAC-related workshop at AAMAS 2015 and contribute papers, presentations, and research leads for a panel discussion.
(iii) ORCHID will host a technical conference in London (September 2014) to share its main
results, demonstrations and finding with the UK and European academic community.
(iv) ORCHID will pull together materials for a new book on HAC science and applications
and seek an academic publisher with a view to final publication in 2016.
22
(i) ORCHID will continue engagement with its primary industry partners: BAE Systems and
Secure Meters. With BAE Systems, ORCHID will exploit its disaster management and
citizen science research through BAE Systems lead on the ASUR programme and participation in the MOD’s CCS programme. With Secure Meters, ORCHID will seek to
transfer its energy application research into Secure through provision of software and
technology consultancy.
(ii) ORCHID will continue engagement with Dstl with a goal to supporting the field trials and
demonstration days that Dstl has scheduled for its CCS programme in Q1 and Q3 in
2015.
(iii) ORCHID will continue engagement with Microsoft Research Lab and Zooniverse with a
view to the maturation, test, evaluation and release of further crowdsourcing technology.
(iv) ORCHID will seek follow-on funding through the MOD’s ASUR programme to exploit its
research on UAV flexible autonomy in military applications.
(v) ORCHID will continue to promote industrial placements and will target 3 of these in
2015.
(i) ORCHID will work closely with Rescue Global on
a number of activities in 2015: transfer of software
applications for a Digital Bird Table in support of
Rescue Global operations; participation in Angel
Thunder 2015; and delivery of a thought piece
(publication or presentation) to inform policy makers about the impact and role of AI applications in
future disaster response.
(ii) ORCHID will pull together its key energy research findings into a white paper and/or
presentation materials that will be used when and where appropriate to inform and potentially influence policy decisions regarding energy and the environment.
(iii) ORCHID Artists in Residence, Steve Beard and Victoria Halford, will present their work
at an exhibition to be hosted by The Hansard Gallery in Southampton 14 July to 5 September 2015.
(iv) ORCHID will hold a showcase event at the Royal Academy of Engineering in September 2015, at which the popular press will be briefed on ORCHID’s key findings and their
impact.
23
APPENDIX A — PUBLICATIONS IN 2014
The project produced the following publications this year; the full list is available on the project
website. The following markings are used to denote publications that:
*: are between more than one of the groups involved in the ORCHID project
+: involve an international co-author
#: involve an industrial co-author
++: were shortlisted or won a best paper award
* Alan, A., Costanza, E., Fischer, J. E., Ramchurn, S. D., Rodden, T., & Jennings, N. R. (2014). A
field study of human-agent interaction for electricity tariff switching. In Proceedings of AAMAS 14:
13th Int. Conf. on Autonomous Agents and Multi-Agent Systems, 965-972.
* Bachour, K., Wetzel, R., Flintham, M., Huynh, D. T., Rodden, T., & Moreau, L. (2015).
Provenance for the People: A User-Centered Look at the W3C PROV Standard through an Online
Game. In Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2015. In
press.
+ Batra, N., Kelly, J., Parson, O., Dutta, H., Knottenbelt, W., Rogers, A., Singh, A. and Srivastava,
M. (2014) NILMTK: An Open Source Toolkit for Non-intrusive Load Monitoring. In e-Energy '14
Proceedings of the 5th international conference on Future energy systems, 265-276.
+ ++Kelly, J., Batra, N., Parson, O., Dutta, H., Knottenbelt, W., Rogers, A., Singh, A. and
Srivastava, M. (2014) NILMTK v0.2: a non-intrusive load monitoring toolkit for large scale data
sets: demo abstract. In BuildSys '14 Proceedings of the 1st ACM Conference on Embedded
Systems for Energy-Efficient Buildings, 182-183
+ Biermann, F., Naroditskiy, V., Polukarov, M., Nguyen, T., Rogers, A., & Jennings, N. R. (2014).
Task Assignment with Controlled and Autonomous Agents. Mathematical Social Sciences, 71,
116-121.
+ Bistaffa, F., Farinelli, A., & Ramchurn, S. D. (2015) Sharing rides with friends: a coalition
formation algorithm for ridesharing. In Proceedings of AAAI Conference on Artificial Intelligence
2015.In Press.
+ Bistaffa, F., Farinelli, A., Cerquides, J., Rodriguez-Aguilar, J. A., & Ramchurn, S. D. (2014).
Anytime Coalition Structure Generation on Synergy Graphs. In Proceedings of the 2014
international conference on Autonomous agents and multi-agent systems, 13-20
* Costanza, E., Fischer, J. E., Colley, J., Rodden, T., Ramchurn, S. D., & Jennings, N. R. (2014).
Doing the Laundry with Agents: a Field Trial of a Future Smart Energy System in the Home. In
Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2014, 813-822.
+ de Clippel, G., Naroditskiy, V., Polukarov, M., Greenwald, A., & Jennings, N. R. (2014). Destroy
to Save. Games and Economic Behavior, 1-13.
24
+ Farinelli, A., Rogers, A. and Jennings, N.R. (2014) Agent-based decentralised coordination for
sensor networks using the max-sum algorithm. Journal of Autonomous Agents and Multi-Agent
Systems, 28, (3), 337-380.
* Fischer, J. E., Reeves, S., Rodden, T., Reece, S., Ramchurn, S. D., & Jones, D. (2015). Building
a Bird's Eye View: Collaborative Work. In Proceedings of ACM CHI Conference on Human Factors
in Computing Systems 2015. In press.
* Fischer, J. E., Costanza, E., Ramchurn, S. D., Colley, J., & Rodden, T. (2014). Energy Advisors
at Work: Charity Work Practices to Support People in Fuel Poverty. In Proceedings of ACM
International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 14, 447-458.
* Fischer, J. E., Jiang, W., Kerne, A., Greenhalgh, C., Ramchurn, S. D., & Reece, S., et al. (2014).
Supporting Team Coordination on the Ground: Requirements from a Mixed Reality Game. In
Proceedings of 11th Int. Conference on the Design of Cooperative Systems (COOP 14), 49-67.
Han, T., Tran-Thanh, L., & Jennings, N. R. (2014). The cost of interference in evolving systems. In
Proceedings of COIN 2014: The 17th International Workshop on Coordination, Organisations,
Institutions and Norms. In press.
Huynh, D. T., & Moreau, L. (2014). ProvStore: a public provenance repository. In Proceedings of
the 5th International Provenance & Annotation Workshop-IPAW 2014. In press.
* Huynh, D. T., Ebden, M., Ramchurn, S. D., Roberts, S. J., & Moreau, L. (2014). Data quality
assessment from provenance graphs. In Proceedings of the 5th International Provenance &
Annotation Workshop-IPAW 2014. In press.
+ Inoue, S., Moran, S., & Nakata, K. (2014). Supporting Tool for Cooperative Work Analysis Based
on Distributed Cognition. In Proceedings of the 12th International Conference on Probabalistic
Safety Assessment and Management (PSAM). In press.
Jager, N., Moran, S., & Schnadelbach, H. (2014). Using Adaptive Architecture to Support Yoga
Practices: Social Considerations for Design. In Proceedings of Third IEEE Workshop on the Social
Implications of Pervasive Computing for Sustainable Living, 364 - 369.
* Jennings, N. R., Moreau, L., Nicholson, D., Ramchurn, S. D., Roberts, S. J., Rodden, T., &
Rogers, A. (2014). On human-agent collectives. Communications of the ACM, 57, (12), 33-42.
* Jiang, W., Fischer, J. E., Greenhalgh, C., Ramchurn, S. D., Wu, F., Jennings, N. R., & Rodden,
T. (2014). Social Implications of Agent-based Planning Support for Human Teams. In Proceedings
of 2014 Int. Conference on Collaboration Technologies and Systems, 310-317.
Keshavarz, A. S., Huynh, D. T., & Moreau, L. (2014). Provenance for online decision making. . In
Proceedings of the 5th International Provenance & Annotation Workshop-IPAW 2014. In press.
# ++ Levenberg, A., Pulman, S., Moilanen,K., Simpson,E. & Roberts, S. (2014). Predicting
Economic Indicators from Web Text Using Sentiment Composition. In Proceedings of ICICA2014. In press.
25
# A Levenberg, S Pulman, K Moilanen, E Simpson, S Roberts, Predicting Economic Indicators
from Web Text Using Sentiment Composition, In International Journal of Computer and
Communication Engineering 3 (2), 109-115.
Maior, H. A., Pike, M., Wilson, M., & Sharples, S. (2014). Continuous detection of workload
overload: An fNIRS approach. In Proceedings of the international conference on Ergonomics &
Human Factors 2014, 450–457 .
Maior, H. A., Pike, M., Wilson, M., & Sharples, S. (2014). Evaluating Mental Workload with fNIRS.
The Ergonomist. In press.
Moran, S. (2014). Social Networks and Ubiquitous monitoring: An Application of the PSA-BI Model.
In Proceedings of Third IEEE Workshop on the Social Implications of Pervasive Computing for
Sustainable Living, 370 - 375.
+ Moran, S., Bachour, K., & Nishida, T. (2014). User Perceptions of Anthropomorphic Robots as
Monitoring Devices. AI & Society: Knowledge, Culture and Communication, 1-21.
Moran, S., Luger, E., & Rodden, T. (2014). An Emerging Tool Kit for Attaining Informed Consent in
UbiComp. In Proceedings of 1st International Workshop on How do you Solve a Problem like
Consent, 635-639.
Moran, S., Luger, E., & Rodden, T. (2014). Exploring Patterns as a Framework for Embedding
Consent Mechanisms in Human-Agent Collectives. In Proceedings of 8th International Workshop
on Human Aspects of Ambient Intelligence, 475-486.
Moran, S., Luger, E., & Rodden, T. (2014). Literatin: Beyond Awareness of Readability in Terms
and Conditions. In 1st International Workshop on How do you Solve a Problem like Consent, 641646.
* Moran, S., Pantidi, N., Rodden, T., Chamberlain, A., Griffiths, C., & Zilli, D., et al. (2014).
Listening to the Forest and its Curators: Lessons Learnt from a Bioacoustic Smartphone
Application Deployment. In Proceedings of SIGCHI Conference on Human Factors in Computing
Systems, 2387-2396.
Moreau, L., Huynh, D. T., & Michaelides, D. (2014). An Online Validator for Provenance:
Algorithmic Design, Testing, and API. In Proceedings of 7th International Conference on
Fundamental Approaches to Software Engineering (FASE'14), 291-305.
+ Naroditskiy, V., Jennings, N. R., Van Hentenryck, P. and Cebrian, M. (2015) Crowdsourcing
Contest Dilemma. Journal of the Royal Society Interface, 11, (99), 20140532.
Naroditskiy, V., Stein, S., Tran-Thanh, L., Vlassopoulos, M., & Jennings, N. R. (2014). Referral
incentives in crowdfunding. In Proceedings of HCOMP2014: Conference on Human Computation
& Crowdsourcing, 171-183 .
+ Obraztsova, S. , Markakis, E., Polukarov, M. , Rabinovich, Z. and Jennings, N. R. (2015) On the
convergence of iterative voting: how restrictive should restricted dynamics be? In Proceedings of
AAAI 2015: Twenty-Ninth AAAI Conference on Artificial Intelligence. In press.
26
Panagopoulos, A. A., Alam, M., Rogers, A., & Jennings, N. R. (2015). ADA-HEAT: A general
adaptive intelligent agent for domestic heating control. In Proceedings of AAMAS-15 : 14th Int.
Conf. on Autonomous Agents and Multi-Agent Systems. In press.
+ Panagopoulos, A., Chalkiadakis, G and Jennings, N.R. (2015) Towards optimal solar tracking: a
dynamic programming approach. In Proceedings of AAAI-2015: 29th AAAI Conference on Artificial
Intelligence. In press.
* Pantidi, N., Moran, S., Bachour, K., Rodden, T., Zilli, D., Merrett, G. V., & Rogers, A. (2014). Field
Testing a Rare Species Bioacoustic Smartphone Application: Challenges and Considerations. In
Proceedings of Third IEEE Workshop on the Social Implications of Pervasive Computing for
Sustainable Living, 376-381.
Parson, O., Ghosh, S., Weal, M., & Rogers, A. (2014). An Unsupervised Training Method for Nonintrusive Appliance Load Monitoring. Artificial Intelligence, 217, 1-19.
+ Pawlowski, K., Kurach, K., Svensson, K., Ramchurn, S. D., Michalak, T., & Rahwan, T. (2014).
Coalition Structure Generation with the Graphics Processing Unit. In Proceedings of AAMAS 14:
Pike, M., Maior, H. A., Porcheron, M., Sharples, S., & Wilson, M. (2014). Measuring the effect of
Think Aloud Protocols on Workload using fNIRS. In Proceedings of ACM SIGCHI Conference on
Human Factors in Computer Systems (CHI 2014), 3807-3816.
* Ramchurn, S. D., Simpson, E., Fischer, J. E., Huynh, D. T., Ikuno, Y., & Reece, S., et al. (2015).
HAC-ER: A disaster response system based on human-agent collectives. In Proceedings of
AAMAS-15 : 14th Int. Conf. on Autonomous Agents and Multi-Agent Systems. In press.
* Ramchurn, S. D., Wu, F., Fischer, J. E., Reece, S., Jiang, W., & Roberts, S. J., et al. (2015).
Human-agent collaboration for disaster response. Journal of Autonomous Agents and Multi-Agent
Systems. In press.
* Reece, S., Roberts, S. J., Ghosh, S., Rogers, A., & Jennings, N. R. (2014). Efficient state-space
inference of periodic latent force models. Journal of Machine Learning Research, 1-66.
* # Simpson, E., Venanzi, M., Reece, S., Kohli, P., Guiver, J., Roberts, S. and Jennings, N. R.
(2015) Language Understanding in the Wild: Combining Crowdsourcing and Machine Learning. In
Proceedings of 24th International World Wide Web Conference (WWW 2015). In Press.
+ Ströhle, P., Gerding, E. H., de Weerdt, M., Stein, S., & Robu, V. (2014). Online Mechanism
Design for Scheduling Non-Preemptive Jobs under Uncertain Supply and Demand. In Proceedings
of AAMAS 14: 13th Int. Conf. on Autonomous Agents and Multi-Agent Systems, 437-444.
Teacy, W. L., Julier, S., Nardi, D. R., Rogers, A., & Jennings, N. R. (2015). Observation modelling
for vision-based target search by unmanned aerial vehicles. In Proceedings of AAMAS-15 : 14th
Int. Conf. on Autonomous Agents and Multi-Agent Systems. In press.
27
+ Tran-Thanh, L., Huynh, T. D., Rosenfeld, A., Ramchurn, S.D. and Jennings, N. R. (2015)
Crowdsourcing Complex Workflows under Budget Constraints. In Proceedings of Twenty-Ninth
AAAI Conference on Artificial Intelligence (AAAI-15). In Press.
+ Tran-Thanh, L., Huynh, D. T., Rosenfeld, A., Ramchurn, S. D., & Jennings, N. R. (2014).
BudgetFix: Budget limited crowdsourcing for interdependent task allocation with quality
guarantees. In Proceedings of AAMAS 14: 13th Int. Conf. on Autonomous Agents and Multi-Agent
Systems, 477-484.
Tran-Thanh, L., Stein, S., Rogers, A., & Jennings, N. R. (2014). Efficient crowdsourcing of
unknown experts using bounded multi-armed bandits. Artificial Intelligence, 214, 89-111.
+ Vallverdu, J., Casacuberta, D., Nishida, T., Ohmoto, Y., Moran, S., & Lazare, S. (2014). From
Computational Emotional Models to HRI. International Journal of Robotics Applications and
Technologies, 1, (2), 11-25.
# ++ Venanzi, M., Guiver, J., Kazai, G., Kohli, P., & Shokouhi, M. (2014). Community-Based
Bayesian Aggregation Models for Crowdsourcing. In Proceedings of 23rd International World Wide
Web Conference (WWW 14), 155-164.
Vinyals, M., Robu, V., Rogers, A., & Jennings, N. R. (2014). Prediction-of-use games: a
cooperative game theory approach to sustainable energy tariffs. In Proceedings of AAMAS 14:
Wetzel, R. (2014). Introducing Pattern Cards for Mixed Reality Game Design. In Proceedings of
Third Workshop on Design Patterns in Games. Foundations in Digital Games Conference 2014. In
press.
Wu, F., & Jennings, N. R. (2014). Regret-based multi-agent coordination with uncertain task
rewards. In Proceedings of AAAI-14: Twenty-Eighth Conference on Artificial Intelligence (AAAI),
1492-1499.
Ye, C., & Wilson, M. (2014). A user defined taxonomy of factors that divide online information
retrieval sessions. In Proceedings of IIiX '14 the 5th Information Interaction in Context Symposium,
48-57.
Ye, C., & Wilson, M. (2014). The characteristics of casual sessions in search behaviour logs. In
Proceedings of Search4Fun workshop in conference IIiX2014, 6-6.
Zenonos, A., Stein, S., & Jennings, N. R. (2015). Coordinating measurements for air pollution
monitoring in participatory sensing settings. In Proceedings of AAMAS-15 : 14th Int. Conf. on
Autonomous Agents and Multi-Agent Systems. In press.
Zilli, D., Parson, O., Merrett, G. V. and Rogers, A (2014) A Hidden Markov Model-Based Acoustic
Cicada Detector for Crowdsourced Smartphone Biodiversity Monitoring. Journal of Artificial
Intelligence Research, 51, 805-827
28
APPENDIX B — HAC APPLICATIONS
Name
Description
Provenance game run by nottingham to investigate people's
Apocalypse of MOP manipulation of provenance data and understandings of it.
Atomic Orchid
Agent B
AgentSwitch
ArgusUAV
BioSound
Cicada App
CollabMap
Crowdscanner
Fantasy Football
Figure Energy
GalaxyZoo
Hire and Fire
MyJoulo
Publication
http://apocalypseofmop.com/
Contact
Dong Huynh,
Khaled Baschour,
Richard Wetzel
Mixed-reality game to investigate coordination in disaster response. Fischer, J. E., et al. (2012). Serious Mixed Reality Games. In Joel Fischer,
The scenario involves having a planner agent guiding rescuers to
ACM Converence on Computer-Supported Cooperative
Wenchao Jiang,
complete tasks in teams.
Work
Gopal Ramchurn
A HAC application to help evaluate interfaces for demand-response in Costanza et al. (2014) 'Doing the Laundry with Agents' - CHI Enrico Costanza,
the home. An agent advises users as to when is cheapest to schedule 2014
Joel Fischer
their washing machine loads and notifies them if prices change.
Personalized energy tariff-recommender system:
Ramchurn, Sarvapali, et al. (2013) AgentSwitch: towards
Gopal Ramchurn
www.agentswitch.org. Incorporates a prediction algorithm for yearly smart electricity tariff selection. In, 12th International
energy consumption, disaggregation of energy feeds, and collective
Conference on Autonomous Agents and Multiagent
energy purchasing algorithms.
Systems (AAMAS 2013), Saint Paul, US. 8pp.
Interfaces for UAV/UGV control by an emergency responder.
http://player.vimeo.com/video/34837704
Feng Wu, Gopal
Ramchurn
Audio classification of birdsong implemented on mobile devices with http://www.robots.ox.ac.uk/~parg/biosound/doku.php
Steve Roberts,
user input.
Timos
Papadopoulos
Android and iPhone apps to detect an insect in the New Forest. More Zilli, D. et al. (2013). A Hidden Markov Model-Based
Alex Rogers,
details at: http://newforestcicada.info
Acoustic Cicada Detector for Crowdsourced Smartphone
Davide Zilli
Biodiversity
IJCAI
A crowdsourcing platform to help create high resolution evacuation
Ramchurn, etMonitoring.
al. (2013). Collabmap:
crowdsourcing maps for Gopal Ramchurn,
maps. Can be used to create maps using help from voluteers through emergency planning.. In ACM Web Science
Dong Huynh
gamified processes. See more at www.collabmap.org.
App that won the TAG challenge. Using monetary incentives, the app A. Rutherford, M. Cebrian, I. Rahwan, S. Dsouza, J.
Victor Naroditskiy
was able to gather a large team of contributors to identify terrorists in McInerney, V. Naroditskiy, M. Venanzi, N. R. Jennings, J.R. Nick Jennings
4 cities across the world. The mechanism draws upon the DARPA red deLara, E. Wahlstedt, S. U. Miller. Targeted social
balloon, networked incentives method.
mobilization in a global manhunt. PLoS ONE, 2013
http://scailab.media.mit.edu/crowdscanner
URL: www.squadguru.com - An algorithm for sequential decision
Matthews, T., et al.(2013). Competing with humans at
Gopal Ramchurn
making in the FF domain. An interface for human input is under
fantasy football: team formation in large partiallydevelopment.
observable domains. . In Twenty-Sixth Conference of the
Association for the Advancement for Artificial Intelligence.
A web application that provides an interactive interface to your
electricity consumption. See a demo at http://vimeo.com/42328926
Costanza, E., et al. (2012). Understanding domestic energy
consumption through interactive visualisation: a field
study. UBICOMP
Crowdsourcing classification of astronomical objects. The IBCC is used E. Simpson, et al. (2011).
here to help understand the types of users in the system and avoid
Bayesian Combination of Multiple, Imperfect Classifiers.
them getting bored or improve their performance by sending them
Proceedings of NIPS 2011 workshop.
tasks that fit the purpose.
Enrico Costanza,
Gopal Ramchurn
IBCC applied to crowdsourcing to untrustworthy workers. The
algorithm detects the bad workers and fires them. It also hires new
workers and keeps them in if it needs more information.
At myJoulo.com - an 'internet-of-things' application where a low-cost
temperature logger can be used to collect home heating data and a
web interface that is driven by machine learning in order to do
estimates of usage/savings from the data.
Edwin Simpson,
Steve Reece,
Gopal Ramchurn
Alex Rogers
Incentive mechanisms for engaging people in crowdsourcing
activities. The project was carried out in partnership with Cancer
Research UK. The website was viewed almost 9,000 times and over
1,500 users signed up over the course of three months
A suite of python modules providing: an implementation of the W3C
PROV Data model with JSON import/export support; a light-weight
provenance tracker for recording provenance (in python code) with
Provenance Service minimal effort; a provenance repository service that provides a REST
API for storing/retrieving provenance and a Web interface for
browsing the repository's content.
working paper based on TREC challenge work (See below)
Rogers, A., Ghosh, S., Wilcock, R. and Jennings, N. R. (2013)
A Scalable Low-Cost Solution to Provide Personalised Home
Heating Advice to Households [Best Paper Award BuildSys
2013]. In, 5th ACM Workshop On Embedded Systems For
Energy-Efficient Buildings (BuildSys) 8pp.
http://outruncancer.co.uk/
Outrun Cancer
SharedTask BCC
App
TariffAgent
Text Retrieval
Crowdsourcing
Challenge
Ver.ily
An algorithm developed in collaboration with MSR to classify tweets
that won the SharedTask Challenge at Crowdsourcing at Scale
competition at HCOMP 2013
An app to simulate real-time tariffs and to evaluate human-agent
interaction for tariff switching on a daily basis in the face of uncertain
prices due to changing weather conditions.
A challenge as part of the Text Retrieval Conference run by NIST. The
'Crowdsourcing' challenge requires the use of crowdsourcing to help
match topic-document pairs. The approach taken by the Soton-Oxford
team involved the use of IBCC (Independent Bayesian Classifier
Combination) and an Amazon Turk Deployment.
A web application that tasks volunteers with verifying reports from
the ground or on-line during major disasters.
Steve Roberts,
Edwin Simpson
Victor Naroditskiy
Seb Stein
http://www.w3.org/2011/prov/wiki/Main_Page , Luc
Dong Huynh, Luc
Moreau, Paul T. Groth: Provenance: An Introduction to
Moreau
PROV. Synthesis Lectures on the Semantic Web: Theory and
Technology, Morgan & Claypool Publishers 2013, ISBN
9781627052214, pp. 1-129
https://crowdflower.com/blog/2013/12/crowdsourcing-at- Matteo Venanzi
scale-shared-task-challenge-winners/
Alan, A., et al. (2014). Mixed-initiative electricity tariff
switching for dynamic environments. AAMAS 2014
Simpson, E., et al. (2013). Using a Bayesian Model to
Combine LDA Features with Crowdsourced Responses. In ,
the Twenty-First Text REtrieval Conference (TREC 2012)
Alper Alan, Enrico
Costanza, Gopal
Ramchurn
Gopal Ramchurn,
Edwin Simpson,
Steve Reece
http://veri.ly/
Enrico Costanza
Further details of these applications can be found at http://www.orchid.ac.uk/technology-genres/citizenscience, http://www.orchid.ac.uk/technology-genres/disaster-response/, http://www.orchid.ac.uk/technologygenres/smart-grid/.
29
APPENDIX C - DELIVERABLES
Date
Deliverable Title
Type
07/01/2014 D1.1
Flexible forms of control and influence
Report
14/01/2014 D3.1
Learning and Inference in Decentralised
Report
Systems of Mixed Humans and Agents
16/01/2014 D5.1
User experience of crowd sourcing biodiReport
versity data
05/02/2014 D2.1
Models of Consent in Autonomous Systems Report
18/02/2014 D3.1
Referral Incentives in Crowdfunding
Report
19/02/2014 D5.1
New online mechanisms for settings with
Report
probablistic supply
11/03/2014 D4.3
Transformations and patterns discovery in Report
provenance graphs
24/03/2014 D5.2
Supporting the work of the energy advisor Report
24/03/2014 D1.1
Computational planning support of disaster Report
responders in a mixed-reality game probe
09/06/2014 D1.1/1.2
Report on applying decentralised coordina- Report
tion and planning algorithms to disaster response scenarios with human-agent teams.
25/06/2014 D1.1
A new information gathering framework for Report
co-ordinating UAVs in finding victims in disaster scenarios.
15/07/2014 D5.1
Applying decentralised hard/soft data fusion Report
algorithms to crowd sourced data
01/09/2014 D3.1
Development of a coordination framework Report
and presentation of the results obtained
24/09/2014 D5.1/5
Knowledge Transfer Status Report 4
Report
24/09/2014 D1.1
Multi-User Experience of Occupancy within Report
a Human-Agent Collective
30/09/2014 D3.1
Theoretical properties and empirical eval- Report
uation of referral incentives
01/10/2014 D2.1
Collaboration between heterogeneous
Report
agent groups in SAR
01/10/2014 D5.2
Developing appropriate preheating control Report
methods
01/10/2014 D1.1
Consumer oriented residential demand re- Report
sponse
22/10/2014 D5.3
Understanding the work of disaster reReport
sponse: a field study
27/10/2014 D4.4
Trust-based algorithms for fusing
Report
crowdsourced estimates of continuous
quantities
03/12/2014 D5.2
Deployment of Electric Vehicle Charging
Report
Simulation
03/12/2014 D1.1
Multi-UAV Control - a lab student using
Report
fNIRS
30/12/2014 D3.1
Report on new task allocation algorithms
Report
for the budget-limited task allocation problem in crowdsourcing systems
30
Author
Feng Wu
Steve Reece
Stuart Moran
Stuart Moran
Victor Naroditskiy
Sebastian Stein
T Dong Huynh
Joel Fischer
Joel Fischer
Feng Wu
Chris Baker
Steve Reece
Alexandros Zenonos
Dave Nicholson
Stuart Moran
Victor Naroditskiy
Zoltán Beck
Thanos Panagopoulos
James Holyhead
Joel Fischer
Matteo Venanzi
Sebastian Stein
Joel Fischer
Long Tran-Thanh
ORCHID Fourth Annual Report 2014
APPENDIX D — STATEMENTS OF WORK
All research undertaken in the project is covered by a statement of work. In this section, the
statements that were active at some point in the report period are listed.
Commenced Researcher/PhD student
14/01/2014
Edwin Simpson
31/01/2014
Amr Hussain
31/01/2014
Chris Baker
10/02/2014
Matteo Venanzi
10/02/2014
Elliot Salisbury
10/02/2014
11/02/2014
Title
Work Area and Packages* Location
WA1, WA3, WA5
Interactive Informative Feature Learning in HACs
Scheduling and Coordination of Telescopes for
Transient Events Follow-Ups
WA1
Maximising Information and Utility Value in
WA1: WP1. WA2: WP1. WA5:
Heterogeneous Teams of UAVs and UGVs
WP3
Bayesian Aggregation Models for Crowdsourced SpatialTemporal Data
WA4:WP4, WP6
Oxford
Southampton
Southampton
Southampton
Feng Wu
Crowd Sourcing in Disaster Response Scenarios
Efficient Algorithms for Human-Agent Coordination in
Disaster Response Scenarios
WA5:WP1, WP3
WA1:WP1, WP2, WA5:WP1,
WP3
Matthew Pike
The role of fNIRS in HCI evaluation
WA1:WP1, WA2: WP3, WA5:WP1,
Nottingham
WP5
11/02/2014
Alexandros Zenonos
Development of coordination mechanisms and
WA3:WP3, WA4:WP4, WA5:
incentive strategies to tackle crowdsourcing problems WP3
18/02/2014
Thanos Panagopoulos
20/02/2014
Sasan Maleki
20/02/2014
James Holyhead
27/02/2014
Mark Ebden
01/03/2014
Victor Naroditskiy
03/03/2014
Southampton
Southampton
Southampton
Towards Optimal House Preheating
Fair Payoff Division Mechanisms for Scenarios with
Large Number of Agents
Consumer Oriented Residential Demand Response in
the Smart Grid
WA1:WP1, WA5:WP2
Southampton
WA2
WA1:WP1, WP2, WA5:WP1,
WP3
Southampton
WA4:WP2, WP5, WA3:WP1
Oxford
WA3:WP1,WP2
Southampton
Tom Nickson
Development of Provenance-based Incentives in HACs
Referral Incentives in Citizen Science and Disaster
Response
Decentralised route planning for real world Bayesian
Optimisation using UAVs
10/03/2014
Stuart Moran
Patterns and Occupancy in Human-Agent Collectives
26/06/2014
Joel Fischer
27/06/2014
Justin Bewsher
02/07/2014
Long Tran-Thanh
HACs in the Wild
Prognostic changepoint and fault detection in
electrical signals
Budget-Limited Task Allocation in Crowdsourcing
Systems
16/07/2014
Zoltan Beck
08/08/2014
Steve Reece
*Work Areas
Agent Collaboration in Search and Rescue
Decentralised Heterogeneous Bayesian Classifier
Combination
Southampton
WA1, WA2, WA5
Oxford
WA1: WP1, WP2, WA5: WP3,
WP4, WP5.
Nottingham
WA1 : WP1, WP2, WP3. WA5:
WP2, WP3, WP5
Nottingham
WA1 WA2 WA3 WA5
Oxford
WA3: WP1, WA5: WP2
Southampton
WA2 : WP1. WA5 : WP3
Southampton
WA3: WP1. WA5: WP1, WP3, WP4
Oxford
Work Packages
WP1 Flexible forms of control and influence, WP2 Awareness and attention in HACs, WP3 New forms
of agency
WP1 Decentralised coordination under uncertainty, WP2 Human agent interaction in coordination,
WA2 Agile Teaming
WP3 Integration and demonstration
WP1 Decentralised inference and control, WP2 Bayesian Games and weak control, WP3 Inference over
WA3 Incentive Engineering
dynamic collectives
WP1 Accountable information infrastructure, WP2 Provenance Model and reasoning, WP3
Architectures for Provenance/Trust, WP4 Representing human actions in crowdsourcing, WP5 User
WA4 Accountable Information Architecture exploitation of accountability info, WP6 Trust and information fusion models
WP1 Defining scenarios, WP2 Energy systems applications, WP3 Disaster management applications,
WA5 Applications
WP4 Monitoring of new application areas, WP5 Study of ethics and privacy issues in HACs
WA1 Flexible Autonomy
If you would like copies of any (or all) statements of work please email info@orchid.ac.uk
31
APPENDIX E — KEY MEASURES OF SUCCESS
Research Quality

Number of publications: 226 @ 31 January 2015

Number of top-tier publications (Core A* conferences and top journals GEB, AIJ,
JMLR, JAIR) 169
o 41 journal articles
o 128 conference papers

Number of citations (based on Google Scholar): 2168

Prizes and Awards (including best papers): 23
o
ICICA 2014 Best paper – Levenberg et al
o
Buildsys 2014 Best Demo 2014 – Parson et al
o
World 3rd in International Microelectronics Competition 2014 - Beck
o Winner Google Impact Challenge 2014 – Roberts et al.
o WWW 2014 Best Paper Runner-up - Venanzi et al
o BuildSys 2013 Best Paper - Rogers et al.
o Winner HCOMP 2013 Crowdsourcing at Scale Shared Task Challenge - Venanzi et al
o Winner 2013 British Gas Connecting Homes Start-up Competition – MyJoulo
o EPSRC Doctoral Award 2013 – Parson
o UbiComp 2013 Best Paper (Honourable Mention) – Luger
o IJCAI 2013 Outstanding Student Paper - Zilli and Parson
o CHI 2013 Best Paper - Rodden et al
o Dean's Award for Research Excellence University of Southampton 2013 – Rahwan
o
AAMAS 2013 Best Student Paper Award – Alam et al
o BCS Distinguished Dissertation Award (honourable mention) – Tran-Thanh
o ECCAI 2012 Artificial Intelligence Dissertation Award (honourable mention) – Tran-Thanh
o Runner-up TREC Challenge 2012 – Simpson et al
o Winner TAG Challenge 2012 – Naroditskiy et al
o AAAI 2012 Honourable Mention Award – Tran-Thanh et al
o ECAI 2012 Nomination Best Student Paper Award –Tran-Thanh et al
o IEEE International Conference on Data Mining 2012 Best paper award - Reece et al
o NIPS 2012 Workshop on Human Computation for Science and Sustainability Best
Contribution Award - Simpson et al
32
Research Impact

Collaborations formed with academic groups outside ORCHID: 25
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

Hans Fangohr, CDT in Next Generation Computational Modelling (Southampton)
Niki Trigoni, CDT in Autonomous Intelligent Machines and Systems (Oxford)
Sven Seuken, University of Zurich
Ruth Retie, Kingston Business School
Mike Hazas, Lancaster University
James Scott, Microsoft Research
Abigal Sellen, Microsoft Research
Keith Edwards, Beki Grinter, Georgia Tech
Kathy Willis, Oxford Biodiversity Institute, Kew Gardens
Derek McAuley, Horizon
Mario Berges, Carnegie Mellon University
Mark Girorlami, UCL,
Muffy Calder, Glasgow University
Kepler pipeline team, NASA
Prof Rob Fender, University of Oxford (LOFAR & SKA)
Tao Qin, Microsoft Research Asia
Peter Key, Microsoft Research Cambridge
Botond Szabo, CREST Paris
Jia Yuan Yu, IBM Research Dublin
Suzanne Aigrain, Exoplanet research group, Oxford
Aris Karastergiou, Radio Astronomy group, Oxford
Marian Dawkins, Animal Behaviour Group, Oxford
Ioannis Psorakis, Doyne Farmer, Institute for New Economic thinking, Oxford
Simon Benjamin, Quantum Computing Group, Oxford
Interaction Design, Chalmers University of Technology, Gothenburg, Sweden
Systems Group University of Cambridge
Warwick Management Group
Edinburgh School of Design
Royal College of Art
Invited Keynotes: 11
o
o
o
o
o
o
o
o
o
o
o
ICAART 2015
Big Data, Big Models, Big Deal - CSML 2014
MaxEnt 2013
ECML 2013
EPSRC student future workshop
AAMAS HADM 2013 workshop
AAMAS 2012
Computational Sustainability 2012
UKRC Digital Economy All Hands Meeting 2012
Royal Society “time series modelling for physical sciences”
NIPS workshop on crowd-sourcing 2012
33
Nick Jennings
Steve Roberts
Steve Roberts
Steve Roberts
Tom Rodden
Tom Rodden
Alex Rogers
Alex Rogers
Alex Rogers
Steve Roberts
Steve Roberts

Workshops in Major Venues: 24
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

Massive MAS, 2015
HAIDM, AAMAS 2015
Ubicomp 2014
HAIDM, AAMAS 2014
NordiCHI 2014
RSS 2014
Provenance Week 2014
IPAW 2014
Provenance Analytics Workshop 2014
Sigcomm 2013
Ubicomp 2013
AMEC 2013
NIPS 2013
HAIDM, AAMAS 2013
OPTMAS 2013
ATES, AAMAS 2013
ECSCW 2013
ATES, AAMAS 2012
NIPS 2012
NCAF 2012
TADA/AMEC 2012
HAIDM, AAMAS 2012
OPTMAS 2012
OPTMAS 2011
ATES, AAMAS 2011
Long Tran-Thanh
Gopal Ramchurn
Tom Rodden/Ewa Luger
Gopal Ramchurn
Joel Fischer
Steve Roberts
Luc Moreau
Luc Moreau
Luc Moreau/Dong Huynh
Tom Rodden
Tom Rodden
Seb Stein
Mike Osborne
Gopal Ramchurn
Gopal Ramchurn
Alex Rogers
Joel Fischer
Alex Rogers
Mike Osborne
Steve Roberts
Seb Stein
Gopal Ramchurn
Gopal Ramchurn
Gopal Ramchurn
Alex Rogers
Research Visits from other organisations to work with ORCHID: 23
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
Filippo Bistaffa
Sofia Ceppi
Renzo de Nardi,
Mathijs de Weerdt
Geradline Fitzpatrick
Keiichiro Hayakawa
Francois Jean
Marina Jirotka
Marc Langienrich
Fabio Maffioletti
Areej Malibari
Robert Niven
Toni Penya-Alba
Riccardo Reffato
Avi Rosenfeld
Simo Sarkka
Mike Shann
Arfon Smith
Piotr Szczepanski
Mike Twidale
Twan van Laarhoven
Ioannis Vetsikas
Yair Zick
University of Verona, Verona, Italy
Politecnico di Milano, Italy
UCL, UK
TU Delft, The Netherlands
University of Vienna, Austria
Toyota Central R&D labs
ENSTA-Bretagne, France
University of Oxford, UK
University of Lugano, Switzerland
King AbdulAziz University, Saudi Arabia
CarbonCure
IIIA – CSIC, Spain
Jerusalem College of Technology, Israel
Aalto University, Finland
University of Zurich, Switzerland
Adler Planetarium, Chicago, US
Warsaw University of Technology, Poland
University of Illinois, US
Radboud University, Nijmegen, The Netherlands
IIT Demokritos, Greece
Nanyang Technological University, Singapore
34

•
Extended visits by ORCHID researchers to other organisations: 10
o
Internship at MSR 2014 - Matteo Venanzi
o
Internship at MSR 2014 - Davide Zilli
o
Internship at MIT 2014 – Davide Zilli
o
The Moller-Maersk Institute in Denmark 2013 – Gopal Ramchurn
o
University of Southern California 2013 – Gopal Ramchurn
o
Internship at MSR 2013 - Matteo Venanzi
o
Internship at PlotWatt, 2013 – Oliver Parson
o
Internship at BAE Systems 2013 - Sam Miller
o
University of Verona 2012– Gopal Ramchurn
o
Internship at PlotWatt 2012 – Oliver Parson
Software and standards released: 10
o
Latent force software accompanying JMLR article (http://jmlr.csail.mit.edu/papers/v15/),
2014
o
Smart Home Network (http://www.smarthomeframework.org/), 2014
o
NILMTK: An Open Source Non-intrusive Load Monitoring Toolkit
(https://github.com/nilmtk/nilmtk)., 2014 (40 starred on Github)
o
Literatin via chrome web store. This plug-in allows users to explore the complexity of any
text found within a webpage.
(https://chrome.google.com/webstore/detail/literatin/igpbgncfceidjgcjddcdamjgbpdcmbnl?
hl=en-GB), 2014 (1,172 downloads)
o
Infer.NET Community-based Bayesian Classifier Combination
(http://research.microsoft.com/en-us/um/cambridge/projects/infernet/), 2013 (550
downloads)
o
Underpinning machine learning software developed for ORCHID-funded mobility analysis
work (https://pypi.python.org/pypi/vbihmm/), 2013 (1,351 downloads)
o
PROV Standard (http://www.w3.org/TR/prov-overview/), 2013
o
ProvToolbox (http://lucmoreau.github.io/ProvToolbox/), 2013
o
ProvStore (https://provenance.ecs.soton.ac.uk/store/): a public online repository for
provenance documents, 2012 (Active users (those with data): 33, Total number of
document stored: over 13K)
o
Prov Python library version 1.1.0 (https://pypi.python.org/pypi/prov), 2012 (46,000
downloads)
35
Industrial Impact


Number of technologies pulled through: 8
o
Cicada app and analysis expanded and applied by Centre for Ecology and Hydrology to
cover all UK opthoptera.
o
Human mobility analysis technology applied by BAE Systems in security and counterterrorism context.
o
DEMS (Portsmouth BAE): implemented algorithms to monitor and control loads in micro
grids.
o
IBCC technology applied by BAE Systems in context of Cyber and Aircraft fault
classification.
o
PyIBCC software supplied to Zooniverse for use with citizen science projects (planet
hunters, snapshot Serengeti, space warps).
o
Crowdsourcing Module developed with Microsoft Research being tested on an
experimental platform used by Bing researchers prior to further development and
application.
o
Joulo Ltd founded. Licensing analysis algorithms from the University of Southampton.
Acquired by Quby early 2015.
o
AgentSwitch spin off in progress. Working with Hampshire County Council for community
energy purchasing programme.
Patents/Licenses given out: 6
o
“Crowdsourcing System with Community Learning”, Inventors: Matteo Venanzi, John
Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi, MS 340522.01, January 2014
o
“Adaptive Task Assignment”, Inventors: Matteo Venanzi, John Guiver, Pushmeet Kohli,
MS355469.01, August 2014
o
“Sensor Data Processing”, Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson
(BAE), Patent No: PCT/GB2012/000481, 2012
o
“A method for inferring an output distribution using a Gaussian Process (on kernelised
data)”, Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson (BAE) Patent No:
GB1109209.5., 2011.
o “A method for inferring a relevance for a data source using a regression technique (on
kernelised data), Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson (BAE),
Patent No: GB1109210.3
o “A method for inferring the relevance for a pre-processing technique using machine
learning”, Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson (BAE) Patent No:
GB1109206.1, 2011.
36

Number of external organisations engaged with: 42
o
AHL
○ ASUR
o
Avon Fire & Rescue Service
○ Babcock
o
BAE Systems
○ BBC
o
Blue Bear
○ BP
o
British Gas
○ Cancer Research UK
o
Centre for Ecology and Hydrology
○ Centre for Sustainable Energy
o
Department of Energy and Climate Change
○ DSTL
o
EDF
○ Energy Savings Trust
o
Energy Technologies Institute
○ EPSRC
o
ESA
○ Envitia
o
Google
○ Halo Crisis Management
o
Home Office
○ IBM
o
Kiwi Power
○ Microsoft
o
NASA
○ Navetas
o
Npower
○ PassivSystems
o
PlotWatt
○ QinetiQ
o
Rescue Global
○ Schlumberger
o
Secure Meters Ltd.
○ Toshiba Research Lab
o
Toyota Central R&D Labs, Inc.
○ UKERC
o
Urban Angel
○ Ushahidi
o
YouGov
37
Outreach

Media mentions (including BBC, New Scientist, The Guardian, WIRED and Economist ): 88

Deployments of trial systems (number of participants):
o
o
o
o
o
o
o
o
o
o

MyJoulo 2245 users
Cicada 5000 users
AgentSwitch 3000 users
AtomicORCHID 82 users
TariffAgent 20 users
Apocalypse of MoP 850 users
Outrun Cancer 1577
Veri.ly 70 users
Demand Response System (Agent B) 20 users
Fukushima Heatmap 7,064 page views
Demonstrations and presentations at public events: 16
Demo/Presentation
Where
Disaster response
technologies
New Forest Cicada
British Science Festival, Birmingham
Whe
n
2014
The Wood Fair, New Forest
2014
ORCHID Industry Day
London
2014
Autonomous Systems
Showcase
HACs in the Smart Grid
Southampton
2014
Webinar
2013
Disaster response
technologies
New Forest Cicada
First Lego League Event
2013
BBC Summer of Wildlife, Birmingham
2013
New Forest Cicada
British Science Festival, Newcastle
2013
I’m a Scientist…
Online
2013
ORCHID Industry Day
Farnborough
2013
New Forest Cicada
Big Bang Fair, Solent Region
2013
New Forest Cicada
2013
ORCHID Overview
University of Southampton Science and
Engineering Day
Big Bang Science and Engineering Fair,
Birmingham
Farnborough Air Show
AtomicORCHID
University of Southampton Open Day
2012
CARGO
igFest, Bristol
2012
ORCHID Overview

2012
2012
Master classes, learning and classroom materials provided: 9
o
o
o
o
o
o
o
o
o
Researcher Links Workshop for British Council on Disaster Response (Mexico), 2015
Incentive mechanisms for electric vehicle charging for Toyota Central R&D Labs, 2014
Designed a learning activity and video based on Outrun Cancer for the University of
Southampton MOOC on Digital Marketing, 2014
NILM 2014 @ London (European venue for discussing non-intrusive load monitoring
research from industry and academia)
Keynote on energy disaggregation at Human-Centered Energy Management workshop,
2014
Training course on energy disaggregation for Telekom Malaysia R&D, 2014
Training course on energy disaggregation for British Gas Connected Homes, 2014
Police Workshop on Provenance (NCA, South East Regional Cyber Crime Unit), 2014
Provenance Tutorials (Southampton and Nottingham), 2013
38
Research Capacity Building

Number of PhDs graduated: 10
o
o
o
o
o
o
o
o
o
o



Jan-Peter Calliess
Ewa Luger
Matteo Venanzi
James McInerney
Sam Miller
Oliver Parson
Muddasser Alam
Edwin Simpson
Oleksandr Pryymak
Francesco Delle Fave
2014
2014
2014
2014
2014
2013
2013
2013
2013
2012
Oxford
Nottingham
Southampton
Southampton
Southampton
Southampton
Southampton
Oxford
Southampton
Southampton
PDRAs appointed to faculty positions: 7
o
Long Tran-Thanh
2015
University of Southampton
o
Joel Fischer
2014
University of Nottingham
o
Nadia Pantidi
2014
University of Cork
o
o
o
o
Feng Wu
Talal Rahwan
Sebastian Stein
Michael Osborne
2014
2014
2013
2012
University of Science and Technology of China
Masdar Institute of Technology
University of Southampton
University of Oxford
Academic promotions: 3
o
o
Michael Osborne (Lecturer to Associate Professor) (2014)
Nick Jennings (Professor to Regius Professor) (2014)
o
Alex Rogers (Reader to Professor) (2012)
Research interns employed on ORCHID: 12
o
o
o
o
o
o
o
o
o
o
o
o
Petre Andre
Andrei Petre
David Smith
Debanjan Ghosh
Alex Botev
Po Ting Tse
Quan Tran
Sam Millar
Jack Flann
Po Ting Tse
Quan Tran
Sam Millar
2011
2012
2012
2012
2013
2013
2013
2013
2014
2014
2014
2014
39

Number of follow on grants:
Follow on Grant
25
Funder
When
Where
Value(£k)
Southampton/CSE/
Open System
solutions Ltd/Local
authorities
Southampton/
KiwiPower
Southampton
494
Aperio: Low cost façade management in
naturally ventilated buildings
EPSRC
2014
Adaptive Demand Response
TSB
2014
Researcher Links Workshop Grant for
Disaster Response (Mexico)
British Council
2014
Casma
ESRC
2014
Nottingham
500
Biosound/Mosquito sensor project
Google
2014
Oxford/ Southampton
500
APERIO: Low cost facade management in
naturally ventilated buildings
EPSRC
2014
Southampton
500
Intelligent SME energy management and
trading with ancillary services.
EPSRC / TSB
2014
Southampton
200
Thermal Modeling and Inference with
HomeOS
ASUR: Human-Agent Collaboration for
Multi-UAV Control
EPSRC / MSR
2014
Southampton
60
DSTL
2014
Southampton
50
KTS with BAE Systems
EPSRC
2014
50
CharIoT: Leveraging the Internet of
Things to Reduce Fuel Poverty
EPSRC
2014
Southampton/ BAE
Systems
Southampton/
Nottingham/ CSE
Human Mobility Analysis and Anomaly
Detection
CDE
2013
Southampton/ BAE
Systems
77
Creating the energy for change
EPSRC
2013
Southampton/
Nottingham
1076
ORCHID Artist in Residence
Leverhulme Trust
2013
Southampton
15
Domestic energy management
HEIF Fund
2013
Southampton
15
Referral incentives in crowdsourcing
British Academy
2013
Southampton
8
Referral incentives in crowdsourcing
FSHS Strategic
Interdisciplinary
Pilot study with
Research
Kingston
Development Fund
EPSRC
2013
Southampton
7
2013
50
2013
Southampton/
Kingston Business
School
Southampton
1316
Autonomous behaviour and learning in an
uncertain world
Prototyping open innovation models for
ICT-enabled manufacturing in food and
packaging
Home Hub-of-all-Things as platform for
multi-sided market powered by internet-ofthings
User centred networking
EPSRC
2013
Oxford
1200
EPSRC
2013
Nottingham
1827
EPSRC
2013
Nottingham
982
EU
2013
Nottingham
3947
Automated assessment of broiler chicken
welfare using optical flow patterns in
relation to behaviour, disease risk and
production
A robust toolbox for exoplanet analysis
BBSRC
2013
Oxford
738
Leverhulme Trust
2013
Oxford
220
Biosound
James Martin
School
2013
Oxford
225
TOTAL
14634
Transforming Feedback –Interactive,
Practice-level Visualisation of Electricity
Consumption
Multiagent collectives for sensing,
250
50
277
autonomy, intelligence and control
40
APPENDIX F — PROJECT MANAGEMENT DATA
The programme started on the 1 January 2011 and received £5.5M in funding from EPSRC. This was supplemented by £3.67M of
additional investments: 24 PhD studentships provided by the partner universities at a cost of £1.35M; a £1M commitment from BAE
Systems to an aligned programme to pull through the results of ORCHID into their corporate research programme and relevant
Business Units; a £400k commitment from Secure Meters to a parallel programme in which they allocate a full-time engineer to pull
through the results of ORCHID into their commercial activities; a £255k contribution in kind from BAE Systems and Secure Meters for
their senior managers and scientists to fulfill their various research and advisory roles associated with ORCHID; a £320k contribution
from ACFR to support engagement with their field robotics capability; and a £250k contribution from Southampton’s Strategic
Development Fund to fund a research lecturer (Dr S Stein) and co-fund a research fellow (Dr T Dong Huynh) and to interface with
their University Strategic Research Groups.
Spend Profile
Actual December 2014
Directly incurred
Staff
Travel
Equipment
Other
Seedcorn
Sub total
Exceptions
Student Maintenance 100%
Student Fees
Sub Total
Total 100% (includes directly allocated costs)
Southampton
Total Budget Actual to date
£1,187,606.53 £1,076,828.89
£266,710.00 £283,587.74
£24,500.00
£9,469.23
£145,114.65 £96,544.11
£200,000.00 £163,701.21
£1,823,931.18 £1,630,131.18
£84,777.00 £84,777.00
£21,340.50 £21,340.50
£106,117.50 £106,117.50
£3,587,361.49 £2,657,762.86
Oxford
Remaining
£110,777.64
£548,728.02
£402,508.40
-£16,877.74
£84,168.00
£57,747.86
£15,030.77
£0.00
£0.00
£48,570.54
£28,755.68
£28,755.68
£36,298.79
£0.00
£0.00
£193,800.00
£635,149.70
£489,011.94
£0.00
£0.00
£0.00
£0.00
£159,176.25
£102,264.57
£0.00
£159,176.25
£98,664.57
£929,598.63 £1,728,062.67 £1,302,311.07
Nottingham
Remaining
£146,219.62
£523,192.70
£437,929.00
£26,420.14
£67,334.40
£82,655.57
£0.00
£0.00
£0.00
£0.00
£45,589.28
£43,565.27
£0.00
£0.00
£0.00
£172,639.76
£636,116.38
£564,149.84
£0.00
£127,165.50
£107,608.71
£56,911.68
£32,010.75
£26,319.66
£56,911.68
£159,176.25
£133,928.37
£425,751.60 £1,415,893.60
£999,038.16
Remaining
£85,263.70
-£15,321.17
£0.00
£2,024.01
£0.00
£71,966.54
£19,556.79
£5,691.09
£25,247.88
£416,855.44
Southampton and Nottingham have overspent on their travel budgets, as researchers have actively collaborated on the development
of the main HAC this year. The budgets will be actively managed to ensure that researchers can continue to actively work together
as ORCHID draws to a close.
41
Staff Employed
Oxford
PI
CI
CI
CI
CI
Steve Roberts
CI
Mike Osbourne
CI
Researcher
Researcher
Researcher
Researcher
Researcher
Researcher
Researcher
Researcher
Researcher
Researcher
Researcher
Steve Reece
Researcher
Mark Ebden
Researcher
Edwin Simpson
Researcher
Researcher
Researcher
Researcher
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student Jan Calliess
PHD Student Justin Bewsher
PHD Student Tom Gunter
PHD Student Tom Nickson
PHD Student Mark Mcleod
PHD Student Jonathan Downing
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
PHD Student
Technical Staff
Technical Staff
Technical Staff
Project Administrator
Knowledge Transfer Officer
Nottingham
Associated Researchers*
Derek McAuley
Brian Logan
Maria Polukarov
Enrico Costanza
Oliver Parson
Siddhartha Ghosh (left 30/09/2013)
Napta Gupta
Amir Sezavar Keshavarz
Obaid Malik
Associated
PhD Student*
Southampton
Nick Jennings
Luc Moreau
Alex Rogers
Gopal Ramchurn
Tom Rodden
Seb Stein
Talal Rahwan
T Dong Huynh
Victor Naroditskiy
Feng Wu
Greg Hines
Meritxell Vinyals
Long Tran Thanh
Ruben Stranders
Matteo Venanzi
Joel Fischer
Nadia Pantidi
Khaled Bachour
Stuart Moran
Sasan Maleki
Amr Hussein
Chris Baker
Hale Harding
Zoltán Beck
Alexandros Zenonos
James Holyhead
Elliot Salisbury
Thanos Panagopoulos
Radu Pruna
Jhim Verame
Wenchao Jiang
Daniela Dybalova
Robert Spencer
Leigh Clark
Richard Wetzel
Chaoyu Ye
Matthew Pike
Yukki Ikuno
Michele Roncalli
Andrei Petre
Angela Westley
Dave Nicholson
Start date
01/01/2011
01/01/2011
01/01/2011
01/01/2011
01/01/2011
01/10/2011
01/01/2011
01/01/2011
01/01/2011
12/05/2011
01/04/2011
14/10/2011
01/09/2011
01/06/2012
01/11/2011
01/09/2011
21/07/2014
01/01/2011
01/01/2011
01/10/2013
01/04/2011
01/08/2011
01/01/2012
01/02/2012
01/10/2011
26/09/2012
26/09/2012
26/09/2012
04/03/2013
01/10/2013
01/10/2013
01/10/2013
01/10/2013
01/10/2014
01/10/2014
01/10/2011
01/10/2012
01/10/2012
01/10/2012
01/10/2013
01/10/2014
26/09/2011
01/10/2011
26/09/2011
01/10/2012
01/12/2012
01/01/2013
01/01/2013
02/06/2014
01/10/2012
18/06/2013
07/03/2011
05/05/2011
End date
31/12/2015
31/12/2015
31/12/2015
31/12/2015
31/12/2015
31/12/2015
31/12/2015
31/12/2015
31/12/2013
30/12/2015
30/09/2014
30/06/2014
11/10/2013
31/07/2012
31/12/2015
29/02/2012
31/12/2015
31/12/2014
31/12/2014
31/12/2015
31/03/2014
31/07/2014
31/01/2015
31/10/2014
30/09/2014
31/03/2016
31/03/2016
31/12/2013
03/09/2016
30/03/2017
30/03/2017
30/03/2017
30/03/2017
30/03/2018
30/03/2018
30/09/2013
30/09/2015
30/09/2015
30/09/2015
30/09/2016
30/09/2017
25/09/2014
30/09/2014
25/09/2014
30/09/2015
30/11/2015
31/12/2015
31/12/2015
31/12/2015
24/06/2013
31/10/2013
06/03/2016
31/12/2015
*Associated researchers and students are those who have contributed to the project, but are not directly funded by it.
42
www.orchid.ac.uk
info@orchid.ac.uk
+44 (0) 23 8059 9512

the Annual Report

Transcription

Similar documents

Orquidia Violeta is sewn singularly in Silverlake by Orquidia Violeta

All Leis are $25, or two for $45 - Tamalpais Union High School District

Using indicator species for biodiversity decline

3 Calendar of Events - January to March 2015

Self Drive Wildflower Tours within the Shire of Kojonup Self Drive

Vanda Miss Joaquim Bamboo Orchid Tiger Orchid

Stanhopea Culture How to Grow These Bizarre Beauties

St. Augustine Orchid Society What is an Orchid Species?

ROYAL PARKS HALF MARATHON SUNDAY 12 OCTOBER 2014 GUARANTEED PLACE APPLICATION FORM

IN Bud