\documentclass[link]{IWCOMP}
\copyrightyear{2012}
\DOI{xxxxxx}
\begin{document}
\title[CBRNE Teamwork and Communication]{Probing PROBE: A Field Study of an Advanced
Decision-Support Prototype for Managing Chemical, Biological, Radiological,
Nuclear and Explosives (CBRNE) Events}
\author{Milica Stojmenovic$^{1,4}$ and Gitte Lindgaard$^{2,3}$}
\affiliation{$^{1}$Carleton University, Ottawa, Canada \\
$^{2}$Department of Psychology, Human Oriented Technology Lab (HOTLab), Carleton University, 1125 Colonel By Drive, Ottawa, ON, Canada K1S
5B6 \\
$^{3}$Faculty of Design, Swinburne University of Technology, 144 High Street, Prahran, Vic. 3181, Australia \\
$^{4}$Present address: Faculty of ICT, Swinburne University of Technology, Prahran, Vict. 3181, Australia}
\shortauthors{M. Stojmenovic and G. Lindgaard}
\begin{abstract}
The purpose of this field study was to investigate teamwork and
communication among event management personnels and to assess the degree to
which PROBE, the advanced prototype they were using to manage a Chemical,
Biological, Radiological, Nuclear and Explosives (CBRNE) simulation, would
adequately meet their needs. The study was a continuation of previous
research conducted in the early phase of the PROBE development. From the
verbatim transcripts, two communication-related analyses were applied to
identify the instances of effective and ineffective communications among the
management team. These revealed that communication was mostly effective.
However, a serious communication breakdown that was observed could have had
fatal consequences. It showed that great care must be taken to ensure the
safety of first responders at all times when evaluating prototypes in the
field. A checklist was generated from the lessons learned in to assist
future researchers to prepare for CBRNE-field studies.
\end{abstract}
\keywords{teamwork; communication; management decision-support system}
\category{management decision-support system; teamwork; communication}
\editorial{Name}
\maketitle
\section{INTRODUCTION}\label{sec1}\vspace*{5pt}
Emergency services responders deal with several types of emergencies as part
of their normal, daily routines \citep{bib18}. For example, the
discovery of an unattended brown parcel left at an airport may lead to its
immediate closure; departures are postponed until further notice, arrivals
are rerouted, people are evacuated and luggage service is halted, all of
which are both time-consuming and very costly \citep{bib37}.
The unpredictable contents of the parcel, and hence the possible
safety-related consequences, make it difficult to manage such an event. The
management of Chemical, Biological, Radiological, Nuclear and Explosives
(CBRNE) events is highly complex; it includes, and goes beyond, crisis
management in search and rescue situations, emergency medical events or
hazard mitigation (Waugh and Streib, 2006). Many professionals representing
different agencies and disciplines are involved in the CBRNE event
management, including police, fire department and emergency medical services
\citep{bib35}. They all need to work closely together as
a team in order to successfully manage a timely response to a potentially
criminal action that may be, or that could escalate to become, very large
and very dangerous.
Amid the chaos in a CBRNE event environment, and with the physical distance
between different responders and responder groups, communication breakdowns
are likely to occur because so much is happening simultaneously. Everyone in
the command post is receiving information from different sources and from
different technologies. Thus, for example, appointed first responders in the
field are updating their team leader in the command post through radio;
members of the command post are coordinating resources with dispatch using
phone and communicating with the incident commander (IC) in face-to-face
interactions. In very large events involving mass casualties, public health
authorities, surrounding hospitals and makeshift field hospitals, other
personnels may also be directly involved in the event management. The wide
variety of communication modes and technologies can lead to
misunderstandings and incorrect actions; some of which could potentially
have fatal consequences.
The CBRNE simulation described here was part of a large project sponsored by
the CBRNE Research and Technology Initiative and carried out in partnership
with a team of experts representing a wide variety of organizations. These
include the Royal Canadian Mounted Police---Canadian Bomb Data Centre, The
National Research Council---Canadian Police Research Centre, the Department
of National Defence---Defence R{\&}D Canada and Director General Nuclear
Safety, Carleton University---HOTLab, Loraday Environmental Products Ltd.,
International Safety Research Inc., responder teams from seven major
Canadian Centres and the AMITA Corporation \citep{bib1}. The present
research is a continuation of the work conducted in the earlier phase of the
development of PROBE used here \citep{bib22}.
The remainder of the paper is organized as follows. The next section
discusses the responsibilities of an emergency response team. This is
followed by CBRNE management and structure. The introduction then concludes
with a brief outline of the evolving software used in this CBRNE simulation.
The theoretical section begins with a discussion of distributed cognition
and cognitive ethnography. This is followed by a discussion of teamwork and
communication. Next, two analysis methods used in this paper are discussed,
concluding the theoretical section. The results of the two analysis methods
are presented thereafter, followed by the general discussion and conclusion
sections.
\subsection{Responsibilities of an emergency response team}\label{subsec1.1}
CBRNE emergency response teams typically include EMS, teams comprising
different police specialists and teams of fire fighters including hazardous
material technicians (hazmat). Each team has very specific and different
responsibilities that all have to come together in a coordinated fashion so
as to effectively manage the adverse event. All responders receive
domain-specific training in addition to specialized CBRNE training. The role
of the EMS team is to take the vital signs of CBRNE first responders who are
about to enter the so-called `hot zone' containing the offending agent(s) or
object(s)\break \citep{bib24,bib12}. Vital signs usually include
heart rate, blood pressure, body temperature and respiration rate. The EMS
team also performs triage, reporting the types of symptoms casualties,
experience to their command post-team leader, and they assess the severity
of injuries as well as forwarding casualties to nearby hospitals for
treatment once they have been decontaminated. The police team consists of
bomb technicians, the forensic identification section (also referred to as
FIS or Ident) and generalists. Bomb technicians are included if it is
suspected that explosives are involved. Their job is to disable the bomb to
prevent it from detonating. Ident officers are responsible for collecting
and cataloguing evidence from the scene to document any proof, in case the
event is likely to result in a future criminal court case \citep{bib31}. Generalists help to organize the refuge of hostages and casualties;
they seal off the entire area and help to keep public order, for example, by
redirecting traffic. The hazardous materials (hazmat) team comprises
firefighters with specialized training. This group deals primarily with
sampling and testing harmful chemical agents enabling their team to identify
and neutralize these. Besides controlling the entry into, and exit from, the
hot zone, the hazmat team is responsible for setting up and managing a
decontamination site to neutralize whatever damage might have been done to
people, equipment and property if a harmful chemical is involved \citep{bib24}. Once the command post members consider the site safe for forensic
processing, and the Ident officers have acquired all the necessary evidence
from the hot zone, the hazmat team's decontaminate and clean the entire
area.
\subsection{CBRNE management and team structure}\label{subsec1.2}
The CBRNE event response team is comprises of a chain of command of
responders with different responsibilities \citep{bib12}. The
structure of each event varies as a function of its size and complexity
\citep{bib28}, and in many situations one person may perform multiple
roles. In the Province of Ontario, Canada, there are up to three levels of
operational response, depending on the magnitude and severity of the
emergency and thus on the number of personnels required to respond
\citep{bib7}. As shown in Fig.~\ref{fig1}, the first responders deal with the
dangerous situation in the hot zone in a hands-on manner. The responder team
typically includes EMS paramedics, generalists, bomb technicians, Ident and
forensic officers, firefighters and hazmat technicians \citep{bib38}.
\begin{figure}[hbt!]
%\centering{\includegraphics{IWCOMP_CE_Samplef01.eps}}
\centerline{\fbox{\vbox to 10pc{\vfill\hbox to 20pc{\hfill\Huge FPO\hfill}\vfill}}}
\caption{Response structure and positioning in the three zones.
FR, first responders.\label{fig1}}
\end{figure}
\looseness=1 In larger events, the first responders report to their Ops officers who, in
turn, report to their commanders, as shown in Fig.~\ref{fig1}. In such cases, Ops
teams representing each of the three agencies are set-up in different
designated areas in the `warm zone', the area directly surrounding the hot
zone but that is not in immediate danger \citep{bib12}. The
decontamination tent is also set up there and bridges to the cold zone. The
Ops officers all work together to create and implement an incident action
plan (Emergency Management Ontario Ministry of Community Safety and
Correctional Services, 2008, 2009), which is a short-term to-do list. Ops
officers manage the first responders in the hot zone and manage the deployed
equipment, and they are responsible for keeping the commanders in the
command post up-to-date \citet{bib7}, \citet{bib8}; \citep{bib24}. In smaller, less
complex events, no Ops officers are required, and members of the command
post adopt the Ops officers' responsibilities instead. The simulation on
which the present research was based include an Ops layer.
In any CBRNE event, the command post is always located in the cold zone or
even further away from the scene, for the safety of the commanders. They are
often also in contact with representatives from other agencies that only
take part in event management under special circumstances (e.g. Health
Canada, Public Safety Canada). The command post-team's objectives are to
coordinate the emergency response to preserve life, maximize safety and
diminish the threat while protecting the affected property, minimizing cross
contamination, as well as preserving and collecting evidence \citep{bib24}.
Commanders approve the incident action plan \citet{bib7},\citet{bib8}; they
oversee their own agency's progress throughout the event response and are
responsible for coordinating their agency's actions with the others. In
addition, commanders are in charge of identifying and resolving response
issues (e.g. a lack of equipment on site), providing advice to responders
and implementing the action plan \citet{bib7}, \citet{bib8}. There are usually
at least three commanders in the command post, one representing each of the
police, fire and EMS agencies. Commanders also determine the level of
personal protective equipment (PPE) that responders and casualties need to
wear in the hot zone.
The IC is in charge of coordinating all participating professionals
\citep{bib26}; \citep{bib35}. The IC communicates with, and
gives orders to, all officers in the event. In addition, he or she can veto
the incident action plan and, in some cases, their approval might also be
necessary. If the action plan is vetoed, the IC is responsible for providing
alternatives \citep{bib14}. Ideally, the IC is even further
removed from the hot zone and separated from the command post for safety
reasons. However, the IC usually spends a great deal of time with the
commanders in the command post.
As the command post-team makes the crucial strategic event management
decisions, it is considered the most important team. Therefore, this
research focused on that team.
\subsection{Brief outline of PROBE and accompanying applications}\label{subsec1.3}
\begin{figure}[!b]
%\centering{\includegraphics{IWCOMP_CE_Samplef02.eps}}
\centerline{\fbox{\vbox to 10pc{\vfill\hbox to 20pc{\hfill\Huge FPO\hfill}\vfill}}}
\caption{An overview of the RTMW software.\label{fig2}}
\end{figure}
PROBE is a CBRNE management decision-support system. Its purpose is to aid
interoperability among the response agencies by recording, storing and
sharing CBRNE event information \citep{bib1}. The user population comprises
predominantly commanders and operations officers involved in the event
management. PROBE provides a suite of CBRNE management applications, CBRNE
databases, standardized forms, automated evidence collection using RFID tags
and information on patient triage. The suite includes applications for
hazmat technicians, for example, the chemical biological response aid, a
large chemical database that includes another application, PALM Emergency
Action for Chemical-WMD (PEAC-WMD). It stores information on chemicals that
help to identify and render CBRNE materials safe \citep{bib1}. To support
the bomb technicians, Socius, a database in which RCMP bomb technicians
enter and store textual and photographic records of incidents involving
explosive devices \citep{bib1}, is already fully functional. Finally, the
Rapid Triage Management Workbench (RTMW) and the Medical Command Post
(MedPost) are designed to support EMS efforts. The RTMW \citep{bib26},
shown in Fig.~\ref{fig2}, tracks casualty information in one central database
allowing all treatment centres, alternate care facilities and hospitals to
work with the same information. The MedPost application, an overview which
is shown in Fig.~\ref{fig3}, is designed to help medical decision-makers gain access
to timely and accurate medical information in their effort to save lives. As
the figure shows, it provides caregivers with a higher level view, helping
them to identify, isolate and manage disease outbreaks and alert public
health communities, as necessary, regardless of the magnitude of the event.
It is thus particularly useful in the face of a potential radiological or
nuclear threat.
\begin{figure}[hbt!]
%\centering{\includegraphics{IWCOMP_CE_Samplef03.eps}}
\centerline{\fbox{\vbox to 10pc{\vfill\hbox to 20pc{\hfill\Huge FPO\hfill}\vfill}}}
\caption{An overview of the MedPost system.\label{fig3}}
\end{figure}
In addition to linking all of these applications, PROBE will also be capable
of supplementing real-time communication across agencies. This is especially
important for the comman-\break ders located in the command post where it is often
very busy and noisy, with multiple radios going simultaneously, and people
coming and going. The mode of communication is predominantly face-to-face
and via radio \citep{bib12}. PROBE will not replace the radios.
Rather, it records and integrates communications and information during a
CBRNE event, which also helps the commanders to produce their incident
report after the event.
\section{THEORETICAL FRAMEWORK: DISTRIBUTED COGNITION AND COGNITIVE ETHNOGRAPHY}\label{sec2}
In the late 1980s, Hutchins examined the phenomenon of shared knowledge and
proposed what he called distributed cognition \citep{bib13}. According to
this framework, knowledge and thought processes are shared between an
individual and the individual's social (i.e. other people) and physical
(i.e. tools and artefacts) environments \citep{bib10}. The aim is to
explain the interactions and exchanges of information between the individual
and these environments. When investigating group work, researchers examine
people's activities, communications and artefact interactions through
detailed ethnographic study \citep{bib32}. Studying the interactions
between people and technology is fruitful for understanding the role and
function of the relevant technology \citep{bib33}; \citep{bib15}.
Hutchins also coined the term `Cognitive Ethnography', in which a
researcher spends a lot of time in the field \citep{bib19}. Cognitive
ethnography provides data that can then be explained by distributed
cognition \citep{bib10}. It assumes that human communication and
activity, including breakdowns, are meaningful and culturally determined.
Cognitive ethnography was useful in the present research \citep{bib6}, as
it allows for analysis of the way artefacts are used \citep{bib40}.
Distributed cognition and cognitive ethnography were therefore employed in
the data collection process.
\subsection{Teamwork and communication}\label{subsec2.1}
Given that different agencies share the management of CBRNE events,
excellent teamwork is essential. Teamwork is the interaction of two or more
individuals working together to accomplish a common goal \citep{bib16}
which, in the CBRNE event, is the management of the crisis
\citep{bib26}. To manage an event effectively, each team member
is assigned specific tasks, much like each agency's responsibilities are
divided among its members. Effective team coordination and collaboration
\citep{bib30} involving extensive information exchange \citep{bib25}
is essential to accomplish those tasks. Communication
enables the planning of actions and the forwarding of updates to alter
action plans \citep{bib9}, and it can also help teams recover
from interruptions \citep{bib27}. Teamwork effectiveness can be measured
by examining communication between team members \citep{bib3}.
Instances of communication can vary in effectiveness. Communication is
effective when the meaning of a message is successfully conveyed from
speaker to listener. Closed loop is one of the most common examples of
effective communication. It comprises three main parts: (i) the speaker
initiates a message; (ii) the intended listener receives, interprets and
acknowledges receipt of the message and (iii) the speaker ensures correct
reception and interpretation of the message \citep{bib34}. In a CBRNE
command post, an instance of closed-loop communication could be something
like this: the hazmat commander tells the EMS commander that there are 20
casualties in the hot zone; the EMS commander responds that he will send the
paramedics in, and the hazmat commander closes the loop by saying `O.K.'.
Closed-loop communication is a good indicator of successful teamwork leading
to successful team performance. For the purpose of this research, instances
of effective communication were operationalized as closed-loop
communication.
Communication breakdowns are defined as faulty verbal interactions and
appear in the forms of ineffective timing (i.e. late), incomplete and/or
inaccurate content and key individuals not being informed \citep{bib20}. Breakdowns are important indicators of teamwork effectiveness and
hence also of team performance. Interruptions are one type of essential
breakdowns in the CBRNE event because they often update team members on the
constantly changing environment, enabling them to fine-tune the management
of their response. Indeed, there would be little or no progress in the
shared understanding and knowledge of the event without interruptions.
However, the effectiveness of the event management could suffer when an
interruption is not managed appropriately, for example, in the event of an
incomplete or inaccurate update, or when an important issue noted earlier
has been forgotten. A command post member whose attention is constantly
shifting between her radio and her command post-team members may not have
all current status information of the event. Information communicated among
commanders may be incorrect, and/or messages may be overlooked, missed or
forgotten, all of which may decrease the effectiveness of the emergency
response.
Open loop is another example of ineffective communication. It can occur when
a speaker initiates a message, for example, a question, which is then either
not received, not interpreted, acknowledged or answered \citep{bib34}. Communication breakdowns such as open looped communications increase
the probability of errors. For the purpose of this research, several types
of ineffective communication were analysed, including ineffective timing,
key individuals not being informed and open-loop communication.
To gain an understanding of teamwork and communication, one first needs to
identify where, when, why and how often different communication breakdowns
are likely to occur, along with other errors in event management most
readily uncovered by examining communication breakdowns. Two communication
aspects were important in this research, namely the types of utterances and
the topics of communication. Communication analysis and content analysis
were therefore applied to the data.
\subsection{Communication analysis}\label{subsec2.2}
Communication analysis involves the categorization of the topic (e.g.
equipment, personnel, etc.) and the type of utterance (e.g. question,
answer, etc.; \citep{bib17}; \citep{bib29}. \citep{bib9}
studied coordination and collaboration among team members in a hospital
environment by focusing their analysis solely on the type of verbal
exchange. This was done here as well. The type of utterance helps to
understand the information flow by demonstrating when information is needed
(e.g. question) and when it is being shared (e.g. update). Sequences of
communication types were examined for effective and ineffective
communications.
\subsection{Content analysis}\label{subsec2.3}
Content analysis is a widely used technique \citep{bib11}. The
communication topic is another aspect necessary for a thorough assessment of
communication and teamwork in the command post. In latent inductive
analysis, a researcher gradually generates topic categories, as they emerge
from the data. Here, evidence of communication breakdowns was uncovered to
understand communication and teamwork in the CBRNE event management
environment.\vspace*{-3pt}
\begin{table*}[hbt!]
%\setlength{\extrarowheight}{6pt}
\processtable{Format of the transcript used for data analysis.\label{tab1}}
{\begin{tabular*}{\textwidth}{@{}lP{5pc}P{11pc}P{6pc}P{15pc}@{}}\toprule
Time & Source & EMS & Police & Hazmat\\\midrule
10:33 &Video File: 141203 &Radio 1 (X): Can I get an update
on XYZ? &Z (Radio 3): It's green and blue &Radio 2 (Y):
Exercise, exercise exercise---Haz responding \\
& &X (Radio): Yea, XYZ has been ordered and is on the way & & \\
10:34 & &Radio 1 (X): Thank you &Right? & \\\botrule
\end{tabular*}}{}
\end{table*}
\section{METHOD}\label{sec3}
\subsection{Participants}\label{subsec3.1}
A total of 14 experts participated in the study, representing 3 hazmat
experts, 3 EMS officers, 3 police officers, 2 PROBE scribes, 2 event
coordinators and 1 software developer for PROBE. Of these, five were in the
command post (one EMS, one hazmat commander and his scribe, the police
department IC commander and his scribe), and three were Ops officers (one
per agency). The scribes' role was to transcribe into PROBE what the
commanders communicated to other members of the response team. While the IC
was in charge of managing the CBRNE event, the event coordinators were in
charge of managing the logistics such as monitoring the progression of the
scenario and planning lunch. Participation in the simulation was a part of
their normal day jobs; permission for the researchers to be present had been
granted \textit{a priori} by all concerned.\vspace*{-3pt}
\subsection{Design}\label{subsec3.2}
Two researchers located in the command post observed the command post-team
members during the simulation, from opposite sides of the room.
\subsection{Apparatus}\label{subsec3.3}
Each researcher was equipped with a video camera (Sony Handycam DCR-SR-300
HDD) and three stationary audio recorders (Olympus WS-311M Digital Voice
Recorder) were located in different areas of the command post in which four
laptop computers were set up (one per EMS, hazmat, police; one for the IC).
Each commander's radio was running at a unique frequency. An additional
radio allowed the commanders to listen to communication between the Ops
officers located in different areas of the warm zone. As far as possible,
these conversations were captured. Verbatim utterance transcriptions were
transferred to NVivo 9.0 for further analyses.
\subsection{Procedure}\label{subsec3.4}
The event organizers first explained to the participants about the purpose
of the researchers' presence and task. Then, participants read and signed
the ethics approved informed consent form before the researchers proceed to
the command post for their observations. All verbal interactions in person,
radio and software communications were recorded. At the end of the event,
all commanders were given debriefing forms, thanking them for participating.
Finally, a briefing session was held for all the expert participants.
\subsection{Data analysis}\label{subsec3.5}
In an effort to reconstruct the entire event, all video and audio recordings
were transcribed \textit{ad verbatim} and merged into a single file to compare activities across
all command post-team members. Recordings were viewed multiple times to
identify and verify the identity of the speakers and listeners of verbal
communications. To focus the analysis, any talk unrelated to the event
management was removed from the transcript.
Table~\ref{tab1} shows the minute-by-minute formatting of the transcript. The
leftmost column shows the time of observations, followed by the source of
the original data (video/audio/notes). The three rightmost columns show the
data obtained from each agency. All utterances were coded digitally. For the
communication analysis, the researcher focussed on the types of utterances
and on the communication topics and contents in the latent inductive content
analysis. Meaning was extrapolated from the categories to identify and
compare instances of effective and ineffective communications. To allow the
calculation of the inter-rater reliability of both analysis methods, another
researcher independently categorized each sentence from a randomly selected
10{\%} of the transcript by the type and content of communication using
Cohen's kappa. This result is reported; any disagreements between the two
raters were settled by negotiation.
\section{RESULTS}\label{sec4}
The results are presented in the following sections. A description of the
event is provided first, followed by the communication analysis findings
reporting instances of effective and ineffective communications. Next, the
latent inductive content analysis results are shown in which the severity of
ineffective communications was assessed. Then, the inter-rater reliability
of the two analysis methods is presented. Thereafter, a summary of
communication and teamwork is presented, and finally the CBRNE simulation
event management goals are outlined.
\subsection{Event description}\label{subsec4.1}
According to the scenario prepared for the simulation, police officers were
said to have found a makeshift lab in one of the on-site storage containers
in the Port of Saint John. This area was labelled the hot zone. A command
post was set-up to manage the emergency response from the cold zone. The
data were divided into two phases, comprising approximately an hour and a
half each, because two separate incident action plans were prepared during
the simulation. Phase 1 was executed as the first group of responders went
into the hot zone to gather information on the severity and magnitude of the
situation. They found hydrochloric acid and potassium cyanide. Later, an
activated bomb was detected as well. Phase 1 therefore ended with the
deactivation of the bomb and the removal of casualties from the hot zone.
Phase 2 involved the planning and execution of the second incident action
plan, for the re-entry of police and hazmat first responders into the hot
zone for evidence collection and clean-up. EMS officers remained on standby
in the warm zone, just in case.
\subsection{Communication analysis results}\label{subsec4.2}
The communication analysis, divided into two steps, focussed on utterance
types. In the first step, each utterance was coded by type, as these emerged
from the raw transcript data. The second step identified the sequences of
effective and ineffective communications.
\subsubsection{Step 1: coding results}\label{subsubsec4.2.1}
Some 15 categories emerged from the transcript, as seen in the leftmost
column of Table~\ref{tab2}. Definitions are given in the middle column, and the
rightmost column gives examples of the utterance types uncovered.
\begin{table*}[hbt!]
\processtable{Communication analysis types of utterances, as they
emerged from the entire transcript.\label{tab2}}
{\begin{tabular*}{\textwidth}{@{}lP{17pc}P{13pc}@{}}
\toprule
Utterance Type &\multicolumn{1}{c}{Definition} &\multicolumn{1}{c}{Example} \\\midrule
Acknowledgment &Responses from the listener of a communication, letting the speaker know that (s)he had received the message &Yeah; ok; roger; 10-4 \\
Answer &Responses specific to questions &It was red \\
Attention Granting &In response to attention requests, letting the speaker know that the listener is paying attention and communication can proceed &Ops (K): Go ahead \\
Attention Request &Demands for awareness, from speaker to listener, precede a communication &HCo (HOp): Operations officer, this is fire in command \\
Clarification Granting &Clarifying or rewording a previous communication &Yea, today \\
Clarification Request &Asking for clarification on a previous communication &Do you mean today? \\
Complaint &Expressions of discontent &I would never use this. \\
& &It's not working for me \\
Explanation &Justifications and reasons for giving a previous communication &Here's why this is important: 'cuz they are going to meet and create an incident action plan \\
Joke &Intended to amuse, important for alleviating stress in the command post &Fire to police: Do you want to let them know that firefighters are awesome? \\
Order &Instructions or commands to further action &IC to HCo: So I am gunna need you to call your fire Ops \\
Question &Requests for information &What colour was it? \\
Repetition &Forwarding newly learned or planned information down the chain of command &HCo to HOps: The incident action plan has been signed \\
Statement &Expressions of ideas or facts &That's all I can tell you \\
Suggestion &Proposals of possible solutions to problems &You might be able to get it by clicking here \\
Update &Providing the most recent information available &Just to let you know, we just found an IED \\
\hline
\end{tabular*}}{}
\end{table*}
There were a total of 1897 utterances in the entire simulation. The volume
of communication was fairly similar in Phases 1 and 2. Anecdotally, there
seemed to be more activity in Phase 1 in which the most dangerous parts of
the event were handled: the chemical was neutralized, victims were found and
removed from the hot zone and the bomb was deactivated. One would have
expected a high frequency of communication, allowing commanders to organize
the response, leaving little need for activity in Phase 2. However, the
commanders were more involved in altering the second incident action plan
for re-entry into the hot zone for clean-up and evidence collection than for
the first incident action plan.
The most frequent utterance types were questions, statements, answers and
acknowledgments.
\textit{Questions} were expected to comprise the most frequent type of utterance because the
intense decision-making efforts require individuals to obtain and share
information. If important information necessary to make the correct decision
about a plan of action is missing, then the only way to proceed is to ask a
question to obtain that information.
\textit{Answers} were also among the most frequently occurring utterance types. If all
communications had been closed loop, the number of questions and answers
should have been almost equal. Instead, several instances of ineffective,
open-loop communication were uncovered. These are discussed later.
\textit{Statements} included all facts and ideas which were verbalized very frequently.
\textit{Explanations} were elaborations on any utterance, raising their frequency as many
utterances such as answers and statements was elaborated upon by
explanations.
\looseness=-1 \textit{Acknowledgments} also occurred frequently, possibly because acknowledgments were universal
responses to all types of utterances, except attention requests and
questions, informing the speaker that the listener had received the message.
Because communication analysis requires categorization of each utterance in
isolation, it leaves the analysis devoid of context, prompting analysis of
utterance sequences, described later.
\textit{Repetitions and updates} occurred with medium-to-low frequency when a responder forwarded newly
learned or planned information down the chain of command. It was anticipated
that more of these would occur between the commanders and the Ops officers,
because newly acquired information would help to plan the response, and the
plan needs to be passed down the chain of command. Contrary to earlier
studies of CBRNE event management \citep{bib37}, the
commanders' main function here was to approve the incident action plans.
This required less updating from the Ops officers because many details were
grouped in the action plans. The commanders discussed these action plans and
forwarded the documents, again grouping many details, thereby reducing the
need for repetitions.
\textit{Clarification requests and grants} occurred quite infrequently. When an intended listener failed to understand
or hear an utterance, such as a question, he would request a clarification.
This resulted either in a repetition of the utterance or in rephrasing it.
Therefore, clarification grants followed only clarification requests. The
number of clarification requests by far exceeded the number of clarification
grants, especially in Phase 2, indicative of open-loop\break communication.
\textit{Orders and suggestions} were rarely observed. As became clear during the data analysis, the
commanders' main role was to communicate to acquire information on
proceedings in the hot zone and to approve the incident action plans. They
acquired information mainly through questions, answers and updates.
\textit{Jokes and complaints} were the two least frequent types of utterances in the entire event.
However, jokes helped lighten the mood in the command post. Complaints were
mainly about PROBE and not related to the event management.
\subsubsection{Step 2: utterance sequence analyses}\label{subsubsec4.2.2}
Since communication analysis requires examination of each utterance in
isolation to categorize communications by type, this method did not lend
itself to identify effective and ineffective communications. To achieve
that, sequences of communication belonging together were therefore
identified and classified separately. A total of 78.9{\%} of all
communication instances (Phase 1: $n = 136$; Phase 2: $n = 133$) were found to
be effective. However, six types of ineffective communication were also
identified as shown in Table~\ref{tab3}. The leftmost column shows the 21.1{\%}
ineffective communications (Phase 1: $n = 39$; Phase 2: $n = 33$). The middle
two columns show the frequency of each type of ineffective communication in
Phases 1 and 2, and the rightmost column shows the total.
\def\z{\phantom{0}}
\begin{table}[!b]
\processtable{Ineffective communications during the CBRNE simulation.\label{tab3}}
{\begin{tabular*}{21pc}{@{}lccc@{}}
\toprule
&\multicolumn{2}{c}{Event phase} & \\\cline{2-3}\\[-11.5pt]
Ineffective communications &\multicolumn{1}{c}{Phase 1} &\multicolumn{1}{c}{Phase 2} &\multicolumn{1}{c}{Total} \\\midrule
Open loop & \z8 &18 &26 \\
PROBE error &14 &\z3 &17 \\
Misunderstanding &\z4 &\z6 &10 \\
Time lag &\z5 &\z4 &\z9 \\
Key individuals uninformed &\z4 &\z2 &\z6 \\
Incomplete information &\z4 &\z0 &\z4 \\
\botrule
\end{tabular*}}{}
\end{table}
\textit{Open-Loop Communications} occurred most frequently, with more than twice the number of these occurring
in Phase 2 than in Phase 1. They comprised unanswered questions, unfilled
clarification requests, requests for attention and unacknowledged
communication. Some of these may have been miscategorized because non-verbal
communications were not recorded, because the video recordings captured
mainly the commanders' backs and their interactions with PROBE, making it
impossible to capture gestures and facial expressions as well.
\textit{PROBE-related issues} comprised two types, namely a lack of familiarity with PROBE and software
shortcomings/mishaps. The main issue was that not all communications were
received via PROBE because the users did not know, or could not recall, how
PROBE's communication application worked. All but one of the participants
using PROBE had taken a 1-day PROBE training session the day before the
event. Apparently, there were too many functions for everyone to recall when
using PROBE for the first time. Communication sent via PROBE first went to
the IC, who then needed to forward it to everyone else. The IC's scribe had
not received PROBE training and did therefore not know of this requirement
or how to forward messages. Another problem was that some screens did not
populate automatically, need to be refreshed manually. The EMS commander,
who did not rely on a scribe, had forgotten this. The Ops officer attempted
to communicate with the EMS commander through PROBE, but the messages were
not received because of the need to refresh the incident report page
manually. Believing that the IC scribe was not forwarding relevant
information, the EMS commander searched in vain for information, relying
heavily on radio communication for a large part of the event response. Once
the PROBE developer reminded her that the screen had to be refreshed to
receive updates, this problem was solved. The EMS commander was then flooded
with the previously missed messages. Although most PROBE functions did work
as intended, the sending and receiving of attached files, such as the
incident action plan, did not. The failure to receive messages sent via
PROBE may be explained in two ways. One is that the users did not remember
how to send messages correctly; the second could be related to PROBE's
network strength. According to the developer, PROBE uses protocols similar
to the internet, but in a private and secure intranet connection.
Apparently, the wireless intranet connection was weak because the laptops
were distributed in the various Ops and command post locations that were
further apart than had been anticipated. In addition, the network hardware
selected for the prototype ultimately had problems when situated within the
vehicles themselves, i.e. the strength of the device was insufficient to
penetrate the steel/insulated walls of the vehicles. This problem can be
solved with a device capable of sustaining a stronger signal and system
tuning. The severity of these problems is addressed in the latent inductive
content analysis section.
As Table~\ref{tab3} shows, the remaining types of ineffective communications occurred
rather infrequently.
\subsection{Latent inductive content analysis results}\label{subsec4.3}
To assess the severity of the ineffective communications noted above, the
latent inductive content analysis focussed on the topic of utterances. It
was also divided into two steps: (i) coding each utterance by topic as it
emerged from the data and (ii) determining the severity of each.
\subsubsection{Step 1: coding}\label{subsubsec4.3.1}
Seven content categories emerged from the transcript are presented in Table~\ref{tab4} with definitions. The leftmost column shows the topic categories; the
middle column shows the definition of each topic; and the rightmost column
shows an example of each topic. All utterances were coded as one of these
utterance topics.
\begin{table*}[hbt!]
\processtable{Content of communication definitions found in the entire CBRNE simulation.\label{tab4}}
{\begin{tabular*}{\textwidth}{@{}lP{18pc}P{15pc}@{}}
\toprule
Topic &\multicolumn{1}{c}{Definition} &\multicolumn{1}{c}{Example} \\\midrule
Action Plan &An utterance was considered to be about the action plan if it had to do with the CBRNE response-planning process &Did you sign the action plan? \\
Communication &The topic of an utterance was considered communication if it was about talking to or contacting others, unrelated to communication done through PROBE &Did she just say something? \\
Equipment &If the utterance was about any CBRNE response tool (except PROBE), then it was labelled as equipment &A level B suit would be sufficient, really \\
Event &If the utterance was about the management of the simulation (unrelated to the CBRNE threat), it was about the event &Lunch will be served on the fly \\
Offending Agent &If the utterance was about the CBRNE threat, then the topic was offending agent &We've just found an IED \\
Personnel &If the utterance was about the staff, then the topic was personnel &I need a paramedic in the hot zone \\
PROBE &If the utterance was about the advanced prototype or an action associated with it, then the topic was PROBE &Scribe: did you get any updates from me there? I tried to send it through there \\
\botrule
\end{tabular*}}{}
\end{table*}
Table~\ref{tab5} shows the number of times a topic was mentioned during the CBRNE
simulation. The leftmost column shows the utterance topic. The middle two
columns show the frequency with which each topic occurred in Phases 1 and 2.
The rightmost column shows the total\enlargethispage{-4pt} frequency of topic utterances.
\begin{table}[hbt!]
\processtable{Latent inductive content analysis category totals from the CBRNE event.\label{tab5}}
{\begin{tabular*}{21pc}{@{\extracolsep{\fill}}ld{3}d{3}d{3}@{}}
\toprule
&\multicolumn{2}{c}{Event phase} & \\\cline{2-3}\\[-11.5pt]
Topic &\multicolumn{1}{@{}c}{Phase 1} &\multicolumn{1}{@{}c}{Phase 2} &\multicolumn{1}{@{}c}{Total} \\\midrule
PROBE &487 &316 &803 \\
Action plan &107 &161 &268 \\
Personnel &92 &68 &160 \\
Event &47 &95 &142 \\
Equipment &53 &62 &115 \\
Communication &25 &54 &79 \\
Offending agent &58 &17 &75 \\
Total &869 &773 &1642 \\
\botrule
\end{tabular*}}{}
\end{table}
\textit{PROBE} was the most talked-about topic throughout, accounting for nearly one-half
of the utterances (48.9{\%}, as seen in Table~\ref{tab5}. It was discussed more
frequently in Phase 1 than in Phase 2, mainly because the users were
familiarizing themselves with it, sharing their first impressions, concerns
and advice on how to use PROBE. The popularity of PROBE suggests that the
command post-team members were not as preoccupied with the CBRNE mission as
had been observed in two previous simulations. We had therefore expected
that discussion would focus on the action plan, equipment and other
response-related issues. As such, PROBE communications were only of marginal
importance to the event outcome. The hazmat commander even chose to forgo a
situation status update to learn more about PROBE instead. A situation
status update is a meeting between the Ops officers and the commanders in
which everyone reports their team's progress. Another reason for this
prominence of PROBE-related utterances was that the Ops officers did most of
the planning, leaving only the approval of the incident action plans to the
commanders, giving the commanders more time to explore PROBE. Because the
commanders also relied heavily on PROBE for communication, it was talked
about with regards to obtaining information from the Ops officers and
forwarding messages down the chain of command. Occasionally, this led to
ineffective communication, the severity of which is discussed later in this
section.
\textit{Planning the CBRNE event management} was the second most frequently observed topic, followed by \textit{personnel}, \textit{event}, \textit{equipment},
\textit{communication} and \textit{the offending agent(s)}. Although the command post-team makes crucial strategic decisions
necessary to manage the event, making planning and revising the action plan
their primary purpose, these accounted only for 16.3{\%} of all
communications. The action plan was discussed more often in Phase 2 in the
second action plan because the commanders changed the level of PPE suits as
the Ops officers suggested, whereas the first action plan was not altered.
However, the role of the Ops officers in managing the first responders in
the hot zone, such as deploying of equipment and creating the incident
action plan, reduced most of the need for planning in the command post as
evidenced by the figures in Table~\ref{tab5}. Discussions about the personnels
participating in the simulation occurred more frequently in Phase 1 in which
commanders were getting acquainted with everyone's teams and clarifying who
would be among the first entry team's members. In Phase 2 where everyone
knew who was part of the response team, personnel-related discussions became
less frequent. Event management discussions, such as communication about
lunch, doubled in Phase 2 when it was time for lunch and coffee breaks.
Discussions about equipment other than PROBE remained relatively constant.
In Phase 1, the commanders decided on everyone's radio frequency and, in
Phase 2, discussion concerned the PPE needed for re-entry to the hot zone.
Conversations with other members doubled in Phase 2, as commanders talked
about sharing information down the chain of command and asked about other
communications more often. Communication about the offending agents was
minimal in Phase 2 as they had already been neutralized and deactivated in
Phase 1. The finding that the event management was discussed more frequently
than equipment or personnel suggests that the Ops officers were experienced
requiring little or no input from the senior officers.
\subsubsection{Step 2: assessing the severity of ineffective \hbox{communication}}\label{subsubsec4.3.2}
The severity of ineffective communications was assessed by examining the
topic of each such instance as well as the types and topics of utterances
immediately following the ineffective communication, to identify the
consequences of these. This level of analysis is not included in the content
analysis literature, but it was necessary to understand the severity of
ineffective communications.
In total, 23.6{\%} $(n = 17)$ of all ineffective communications were PROBE
related. However, only one of these resulted in a communication breakdown.
Others were averted because the commanders used their radios as backup when
information was not coming in through PROBE. Right from the start of the
simulation, they requested confirmation of all communications, to ensure
that messages were received by the intended listener(s).
The single communication breakdown that had a severe consequence occurred in
Phase 2. It involved the hazmat first responder and the explosive disposal
unit (EDU) officer in the hot zone who was running out of oxygen. Both were
dressed in high-level PPE, which are air-tight safety suits for highly
dangerous situations. This suit is sealed onto the responder's body, with
oxygen delivered via a tank carried on the back inside the suit. The oxygen
lasts for 1 h, but could be as short as 30--35 min depending on the wearer's
level of exertion. The hazmat commander received a somewhat alarming radio
update from his Ops officer. The IC overheard this and requested
clarification from the hazmat commander, who explained that the two
responders had most likely already lost their air supply. Apparently, the
chemicals had already been neutralized and the EDU and hazmat first
responder were in the hot zone, waiting for forensics responders to enter.
The EDU officer's radio signal was too weak to enable him communicating with
his team, relying instead on the hazmat first responder, who was with him in
the hot zone, for relaying communication. Meanwhile, the police forensics
officers were waiting for the commanders' approval of the second incident
plan before entering the hot zone. However, the Ops officers' proposed
action plan had not yet reached the commanders since PROBE was not
forwarding attachments. Therefore, the document had to be written and
physically brought to the commanders. It took the commanders, who were
unaware of the event unfolding in the hot zone, another 6 min to approve the
incident action plan, and the entry was to take place $\sim$10 min
after that. However, because the commanders were then told that only a
negligible amount of radiation had been detected, the responders now needed
to wear a lower level of PPE. This change delayed the response team so that
the forensics team finally made entry 2 min later than the original 10 min
planned after the second action plan had been approved. This timeline is
shown in Fig.~\ref{fig4}. The moment the IC was notified that the forensics had
entered the hot zone, the hazmat commander was informed that the EDU and
hazmat officer were running out of air in their air-tight PPE. Apparently,
the EDU officer had gone to the decontamination area, but no one was there
so he had actually run out of oxygen, collapsing to the ground. Thankfully,
nearby hazmat first responders discovered him and removed his suit. This
chain of events was surprising as the EMS commander had ordered a paramedic
to get suited up in PPE and stand-by the hot zone, 3 min before the second
incident action plan was approved. This should have given the paramedic
enough time to get ready and be in a position by the decontamination area.
The reason for the lack of his immediate presence and assistance is unknown.
Other than that unfortunate incident, `the exercise went excellent[ly]',
according to the police Ops officer.
\begin{figure}[hbt!]
%\centering{\includegraphics{IWCOMP_CE_Samplef04.eps}}
\centerline{\fbox{\vbox to 12pc{\vfill\hbox to 20pc{\hfill\Huge FPO\hfill}\vfill}}}
\caption{Timeline of the event in which a responder ran out of
oxygen.\label{fig4}}\vspace*{-3pt}
\end{figure}
\subsection{Inter-rater reliability}\label{subsec4.4}
Cohen's kappa \citep{bib5} is widely used in behaviour-coding research
\citep{bib2}; \citep{bib4} as well as being relatively simple to
compute \citep{bib23}. Conceptually, it is equal to the observed
proportion of agreement between raters, after adjusting for the proportion
of agreement expected by chance (randomly). An obtained value <0.41
represents a weak inter-rater reliability; a value between 0.41 and 0.60 is
considered `moderate' \citep{bib4}, and a value between 0.60 and
0.80 is deemed `satisfactory'. A value $>$0.80 is almost
`perfect'. The inter-rater reliability was calculated separately for the two
analysis types. For communication analysis, agreement between the two
researchers was 70.2{\%}, yielding a moderate value of 0.5903 by Cohen's
kappa. The main discrepancy was due to the second researcher misclassifying
questions and answers as well as clarifications requested or granted. For
latent inductive content analysis, agreement was 62.9{\%}, yielding a value
of 0.5532 by Cohen's kappa, also considered moderate. The main issue here
was that the second researcher was largely unfamiliar with the context or
the concept of PROBE, so she labelled many `PROBE' communications as
`equipment'. In both cases, the disagreements were settled by negotiation.
\subsection{Communication and teamwork}\label{subsec4.5}
As a thorough social network analysis is reported in detail elsewhere
(Stojmenovic and Lindgaard, in press), only a summary of the speaker-related
findings is provided here. The hazmat commander spoke most frequently,
closely followed by the IC and the EMS commander. One would have expected
that the IC should speak most frequently of all because he is in charge of
the entire event. However, as the hazmat commander was older than the IC, he
may also have had more experience in the command post. At the beginning of
the event, the IC informed the other commanders that their advice was
welcome. Later, he commented that he missed being involved hands-on with the
response, saying that he found that `when you move on, you start to miss the
things you used to do', motioning towards the hot zone as he spoke. It is
thus possible that a difference in experience may explain the apparent
reversal of speech frequency between the two officers. In addition, even
though a bomb was involved in the event, the IC who was from the police
department did not communicate more because he was only notified about it
when it had just been found, and when it was deactivated. His advice on how
to handle it was therefore not needed. Although the IC is in charge of the
event, and therefore is responsible for making the biggest decisions, in the
present case, it was unclear whether he changed the incident action plan
before approving it, which was the biggest decision to be made. Since the
researchers did not have access to that file, it can only be speculated that
the IC had more input when approving it, and his input was not discussed.
The EMS commander spoke the least of the three commanders. It is possible
that this may be attributed to the fact that there were only three
`casualties', presented in the form of pie plates with symptoms written on
them. The `patients' therefore did not require attention. The EMS first
responders' duties were limited to monitor the vital signs of teammates
entering and exiting the hot zone---a routine task for paramedics, not
requiring a lot of communication.
Taken altogether, of the 21.1{\%} of all communications being ineffective,
only one led to an actual communication breakdown. The content and context
of the utterances of ineffective communications showed that 63.4{\%} of all
such instances occurred between commanders and Ops officers, mainly due to
misunderstandings with PROBE. The remainder occurred among commanders,
scribes, the PROBE developer and the event mangers. Only 11.1{\%} of
ineffective communications occurred between commanders, suggesting that
communication among commanders was highly efficient and effective. This is
probably because the commanders were collocated and could hear each other's
radios, leaving little room for confusion. The common goals of neutralizing
the chemical, deactivating the bomb, treating patients and collecting
evidence were all completed in a timely fashion. The one communication
breakdown was not an intended part of the simulation scenario. However, the
high frequency of effective communication, coupled with the ability
efficiently to overcome breakdowns when they did occur, suggests that the
overall teamwork among the group of commanders and Ops officers was
effective and successful.
\subsection{CBRNE simulation goals voiced during debriefing}\label{subsec4.6}
During the debriefing session, it became clear that the agencies had
different agendae and goal with the simulation. The fire and hazmat
personnel's goal was to expose the responders to hands-on CBRNE training.
For example, in the process of approving the second incident action plan for
re-entry into the hot zone, the IC asked the hazmat commander `Why would
they have to go back in?' The hazmat commander replied, `I think it's just
for the practice$\ldots$ I think they're just getting guys into suits'. In
addition, the hazmat team and the police team members wanted to practice
working with the other two agencies. The police personnel and many of the
others were from other cities in New Brunswick, as well as representing
different agencies and different levels of government. Collectively, the
police officers' goal was primarily to get practice coordinating with other
sectors and agencies. The EMS goals were to test the usability of a new
worksheet and to test PROBE. The EMS team had recently become a provincial
team such that, in case of a larger CBRNE-related event, any paramedic,
anywhere in the province could be dispatched to the scene. Therefore, some
procedures for large-scale events were being changed, requiring new forms
for the paramedics to support their tasks. As most of the participating
personnel had taken part in the PROBE training session the previous day,
they focused on testing it during the simulation, which explains why they
relied so heavily on PROBE for communication and used it throughout the
event as noted earlier. Representatives of the media had been invited to the
simulation, which tended to lend it quite a different atmosphere than had
been observed in previous simulations, all of which had been closed to the
press. Reporters, cameras and photographers were everywhere recording the
command post, and interviewing event coordinators at the beginning of Phase
1. The commanders may thus have acted more casually than they typically
would, which may account for the jokes and socialization that occurred in
the command post.
\section{DISCUSSION}\label{sec5}
\subsection{Theoretical implications}\label{subsec5.1}
\looseness=-1 Distributed cognition guided the data collection and analysis in this paper.
It enabled the researchers to examine the interactions and exchanges of
information between commanders, and between commanders and PROBE. It also
helped the researchers to untangle and understand the serious communication
breakdown observed. Since this incident was partially the result of the Ops
officers' interactions with PROBE, it testifies to the importance of the
framework focusing on both human--human and human--computer interactions.
The observation that the IC had to clear the messages before passing them
onto the various teams was a responder-specific procedural issue that had
not been observed in our previous CBRNE-related research. Once PROBE is able
to handle messages in the way intended, this will no longer cause problems.
The need manually to refresh the screen will also, to the best of our
knowledge, be rectified.
\subsection{The role of PROBE}\label{subsec5.2}
The PROBE prototype was intended to aid communication, supplement decision
making and facilitate effective teamwork. Some issues did arise as a result
of shortcomings due to the fact that PROBE was still a prototype rather than
a fully fledged management decision-support system. Other issues arose from
the fact that the responders had only received a minimum of training on the
software the day prior to the simulation. Thus, they had no additional
practical experience using PROBE. In the command post, more time was given
to testing PROBE than to managing the event, leaving little necessity for
teamwork and decision making. PROBE did supply a summary of events inputted
into it, to aid in writing the incident report, a summary of the event
response and outcome by CBRNE responders. With a fully operational version
of PROBE, responders will be able to continue communication even when their
radios may not be working. It will also help to create detailed incident
reports as team members representing each agency must delivers upon
completion of the CBRNE event or simulation. Access to a wide range of CBRNE
information sources necessary for the successful event management should
also prove an advantage for emergency response teams.
\subsection{Modification of analysis methods}\label{subsec5.3}
\looseness=-1 At one level, communication analysis and latent inductive content analysis
were adequate for this research. Communication analysis facilitated
understanding of how information was being shared. However, at another
level, the method as described in the literature did not entirely meet the
analytic requirements. Sequences of communications had to be analysed to
identify instances of effective and ineffective communications in the CBRNE
transcript. This added step provided local context enabling the researchers
to see clearly which utterances demanded a response, and when responses
did/did not occur. We argue that examination of sequences is essential when
interpreting data intended to yield an understanding of communication
effectiveness in instances in which effectiveness is determined by open- and
closed-loop communication. Latent inductive content analysis was used to
categorize the topic of each utterance to determine the severity of
ineffective communications. This too required an additional step to the way
the method is described in the literature. The original analysis involved
the coding of all utterances by topic, and the examination of the utterance
topic(s) involved in the ineffective communication. The additional step
involved examination of types and topics of utterances immediately following
the instances of ineffective communication. This was necessary to reveal the
consequences of such communications as well as to determine the severity of
ineffective communications.
\looseness=-1 Taken together, these additional steps, the two analysis methods helped the
researchers to gain a deeper understanding of communication in the command
post during the CBRNE simulation. Although it is acknowledged that these
additional steps may not always be necessary, it would be helpful for future
researchers to refine the descriptions of these two analysis methods in the
literature by including them. The next step towards such refinement would
thus be to identify the circumstances under which such additional scrutiny
of data is required.
One interesting issue concerns the separation of communication and content
analyses. Communication analysis as described by \citep{bib17} included both
the type and the topic of utterance, whereas, following some research
published in the literature \citep{bib9}, these were kept
separate in this research. However, to gain a better understanding of
communication, it was important to analyse the type and topic for each
utterance simultaneously. Keeping communication and content analyses
separate was thus not necessary. It is therefore suggested that the method
applied by \citep{bib17} be used.
With respect to the somewhat modest inter-rater reliability ratings, it would
have been helpful if the primary researcher had explained the category
definitions in more detail and allowed the second researcher to ask
questions of clarification at the outset. It also suggests that the category
definitions may have been insufficiently clear to allow the second rater to
categorize 10{\%} of the transcript correctly.
\section{CONCLUDING REMARKS}\label{sec6}
\looseness=1 Although all but one participant had received training a day before the
simulation, they had forgotten some of PROBE's features. One way this could
be prevented in a future simulation would be to provide responders a quick
review of the software capabilities immediately before the simulation, to
remind them that PROBE could not send all types of attachments. In addition,
a pocket-size cheat sheet would have been beneficial for the participants
using PROBE. These may have prevented, or at least reduced, the consequences
of the unfortunate mishap. Some types of alarm should be incorporated into
the PPE to signal oxygen levels to the responders wearing them as well as to
the personnel taking care of the wearer.
Because some of the details concerning the simulation reported here were
revealed only in the briefing at the end of the simulation, it would be
helpful for researchers to know more about the management structure, the
scenario, the magnitude of the event and the number of professionals
expected to take part in it ahead of time. That information was not
available to us in this simulation. Because of this, the checklist shown in
Appendix was devised to help future researchers plan and organize field data
collection strategies when observing multi-agency emergency responses.
CBRNE training simulations are very expensive and time-consuming and are
therefore not run very often. Few are open to researchers. The opportunity
to observe and study such simulations is valuable as these add to
researchers' understanding of the CBRNE response as well as enabling them to
test data collection and data analysis methods under real-life
circumstances. These opportunities also add our understanding of some of the
challenges associated with conducting field studies of software prototypes.
The attached checklist is intended to help researchers better to prepare for
observing CBRNE emergency response personnel in the future.
\ack Removed for blind review.
\makeatletter
\def\@biblabel#1{}
\makeatother
\begin{thebibliography}{99}
\bibitem[Amita(2008)]{bib1}
Amita (2008) PROBE Crime Scene Support Tool for Police, Hazmat, EMS, AER {\&} VFR. Poster.
\bibitem[Bakeman(2000)]{bib2}
Bakeman, R. (2000) Behavioral Observation
and Coding. In Reis, H.T. and Judge, C.M. (eds), Handbook
of Research Methods in Social and Personality Psychology,
pp. 138--159. Cambridge University Press, New York.
\bibitem[Bowers(1998)]{bib3}
Bowers, C.A., Jentsch, F., Salas, E. and
Braun, C.C. (1998) Analyzing communication sequences for
team training needs assessment. Hum. Factors, 40, 672--679.
\bibitem[Burla et al.(2008)]{bib4}
Burla, L., Knierim, B., Barth, J., Liewald,
K., Duetz, M. and Abel, T. (2008) From text to codings:
intercoder reliability assessment in qualitative content
analysis. Nursing Res., 57, 113--117.
\bibitem[Cohen(1960)]{bib5}
Cohen, J. (1960) A coefficient of agreement
for nominal scales. Educ. Psychol. Meas., 20, 37--46.
\bibitem[Dubbels(2011)]{bib6}
Dubbels, B. (2011) Cognitive ethnography: a
methodology for measure and analysis of learning for game
studies. Int. J. Gaming Comput. Mediat. Simul., 3, 68--78.
\bibitem[Emergency Management Ontario Ministry of Community Safety and Correctional Services(2008)]{bib7}
Emergency Management Ontario Ministry of Community Safety and
Correctional Services (2008) Province of Ontario emergency
response plan. (retrieved November 28, 2011).
\bibitem[(2009)]{bib8}
Emergency Management Ontario Ministry of Community Safety and
Correctional Services (2009) Provincial nuclear emergency
response plan. (retrieved November 28, 2011)
\bibitem[Hazlehurst et al.(2007)]{bib9}
Hazlehurst, B., McMullen, C.K. and Gorman,
P.N. (2007) Distributed cognition in the heart room: how
situation awareness arises from coordinated communications
during cardiac surgery. J. Biomed. Inform., 40, 539--551.
\bibitem[Hollan et al.(2000)]{bib10}
Hollan, J., Hutchins, E. and Kirsh, D.
(2000) Distributed cognition: toward a new foundation for
human-computer interaction research. ACM Transact.
Comput-Hum. Interact., 7, 174--196.
\bibitem[Hsieh and Shannon(2005)]{bib11}
Hsieh, H.F. and Shannon, S.E. (2005) Three
approaches to qualitative content analysis. Qual. Health
Res., 15, 1277--1288.
\bibitem[Humphrey and Adams(2011)]{bib12}
Humphrey, C.M. and Adams, J.A. (2011) Analysis of complex team-based systems: augmentations to goal-directed task analysis and cognitive work analysis. Theor. Issues Ergonomics Sci., 12, \hbox{149--175}.
\bibitem[Hutchins(1995)]{bib13}
Hutchins, E. (1995) How a cockpit remembers its speed. Cogn. Sci., 19, 265--288.
\bibitem[Jederberg et al.(2002)]{bib14}
Jederberg, W.W., Still, K.R. and Briggs,
G.B. (2002) The utilization of risk assessments in tactical
command decisions. Sci. Total Environ., 288, 119--129.
\bibitem[Kirsh(2004)]{bib15}
Kirsh, D. (2004) Metacognition, Distributed
Cognition and Visual Design. In Gardinfas, P. and
Johansson, P. (eds) Cognition, Education and Communication
Technology. Lawrence Erlbaum.
\bibitem[Kozlowski and Bell(2003)]{bib16}
Kozlowski, S.W.J. and Bell, B.S. (2003)
Work groups and teams in organizations. In Borman, W.C.,
Ilglen, D.R. and Klimoski, R.J. (eds) Handbook of
Psychology: Industrial and Organizational Psychology (Vol.
12), pp. 333--357. Wiley, London.
\bibitem[Kramer(2009)]{bib17}
Kramer, C. (2009) Communication linking
team mental models and team situation awareness in the
operating room. Master's Research, Carleton University.
\bibitem[Kuban et al.(2001)]{bib18}
Kuban, R., MacKenzie-Carey, H. and Gagnon,
A.P. (2001) Paper {\#}4: Disaster Response Systems In
Canada. Institute for Catastrophic Loss Reduction.
\bibitem[Lewis(1985)]{bib19}
Lewis, I.M. (1985) Social Anthropology in
Perspective. Cambridge University Press, Cambridge.
\bibitem[Lingard et al.(2004)]{bib20}
Lingard, L., Espin, S., Whyte, S.,
Regeh, G., Baker, G.R., Reznick, R., Bohnen, J., Orser, B.,
Doran, D. and Grober, E. (2004) Communication failures in
the operating room: an observational classification of
recurrent types and effects. Qual. Saf. Health Care, 13,
330--334.
\bibitem[Lindgaard et al.(2006)]{bib21}
Lindgaard, G., Dillon, R., Trbovich, P.,
White, R., Fernandes, G., Lundahl, S. and Pinnamaneni, A.
(2006) User needs analysis and requirements engineering:
theory and practice. Interact. Comput., 18, 47--70.
\bibitem[Lindgaard et al.(2009)]{bib22}
Lindgaard, G., Dudek, C., Noonan, P., Sen,
D. and Tsuji, B. (2009) PROBE: A Preliminary User Needs
Analysis for a CBRNE Command Post. Technical Report
Prepared for Amita Inc., Ottawa.
\bibitem[Lombard et al.(2010)]{bib23}
Lombard, M., Snyder-Duch, J. and Bracken,
C.C. (2010) Practical resources for assessing and reporting
intercoder reliability in content analysis research
projects.
\bibitem[May(2009)]{bib24}
May, C. (2009) First responder training
program multi-agency response. Public Safety Emergency
Management, Toronto CBRNE Team. Unpublished manuscript.
\bibitem[Mickan and Rodger(2005)]{bib25}
Mickan, S.M. and Rodger, S.A. (2005)
Effective health care teams: a model of six characteristics
developed from shared perceptions. J. Interprof. Care, 19,
358--370.
\bibitem[Moynihan(2009)]{bib26}
Moynihan, D.P. (2009) The network
governance of crisis response: case studies of incident
command systems. J Public Administration Res. Theory, 19,
895--915.
\bibitem[Orasanu(1994)]{bib27}
Orasanu, J. (1994) Shared Problem Models
and Flight Crew Performance. In Johnston, N., McDonald, N.
and Fuller, R. (eds) Aviation Psychology in Practice, pp.
255--285. Ashgate, Aldershot, UK.
\bibitem[Owen et al.(2008)]{bib28}
Owen, C., Douglas, J. and Hickey, G. (2008)
Information flow and teamwork in incident control centers.
In Fiedrich, F. and Van de Walle, B. (eds) Proceedings of
the 5th International ISCRAAM Conference, Washington, DC,
USA, pp. 742--751.
\bibitem[Parush et al.(2011)]{bib29}
Parush, A., Kramer, C., Foster-Hunt, T.,
Momtahan, K., Hunter, A. and Sohmer, B. (2011)
Communication and team situation awareness in the OR:
implications for augmentative information display. J.
Biomed. Inform., 44, 477--485.
\bibitem[Reddy and Spence(2008)]{bib30}
Reddy, M.C. and Spence, P.R. (2008)
Collaborative information seeking: a field study of a
multidisciplinary patient care system. Inform. Process.
Manage., 44, 244--255.
\bibitem[Reutter et al.(2010)]{bib31}
Reutter, D., et al. (2010) Planning for
exercises of Chemical, Biological, Radiological, and
Nuclear (CBRN) forensic capabilities. Biosecur. Bioterror.,
8, 343--355.
\bibitem[Rogers(2006)]{bib32}
Rogers, Y. (2006) Distributed Cognition and
Communication. In Keith Brown (ed) The Encyclopedia of
Language and Linguistics (2nd edn), pp. 181--202. Elsevier,
Oxford.
\bibitem[Rogers and Brignull(2003)]{bib33}
Rogers, Y. and Brignull, H. (2003)
Computational offloading: supporting distributed team
working through visually augmenting verbal communication.
In Proceedings of the 25th Annual Cognitive Science Society
Conference, Boston.
\bibitem[Salas et al.(2009)]{bib34}
Salas, E., Rosen, M.A., Held, J.D. and
Weissmuller, J.J. (2009) Performance measurement in
simulation-based training: a review and best practices.
Simul. Gaming, 40, 328--376.
\bibitem[EMSs; Simpson and Hancock(2009)]{bib35}
Simpson, N.C. and Hancock, P.G. (2009) The
incident commander's problem: resource allocation in the
context of emergency response. Int. J. Serv. Sci., 2,
102--124.
\bibitem[Stojmenovic and Lindgaard(2012)]{bib36}
Stojmenovic, M. and Lindgaard, G. (in
press) The Social Network Analyses (SNA) of two Chemical,
Biological, Radiological, Nuclear, and Explosives (CBRNE)
training exercises. OZCHI 2012.
\bibitem[Stojmenovic et al.(2011)]{bib37}
Stojmenovic, M., Dudek, C., Noonan, P.,
Tsuji, B., Sen, D. and Lindgaard, G. (2011) Identifying
User Requirements for a CBRNE Management System: A
Comparison of Data Analysis Methods. In Proceedings of the
8th International Information Systems for Crisis Response
and Management Conference, Lisbon, Portugal, May 2011.
\bibitem[Van der Kleij et al.(2009)]{bib38}
Van der Kleij, R., de Jong, A., te Brake,
G. and de Greef, T. (2009) Network-aware support for mobile
distributed teams. Comput. Hum. Behav., 25, 940--948.
\bibitem[Waugh(2006)]{bib39}
Waugh, W.L. Jr. and Streib, G. (2006)
Collaboration and leadership for effective emergency
management. Collab. Public Manage., 131--140 (special
issue).
\bibitem[Williams(2006)]{bib40}
Williams, R.F. (2006) Using cognitive
ethnography to study instruction. In Proceedings of the 7th
International Conference of the Learning Sciences,
Bloomington, Indiana,\break pp. 838--844.
\end{thebibliography}
\appendix
\section*{APPENDIX}
Emergency services responders deal with several types of emergencies as part
of their normal, daily routines \citep{bib18}. For example, the
discovery of an unattended brown parcel left at an airport may lead to its
immediate closure; departures are postponed until further notice, arrivals
are rerouted, people are evacuated and luggage service is halted, all of
which are both time-consuming and very costly \citep{bib37}.
\end{document}
