The WorkbookExample 02Aviation

Two Controllers, One Drone.

A drone is launched and flown out by a Mission Controller at a rear station, handed to a Forward Controller inside the battlespace area for the middle of its mission, tasked from there, handed back at the edge of the area, and let go so the Forward Controller can turn to the ground element's next move. Nobody doubted the drone could be flown from the forward station. The question was the two handovers, and the one that lands on top of everything else the Forward Controller has to do.

Domain
Aviation, uncrewed aircraft in the battlespace
System
A forward control station, with control of one drone transferred to it from a rear station and back again in flight
Roles
Forward Controller, Mission Controller, Ground Commander
Modes
Baseline tasking without the drone; forward-controlled mission in four phases, receive, employ, hand back, move on; hand back not acknowledged; link lost while employed
Standards basis
MIL-STD-1472H, governing · STANAG 4586 read · the operator's handover procedure as the criterion
Method chain
  • HTA
  • ErgoAct
  • NASA-TLX
  • SART
Trigger
A trial of forward control, with a decision on the forward station's control page due before the next software baseline
Captures
ErgoSphere 1.6.1 · pending
1

Frame the problem

Plan 1: do 1.1, 1.2, 1.3 in order. Return to 1.1 if 1.3 changes the question.

1.1What the client asked for

Practice

The operator was to trial forward control. A drone would be launched and flown out by a Mission Controller at a rear station, handed to a Forward Controller at a small station inside the battlespace area, tasked from there for the middle of the mission so that the ground element it was supporting could direct its sensor without a relay, handed back to the Mission Controller at the edge of the area, and recovered from the rear. The vendor's control page for the forward station existed as a software baseline. The trial's sponsor had to accept it or send it back before the next one, and the question they asked was the one that comes with every capability that adds a job to a person: "Can the forward station take it?"

Asked as written, the question invites an opinion, and a confident one from anyone who has controlled a drone. The practitioner's job is to turn it into something a method can answer, and the first move is to find out what the person at the forward station actually does.

1.2What actually changed for the Forward Controller

Practice

The Forward Controller is not someone with nothing to do but fly a drone. They are part of a ground element, inside the area the drone is working, with their own communications to keep, their own position and security to think about, and a Ground Commander beside them who wants the picture and wants it now. Adding a drone to command does not add a task to that list. It adds a second thing's state to hold in mind, a control page to work under field conditions, and two transactions with a person at another station that have to succeed at a particular place and time.

Flying the drone in the employ phase was not, on inspection, the hard part. Once it is over the area it holds its pattern and does what it is tasked to do, and the Forward Controller monitors it the way they monitor anything else. The hard parts are the edges. The handover in, when the drone crosses into the area and the Forward Controller must verify it is what the Mission Controller says it is and accept it. The hand back at the edge of the area, when the Forward Controller must put it in a safe state, offer it, confirm the Mission Controller has it and release it. And the move on immediately after, when the Forward Controller has to stop thinking about an aircraft they no longer control and start thinking about their own comms, their own position and the element's next move.

Of those three, the hand back is the one that lands on top of something. The handover in happens while the element is static and waiting for the drone. The hand back happens because the element is about to move, which is the moment the Ground Commander wants the last of the picture and the move time, and the station itself has to be closed down. So the change is not "one more thing to control". The change is that a multi-step transaction with a person at another station, one that has a window and a wrong answer, now happens under interruption, at the busiest point of the Forward Controller's own job. The question the work has to answer:

Can the Forward Controller hand the drone back to the Mission Controller inside the procedure's window, while doing the work the Ground Commander and the move are asking for, reliably enough to accept, and what has to be true of the control page for that to hold?

Two things follow. The measure is time, distributed across many attempts, not an average. And the condition is interruption, which means the analysis has to model the Forward Controller doing something else in the middle of the hand back.

Who has it?

Every handover between two controllers has the same failure at its centre: a moment when both believe the other has the aircraft, or both believe they do. The procedure guards it with voice. The question for the interface is whether the display guards it too, so that a Forward Controller who is interrupted mid-transaction can look down and know, without asking, whether they still hold the drone.

1.3The boundary, the roles and the modes

PracticeIn ErgoSphere

The boundary: the forward station's control page, its displays and alerting, the handover procedure and the voice exchange with the Mission Controller are in. The drone's own autonomy, the datalink hardware, the rear station and the airspace rules are out, and are treated as fixed inputs. The lost-link drill is out of scope on this page: it is a procedure of its own and it is assessed as one. The roles: Forward Controller, Mission Controller and Ground Commander, recorded as positions. The modes: the baseline tasking without the drone, so there is something to compare against; the forward-controlled mission in its four phases, receive, employ, hand back and move on; a hand back that is offered and not acknowledged; and a link lost during the employ phase, which stays in the list even though its drill is out of scope, because the analysis, not the sponsor, decides what is credible.

In ErgoSphere the roles go into the Role Register as positions, and the framing is written as the Early Human Factors Analysis, run as a wizard with the standards lens set to MIL-STD-1472H. The change is entered as two states, the baseline tasking and the forward-controlled mission; the Forward Controller is the primary end-user group; the control page and the two handovers are the elements of note; and of the eight domain probes, workload and vigilance and the HMI probe are rated high, human error potential and task novelty high, and the environment probe medium because the station is used in the field, with the remaining physical probes considered and dismissed with a rationale. Every session that follows is named for its phase and condition, "Hand back, alone" and "Hand back, under the move", which is a practitioner's habit rather than a feature, and it is what lets the results sit side by side at stage 5 without anyone asking which was which.

2

Establish the standards basis

Plan 2: do 2.1; then 2.2 for the governing standard; then 2.3 for the documents read but not loaded.

2.1How to find what applies

Practice

Defence programmes make this step easier in one way and harder in another. Easier, because the contract carries a Human Factors integration requirement and an applicable documents list, so the top of the hierarchy is written down. Harder, because the list is long, much of it is about the whole system rather than one station, and the practitioner has to say which parts bite on this change.

  1. The contract. The Human Factors programme requirement and its applicable documents list. In this scenario the list calls up MIL-STD-1472H for the design of the forward station's displays, controls and alerting, which makes it the governing design standard of this example: the control page's information content, its controls and its alerting are assessed against its clauses directly, not through another standard. MIL-STD-1472H is the standard Australian defence programmes most often call up for operator station design, and it is the one this example teaches. In the rail example it does not appear at all, because rail has its own governing standard and a control-centre standard for the numbers.
  2. The operator's own orders. The handover procedure, with the window it gives the hand back, is the operator's document. It is not a design standard, but it is the criterion the analysis will be read against, so it goes in the basis.
  3. The interoperability standard. STANAG 4586, which defines the levels of control a station can hold over an uncrewed aircraft and the transfer of control between stations. Read to establish what the forward station is: a controlling station with the drone's flight under its command, not a terminal with a picture of it. That distinction decides how much of the standard's control and alerting material applies.
  4. The operator's aviation safety management system and the assurance its regulator expects, read for the form the argument at stage 7 has to take, not for design criteria.
  5. Good practice on handover in other domains, air traffic control and the process industries in particular, where the transfer of control between two people has been studied for decades and the failure modes are well described.

Which document governs, and which is the criterion?

MIL-STD-1472H governs the design: it is the contract's standard for the station, so its information, control and alerting clauses are the ones the requirements trace to and the close-out answers. STANAG 4586 shapes the work without governing it: it says what a controlling station is, which is what tells the practitioner the Forward Controller can act on the drone and is responsible for it. And only one document in the list says what "in time" means: the operator's handover procedure. When a method produces a time, that is what it is compared against. Identify the criterion document at stage 2, on purpose, so that at stage 5 nobody has to argue about what the numbers are being measured against.

2.2MIL-STD-1472H, clause by clause

In ErgoSphere

MIL-STD-1472H is in the Standards Register's library, and it arrives as a tree of sections and clauses with the standard's own text. Screening runs section by section with three chips, Applicable, Partial or N/A, and each decision records who made it and when. For this change the sections on the content and arrangement of displayed information, on controls and on visual and auditory alerting are Applicable in full, because the control page is new and every clause under them bears on it. The sections on display coding and the identification of items are Partial, decided clause by clause, because only the parts about telling the state of one thing apart from the state of another matter here. The sections on the environment are Partial too, because the station is used outdoors and the clauses on legibility in daylight apply. The sections on workstation dimensions and reach are N/A, because the station's hardware is unchanged, and the rationale box says so. The rest of the standard, which runs to hundreds of pages about things this change does not touch, is N/A section by section with its reason, and Confirm locks each record.

09-standards-1472.webpStandards Register with MIL-STD-1472H from the library: the section applicability table with the information content, controls and alerting sections Applicable, the coding and environment sections Partial and the physical sections N/A, screening KPIs at the top.
A big standard, a small change. Most of MIL-STD-1472H is not applicable here. Saying so, block by block, is what lets a reviewer trust the parts that are.

2.3Read, not loaded

Practice

STANAG 4586 shaped the work without being worked through clause by clause. It told the practitioner that the level of control the forward station holds during the employ phase is the level at which the station commands the drone's flight, and that the transfer of that control between stations is a defined transaction with a defined end. Both facts mattered. The first ruled out an easy answer that had been floated, that the Forward Controller was "only tasking the sensor" and the page could be treated as a picture. The second is what gave the hand back a beginning and an end that could be timed. The standard is cited in the professional judgement of the clauses it informed. Read but not loaded is a legitimate status, as long as the trail says which decisions the document touched.

3

Choose the methods

Plan 3: do 3.1; then 3.2 for each candidate; then 3.3 to record the choices.

3.1The questions the methods must answer

Practice
  • Description first. What does the Forward Controller actually do, from the drone crossing into the area to the element moving on? Every goal, every operation, in order, with the decisions and the plans. Nothing else can be chosen until this exists, because until it exists nobody can say where the risk is.
  • Time, as a distribution. How long does the hand back take, across hundreds of attempts, alone and under the interruption of the move, before the control page exists in a trial rig to measure it on?
  • Validation. When the trial rig is available, do the Forward Controller's workload and awareness in each phase stay within the margins the requirements set, against the baseline tasking?

The constraint that shaped everything: the trial rig was booked twelve weeks out, and the software baseline decision was due in six. Whatever answered the time question had to answer it without the rig.

3.2The candidates, and the roads not taken

Practice
Decision point Understanding the role

Taken

Hierarchical Task Analysis

The forward-controlled mission is a sequence of goals with a known order and known decisions, and HTA captures it as goals, operations and plans. It is the first method run and everything else reads from it: the storyboard at 4.2 is built from its hand-back branch, the trial conditions at 4.3 are its phases, and the requirements at stage 6 cite its nodes. It is also where the practitioner learns what the Forward Controller does, which the sponsor's question assumed everybody already knew.

Not taken

Goal-Directed Task Analysis

GDTA would have decomposed the Forward Controller's information needs for situation awareness, and it is a good method for designing a new display from nothing. The question here was time under a known procedure, on a page that already existed. GDTA was noted for the page's redesign if the trial failed.

Decision point Predicting the time

Taken

ErgoAct, an ACT-R cognitive model

ACT-R is the cognitive architecture the research literature uses to predict how long a sequence of perceiving, remembering, deciding and acting takes, with the variability a human brings to it. ErgoAct builds the model from a storyboard of the task and runs it hundreds of times, so the output is the distribution the question needs, available before the rig is. For drone control this is the method's natural home: the tasks are procedural, the interfaces are menus and pages, and the failures are the ones the architecture is built to represent, a retrieval that does not come back, a search among similar things, a step resumed in the wrong place.

Not taken

GOMS / KLM

The keystroke-level model gives a deterministic time for a fixed sequence of expert actions. It cannot represent an interrupted task, a retrieval that sometimes fails, or a tail. It would have produced one number, and one number was the thing stage 1 said could not answer the question.

Decision point Validating in the rig

Taken

NASA-TLX and SART, in session

Perceived workload and situation awareness, rated by the Forward Controllers after each run in the trial rig, phase by phase, for the baseline tasking and the forward-controlled mission. Both are quick, both are standard, and both give the baseline the requirements are written against. Run together they say whether the person was busier and whether they knew less.

Not taken

ErgoNeuro, EEG workload

Measured rather than rated workload would have been better evidence, and the platform has it. The rig sessions were shared with a training serial and there was no time to fit and settle headsets between runs. Deferred, with the note that a follow-on trial should use it.

Also considered and set aside, and both recommended for the next phase: ErgoAIR, to rationalise the drone's alerts before they are routed to a station in the field where the person has other things to listen to, and ErgoTrace, an STPA of the control structure linking the two stations, the drone and the datalinks, which is the right method for the question "what unsafe control actions does shared control make possible". Both are bigger questions than the sponsor asked. Both are in the report as the work that should follow if the page is accepted.

3.3Recording the choice

In ErgoSphere

ErgoCompass, with the industry set to Aviation and the problem kind to mental workload, ranks its palette and records the justification in the same five fields as Example 01, with the three forks above in the "considered and set aside" field. Send to HFIP carries them into the Integration Plan, and here the plan is worth a sentence of its own: with the standards lens on MIL-STD-1472H the wizard produces a Human Engineering Program Plan in the sixteen-section structure defence programmes expect, so the sponsor's Human Factors lead receives a document shaped like the ones they already review. The screening answers are Modified, High and Demanding. Two things are then done deliberately. ErgoNeuro goes into the plan's Planned and not applied section with its reason, so it reads as a decision and not an omission. And the ErgoAIR and ErgoTrace recommendations go into the activity-by-phase table against the next phase, so the next practitioner on the programme inherits them.

4

Run the analysis

Plan 4: do 4.1 first and validate it with people who hold the role; then 4.2 for the hand back alone and again under the interruption of the move; do 4.3 when the rig is available; if 4.2 fails the window, do 4.4 before 4.3.

4.1HTA of the Forward Controller's mission

PracticeIn ErgoSphere

The procedure gave the skeleton. Two people who had held the forward role gave the rest, in a talk-through at a desk with the control page on a laptop and the Mission Controller played by the practitioner: what they would look at first when the drone was offered, what they would say and to whom, what they would want to know before accepting it, and what the Ground Commander would be asking for at the moment they were trying to give it back. The talk-through is where the analysis found that the hand back is not one operation but seven, that three of them require the Forward Controller to read the drone's state off three different pages to confirm it to the Mission Controller, and that "release" is a thing the Forward Controller does after the Mission Controller's acceptance, not at the same time as it, which is the gap "who has it" lives in.

The tree is written from the Forward Controller's point of view. Their own tasks for the ground element, the comms, the position, the security, are not decomposed here. They are the background the tree runs against, and at node 3 the plan says so.

  • 0Control the drone forward: receive it in the area, employ it, return it, move on 0: do 1; then 2; then 3 for the duration of the task; then 4; then 5. If 1 fails, decline, report, and do 5 without 2 to 4. If the link is lost during 3, do the lost-link drill (out of scope here) and then continue 3 or go to 4.
    • 1Receive the drone from the Mission Controller 1: do 1.1, 1.2 in order; do 1.3 when the Mission Controller offers; then 1.4; if 1.4 matches the brief, do 1.5, else decline and return to 1.2.
      • 1.1Confirm the handover point, the time and the drone's reported state
      • 1.2Bring up the control page and confirm the drone's identity and datalink
      • 1.3Acknowledge the offer of control
      • 1.4Verify the drone's mode, fuel and payload state against the brief
      • 1.5Accept control and confirm it to the Mission Controller and the Ground Commander
    • 2Bring it onto task 2: do 2.1, 2.2, 2.3 in order; then 2.4.
      • 2.1Command its pattern over the area of interest
      • 2.2Confirm it is holding the pattern
      • 2.3Set its sensor to the Ground Commander's first task
      • 2.4Report it on task to the Ground Commander and the Mission Controller
    • 3Employ it in the area 3: do 3.1 continuously; do 3.2 as the Ground Commander requires; do 3.3 whenever it raises an alert; do 3.4 at every change of task. All of 3 runs alongside the Forward Controller's own work for the ground element, which this tree does not decompose.
      • 3.1Monitor its position, fuel, link and mode
      • 3.2Task it: sensor task, new pattern, re-task
      • 3.3Respond to its alerts
      • 3.4Report to the Ground Commander what it sees and what it is doing
    • 4Hand it back to the Mission Controller at the edge of the area 4: do 4.1; then 4.2; then 4.3 to 4.6 in order; if 4.5 is not received inside the window, do 4.7 and report.
      • 4.1Confirm the task is complete or the element is about to move
      • 4.2Command it to the handover point in a safe state: mode, sensor, payload
      • 4.3Offer control to the Mission Controller
      • 4.4Confirm mode, fuel, payload and link state to the Mission Controller
      • 4.5Receive the Mission Controller's acceptance
      • 4.6Release control and tell the Ground Commander the drone is no longer held
      • 4.7Retain control: hold at the handover point, re-plan against its fuel, report
    • 5Move on with the element 5: do 5.1; then 5.2 and 5.3 in either order.
      • 5.1Clear the control page and restore the station's own picture
      • 5.2Confirm own comms, position and the move
      • 5.3Brief the Ground Commander for the move

Read the plans, not just the nodes. Plan 1 has a decline path. Plan 3 says the whole phase runs alongside a job this tree does not show. Plan 4 has a window and a failure branch, 4.7, that the procedure did not have until the talk-through asked what happens if the Mission Controller does not answer. And node 5.1 exists because both people, unprompted, said the first thing they would want to do after release is get the page off the screen. The tree found the question the rest of the work answers: node 4, under the conditions of node 5.

How to build an HTA of a role you do not hold

  1. Name the top goal from the role's point of view. Not "conduct the trial". What the person is there to deliver, in one line.New HTA session. The root node is 0; its Goal field is the sentence.
  2. Take the phases from the procedure, then check them against a person. The procedure gives the order. The talk-through gives the operations the procedure left out because they were obvious to whoever wrote it.Select the node, Add Child. Add Sibling for the next one at the same level. Numbering is automatic and never edited by hand.
  3. Write the plan on every parent, and put the failure branches in the plan. Where does the person go if the other party does not answer? If the plan cannot say, the procedure cannot either.The Plan field, or the plan editor: presets for Linear, Conditional, Parallel and Iterative, plus a formal expression the tool plays back.
  4. Draw the boundary in the plan, not by omission. If a whole job runs alongside this one and is not decomposed, say so at the node where it matters.The plan text on node 3; the reasoning in Notes.
  5. Attach what each operation needs. Which page, which person, which state has to be true first.Preconditions, Postconditions, Performer, Tools and HF Notes on the node.
  6. Validate with someone who has held the role. Read the plans aloud. Every "no, we would do that before" is a plan error, and there are always some.The standing Analysis Issues validator flags parents with one child and nodes without plans. Live walk records timings if you observe again.
  7. Find the branch the rest of the work will read. The tree is not the deliverable. It is where you decide what to model.Link HTA from ErgoAct picks the branch up as the storyboard's skeleton.
01-hta-forward-control.webpHTA tool with the forward-control tree, node 4 expanded to show 4.1 to 4.7 and the plan with its window and the 4.7 failure branch.
Five phases, one branch that matters. Node 4 is where the window is, and node 5 is what it collides with. The storyboard at 4.2 is built from this branch.

4.2ErgoAct: the hand back, alone and under the move

PracticeIn ErgoSphere

ACT-R is a cognitive architecture: a model of how perception, memory retrieval, decision and motor action take time and sometimes fail, built from decades of experimental data. Modelling a task in it has traditionally meant writing production rules by hand, which is why so few Human Factors programmes have used it. ErgoAct replaces the hand-written model with a storyboard: the task is laid out as a sequence of seven block types, Look, Listen, Recall, Decide, Act, Speak and Wait, each with its parameters, and the tool compiles the storyboard into a real ACT-R model and runs it.

The storyboard follows HTA node 4, operation by operation. Confirming that the task is complete is a Look at the sensor picture and a Decide. Commanding the safe state is three Acts through the control page's menus, each a Press with its distance and target size, and a Look to confirm each has taken. Offering control is an Act and a Speak. The state confirmation at 4.4 is where the storyboard grows: as the page is designed, mode, fuel and payload are on three different pages, so it is three Looks with a menu Act between each and a Recall of the handover point's identifier, which is set to Familiar rather than Well-learned because the point changes with the task. Then a Speak with the utterance length the controllers actually use. Receiving acceptance is a Listen and a Decide. Release is an Act and a Speak.

The second variant models the hand back under the move. It opens with the blocks the Ground Commander's question triggers, a Look at the sensor picture and a Speak, so the hand back is entered from somewhere else, and it inserts a Recall of where the hand back had got to, and a Decide to resume it, after the Ground Commander's second question lands between 4.4 and 4.5. The identifying Look at 4.4 is set to two similar items in view with the familiar-layout option off, because the drone's fuel page and the station's own status page are styled alike, which forces the serial search a real controller would make. Everything else is identical, and the operator profile on the Model tab, skill, fatigue and stress, is the same for both.

How to model a task in ErgoAct

  1. Start from the HTA operations. One block per operation to begin with; split an operation where it has distinct perceiving, deciding and acting parts.Link HTA, then drag from the STEP PALETTE onto the storyboard.
  2. Pick the block by what the person does. Look, Listen, Recall, Decide, Act, Speak, Wait. If you cannot name it, you do not understand the step yet.Each block's ACT-R mapping expander says what the architecture will do with it.
  3. Set the parameters honestly. How many similar things are in view, how well the item is known, how many options, how far and how big the target, how many words.The inspector on the right: Similar items in view, Familiarity, Number of options, Distance and Target size, Utterance length.
  4. Mark what genuinely overlaps. Two things done at once are only saved if the person can really do both.Runs with previous, on the second block.
  5. Fix the operator and the seed. The same profile for every variant, and a seed so the run can be reproduced.Model tab: Skill, Fatigue, Stress. Run tab: Iterations and Random seed.
  6. Pin, compare, sweep. Comparisons are what the model is good at; absolutes are estimates until measured.Pin baseline on the reference model; Compare models; Sensitivity over Stress, Fatigue or Skill.
  7. Read the generated model. If a reviewer cannot inspect it, it is an opinion with a progress bar.The ACT-R view, read-only and copyable.
02-act-storyboard-handback.webpErgoAct storyboard for the hand back alone: Look, Decide, Act, Speak, Look, Recall, Speak, Listen, Decide, Act blocks in sequence following HTA node 4, each with its predicted duration.
03-act-storyboard-handback-move.webpThe hand back under the move: the same sequence entered from the Ground Commander's question, with the inserted Recall and Decide, and the three-page state confirmation.
Alone and under the move. A different starting point and three inserted blocks. Everything else is identical, which is what makes the comparison fair.

The generated model is there to read. It is a real ACT-R model, declarative memory chunks and production rules, read-only and copyable. A reviewer who knows the architecture can check it; one who does not can at least see that there is something to check. That matters more than it sounds. A model nobody can inspect is an opinion with a progress bar.

04-act-model.webpThe generated ACT-R model for the hand back under the move, production rules visible, read-only.
Auditable, not a black box. The production rules the storyboard compiled to. Read it, copy it, defend it.

Each variant is then run five hundred times from the Run tab with a fixed seed, so the run is reproducible. The results page leads with six tiles: mean total time, the 5th to 95th percentile spread across runs, time spent waiting, retrieval failures per run with the share of runs that ended unrecovered, the busiest cognitive module, and task success, which says how many runs proceeded on a guess, omitted a step or aborted. Under them, the completion-time histogram, the step breakdown, and a findings list the tool writes itself. Two of its sentences on the interrupted run were the finding in a line: completion time varies widely between fast and slow runs, and a retrieval that fails after an interruption is the step most often omitted. The step it named was the resumption after the Ground Commander's question, and what was omitted, in the runs that omitted it, was the confirmation at 4.4. The alone model is pinned as the baseline so the interrupted histogram is drawn over it.

VariantMean5th95thRetrieval failures / runAgainst the 120 s window
Hand back, alone71 s58 s89 s0.2Inside at the 95th
Hand back under the move, as designed96 s74 s131 s0.7Outside at the 95th
05-act-results.webpErgoAct results page for the interrupted run with the alone run pinned as baseline: the six headline tiles, the completion-time histogram drawn over the baseline, and the findings list including the slow-tail and omitted-step sentences.
The tail fails. The interrupted mean is inside the window. The interrupted 95th percentile is not. Stage 1 said the tail was the question, and the model answered it.

4.3NASA-TLX and SART in the rig

PracticeIn ErgoSphere

Twelve weeks later, the trial rig: the forward station on a bench against a synthetic environment, with a rear station played by the trial team. Six Forward Controllers, two runs each in a balanced order, the baseline tasking and the forward-controlled mission, with the mission rated phase by phase: receive, employ, hand back, move on. The Ground Commander was played by a briefed role player who asked for the last of the picture and the move time at the moment the hand back began, because that is what a Ground Commander does. After every phase, NASA-TLX for perceived workload across its six scales, and SART for situation awareness across its three dimensions, both captured live in session in ErgoSphere, one session per phase and condition, named for it.

The trial ran on the control page after the changes from 4.4 had been made, because the model had already failed the original design and there was no point spending rig hours confirming it. That ordering is the entire argument for doing the analysis first.

How to run NASA-TLX and SART in session

  1. Define the conditions and balance the order. Every participant sees every condition; the order rotates so learning does not masquerade as a result.One session per condition, named for it.
  2. Rate immediately after the run, before the debrief. Memory of a run changes the moment it is discussed.The session captures each scale live; the participant rates on screen.
  3. Keep the debrief separate and write it down. What the rating cannot say, the participant will.Session notes against the run.
  4. Compare to the baseline, never to an absolute. A rating scale has no pass mark. It has a condition it is compared against.The session summary beside the baseline session.
07-tlx-session.webpNASA-TLX session named "Forward, hand back": the six scales rated for one controller's run, with the session's summary beside the baseline tasking's move-preparation session for comparison.
08-sart-session.webpSART session for the same phase: demand, supply and understanding, with the derived score against the baseline.
Rated, in session, by phase. One session per phase and condition, named for it, is what lets the baseline and the forward-controlled results sit side by side at stage 5.

4.4Back to the model: testing the fix before building it

PracticeIn ErgoSphere

The timeline view of the interrupted run put the added time in two places. The first was node 4.4: reading the drone's mode, fuel and payload off three pages, with a menu Act between each, and recalling the handover point, took a sequence that the Ground Commander's question could land anywhere in, and when it did, the Recall of where the sequence had got to sometimes failed. The second was the gap between 4.5 and 4.6: after the Listen for acceptance, the model had to Recall whether release had already been commanded, and under interruption that retrieval was the one most often wrong. Neither is a workload problem. Both are interface problems, and interface problems can be tested in the model before anyone builds them.

Two design changes were proposed and storyboarded. A handover summary: one page that presents the drone's mode, fuel, payload and link state together, with the handover point named on it, so 4.4 becomes one Look and a Speak and the Recall disappears. And a positive control-state indication, Mine, Offered or Theirs, that changes on the Mission Controller's acceptance and not on the Forward Controller's command, with release a deliberate confirmed action that is only offered once the state reads Theirs. In the storyboard the Recall before release becomes a Look, because the controller now reads the state instead of remembering it. The interrupted storyboard was saved as a new model, edited to match, and run with the same seed. Compare Models then puts the two persisted runs side by side without recomputing either, and a sensitivity sweep over stress checked that the tail stayed inside the window as the operator's stress setting rose, which matters more here than in most settings, because the person is in the field.

VariantMean5th95thRetrieval failures / runAgainst the 120 s window
Hand back under the move, as designed96 s74 s131 s0.7Outside at the 95th
Hand back under the move, with summary and control state68 s55 s84 s0.1Inside at the 95th
06-act-redesign.webpErgoAct Compare Models: the redesigned interrupted model's last run beside the original interrupted run, the 95th percentile now inside the window, with the stress sensitivity sweep below.
Designed in the model. Two interface changes, storyboarded and re-run in an afternoon, moved the 95th percentile inside the window twelve weeks before the rig could have said so.
5

Interpret the results

Plan 5: do 5.1, 5.2 in order. Return to 3 if any result raises a question the chosen methods cannot answer.

5.1Reading a cognitive model's output

Practice

Reading the result

A model output is a prediction under stated assumptions, and it is read in three moves. First, the tail against the criterion: the 95th percentile against the window, because that is what the requirement says. Second, the shape: a wide spread with retrieval failures means the task depends on remembering something the interface could be showing, which is a design finding rather than a workload one. Third, the timeline: where the time went, because that is what tells you what to change.

Then the honest sentence about what the model is not. It is not a measurement. Its absolute times are only as good as the storyboard's parameters. Its comparisons are much stronger than its absolutes, because the parameters are shared: interrupted against alone, and redesign against original, with everything else held constant. The report leans on the comparisons and treats the absolute times as an estimate to be confirmed at 4.3.

5.2Reading the rig ratings

Practice

Reading the result

NASA-TLX and SART are subjective. That is not a weakness to apologise for, it is a property to read correctly. The ratings say how the Forward Controllers experienced each phase, and they are compared to a baseline, never read as absolutes. On the redesigned page, the hand back rated higher on workload than the baseline tasking's move preparation, mostly on the temporal demand scale, and inside the margin the integration plan had agreed. Receive and employ rated close to baseline. SART held its baseline in every phase but one: two of the six rated the move-on phase below baseline on the understanding dimension, and their debriefs pointed at the same thing. After release, with the drone no longer theirs, they kept looking at the control page. It was still up, it was still showing an aircraft, and part of their attention stayed with it while the element was preparing to move.

That is a finding the model could not have produced, because the model stops when the task ends and does not know what it feels like to let go of something. It went into the register as a design issue: on release, the control page clears and the station's own picture comes back, which is what HTA node 5.1 had said the controllers wanted before anyone had measured why. It is the reason the rig trial was still worth running after the model had passed the design.

When the model and the trial agree, and where they do not

The model said the redesign brought the tail inside the window. The trial's measured times agreed. The model said nothing about the controller's attention after release, and the trial found it. Neither result replaces the other. The model bought twelve weeks and a better design to test. The trial found the thing only a person could find.

6

Write the user requirements

Plan 6: do 6.1 for every finding from 5 that needs a control, and for every clause from 2 that needs a design requirement; then 6.2 across the set; then 6.3 once.

6.1From finding to requirement

Practice

The anatomy is unchanged from Example 01: who, shall, what, under which condition, to what criterion, verified how. And the timing is the same: the set is written now, after the results, because the findings are where most of it comes from. The handover summary and the control-state indication became UR-03 and UR-04 the moment the model showed where the tail came from; the clean break on release became UR-07 the moment the debriefs explained the SART dip. What is different here is that the criterion for UR-02 comes from the operator's procedure, not from the standard. That is normal. The standard says the state must be displayed and the control must be deliberate; the procedure says how fast.

IDSourceRequirementVerified by
UR-01MIL-STD-1472H, displayed information; the handover procedure; HTA 4.5 to 4.6The Forward Controller shall be able to determine from the display, at any time and without a voice exchange, whether control of the drone is held by the forward station, offered, or held by the Mission Controller.Inspection, then test in the rig
UR-02The handover procedure; HTA node 4The Forward Controller shall complete the hand back, from offer to confirmed release, within the procedure's window in at least 95 per cent of trials while performing routine tasks for the ground element.Analysis (ErgoAct), then test in the rig
UR-03MIL-STD-1472H, displayed information; HTA 4.4The state the Forward Controller must confirm at handover, mode, fuel, payload and link, together with the handover point, shall be presented on one page without navigation.Inspection
UR-04MIL-STD-1472H, controls; HTA 4.5 to 4.6Release of control shall require a deliberate confirmed action, and shall not be available until the Mission Controller's acceptance has been received and displayed.Inspection, then test in the rig
UR-05The integration plan's agreed margin; HTA node 4 against node 5Perceived workload at the hand back shall not exceed the baseline tasking's move preparation by more than the margin agreed in the integration plan, measured on NASA-TLX.Test in the rig
UR-06The integration plan's agreed marginThe Forward Controller's situation awareness, measured on SART, shall not fall below the baseline tasking in any phase of the forward-controlled mission.Test in the rig
UR-07MIL-STD-1472H, displayed information; HTA 5.1On confirmed release, the control page shall clear from the forward station's display and the station's own picture shall be restored, without action by the Forward Controller.Inspection, then test in the rig

The HTA node is in the source column on purpose

A requirement that cites a clause says what standard it answers. A requirement that also cites an HTA node says what the person was doing when the need arose. The second is what a designer builds to, and it is what a reviewer two years from now uses to decide whether a proposed change to the page touches the requirement or not.

6.2Keeping the set honest

Practice

Every requirement has a criterion someone can fail. UR-02 is verified twice: first by analysis, so the design can be changed before the rig time is booked, and then by test, because a model is evidence for a decision and not a substitute for measurement. Writing both verification steps on the requirement, in that order, is what makes the model's role clear to a reviewer who is suspicious of models. UR-01 and UR-04 are the pair that answers "who has it": one says the display must show it, the other says the control must not let the Forward Controller get ahead of it. Either alone is half an answer.

6.3Recorded against the clause

In ErgoSphere

The seven go into the HF User Requirements Register, each in its requirement group, with its end-user group, its verification method and stage, an owner and a status of Open. The register does not point at a clause; it carries a source narrative in plain words, and the one worth noticing is UR-02, whose narrative names the operator's handover procedure rather than a standard. The procedure itself is held in the Document Register, so the narrative names a record and not a sentence in a report. The clauses' own side of the story, their compliance status and evidence, waits in the Requirements workspace until stage 7.

7

Close out

Plan 7: do 7.1 for every finding; then 7.2 for every applicable clause; then 7.3; then 7.4 once.

7.1Findings into the Issues Register

In ErgoSphere

Four issues carried the work, each raised from the session that found it so the provenance is the tool's and not typed. The handover summary and the control-state indication with confirmed release, both High, both owned by the vendor, both required before the page is accepted. The clean break on release, Medium, owned by the vendor, targeted at the baseline after next, because it is a display change and not a control change. And the failure branch at HTA 4.7, which the procedure did not have: Medium, owned by the operator's standards cell, to write the "hand back not acknowledged" drill into the procedure. Each links to the user requirements it answers, and the two vendor changes carry the ErgoAct comparison as evidence of effect. When the operator took the procedure change into its own process, that issue was set to Transferred, not Closed, and the register keeps the distinction wherever a number is shown.

10-issues-register.webpIssues Register for this project: the four issues with severity, status, owner, target and overdue columns, the procedure issue at Transferred, the two vendor changes Open with their linked user requirements in the detail pane.
Conditions of acceptance. The page was accepted with two issues open and owned. That is a decision the register can hold; an email chain cannot.

7.2Closing the loop on the clauses

In ErgoSphere

In the Requirements workspace the MIL-STD-1472H alerting clauses are set Compliant with the trial sessions under Linked Assessments and a professional judgement statement each. The displayed-information and controls clauses are Partially Compliant, with the summary and control-state issues raised against them, and will move when they close. The workspace's cards show the standard answered where it can be and open where it honestly is. The word in the report is "evidence provided against", and the claim of acceptability is the sponsor's to make on that evidence, which is exactly what they asked for at 1.1 without quite saying so.

7.3The EHFA and the HFIP, compiled

In ErgoSphere

The Early Human Factors Analysis from stage 1 and the Program Plan from stage 3 are finalised and signed, which freezes each as a dated, hashed version. The close-out wizard then builds the assurance report as an argument: each of the seven requirements closed with its evidence named, every issue traced to its requirement, and the four sub-claims positioned and reasoned. The Influence sub-claim, the one the wizard says should carry rejected recommendations, took the ErgoAIR and ErgoTrace work the sponsor chose not to fund this phase. The conclusion was Implemented with conditions, the two vendor issues being the conditions, and the completeness check listed the open High issues by name. The compiled deliverable is Part I, the assurance summary; Part II, the frozen EHFA; Part III, the frozen plan, with ErgoNeuro visible in its Planned and not applied section; then one appendix per session: the HTA, the three ErgoAct models, and the NASA-TLX and SART sessions from the trial, phase by phase.

7.4Issued, not emailed

In ErgoSphere

The reports were marked ready, composed into one package and issued as Revision A to the sponsor for the free Viewer. The reviewer who had been most sceptical of "a model" opened the generated ACT-R model in the Viewer, read the production rules, and sent the package back with one comment anchored to the ErgoAct assessment asking why the state confirmation Speak was as long as it was. Import a Returned Package verified the review chain against what had been issued and filed the comment where it was written. The answer, the controllers' actual phraseology timed in the talk-through, went into Revision B, and the comment was resolved against it. That is what an issued deliverable is for.

11-closeout-argument.webpClose-out wizard, The argument: the four sub-claims positioned with their reasoning, the Influence sub-claim carrying the unfunded ErgoAIR and ErgoTrace recommendations, and the conclusion Implemented with conditions.
Argued, not assembled. The unfunded recommendations sit in the Influence sub-claim on purpose. A programme in which everything was accepted reads as advocacy.
·

Common mistakes

What a less experienced practitioner would have done on this job, and what to do instead.

  • The mistake

    Answer "can the forward station take it" with a workload study of the whole mission, because the question sounds like workload.

    Instead

    Build the HTA first and read its plans. The employ phase was never the risk. The hand back, under the move, was, and only the tree could show that.

  • The mistake

    Model the hand back at a quiet station, because that is how the procedure describes it.

    Instead

    Model it interrupted. The hand back happens when the Ground Commander wants the picture and the element is packing up. A model of the procedure as written passes; a model of the procedure as done does not.

  • The mistake

    Report the mean hand-back time against the window and call it a pass.

    Instead

    Report the 95th percentile, because the requirement says 95 per cent of trials. The mean passed. The tail did not.

  • The mistake

    Wait for the rig, because a model is "just a model".

    Instead

    Model first, redesign in the model, then test the redesign. The rig confirmed a design that had already been fixed, instead of failing one that had not.

  • The mistake

    Treat "who has it" as a radio-discipline problem and leave it to the procedure.

    Instead

    Make it a display state. Voice confirms it; the display holds it. A controller interrupted mid-transaction must be able to look down and know, and the release control must not let them get ahead of what the display says.

  • The mistake

    Leave the control page up after release because "it is only a picture now".

    Instead

    Clear it. The trial showed the eyes kept going back to an aircraft the station no longer held, while the element was preparing to move. The HTA had node 5.1 before the trial explained why.

Do it yourself

The project this example was built in is issued as an ErgoSphere project file and as the Revision A deliverable. Open the deliverable in the free Viewer to walk the HTA, read the storyboards, the generated ACT-R model and the run results. Open the project file in ErgoSphere to edit the hand-back storyboard and re-run it with your own design change, or to decompose node 3 with the Forward Controller's own tasks and see what the employ phase looks like with nothing left out. Both are built for the scenario and carry no client data.

  • workbook-02-drone-handover.ergsphr · pending
  • workbook-02-drone-handover-revA.ergview · pending
Captures from ErgoSphere 1.6.1 · pending
To Top