The evaluation of a human-swarm interaction interface (SGP1AB)

Prolific ID: 639c8127verification | Study ID: 63b4a06fdemo | Session ID: study_demo
Information
(3 min) Consent
(2 min) Briefing
(8 min) Tutorial
(4 min) Scenario I
(8 min) Questionnaire
(5 min) Scenario II
(8 min) Questionnaire
(5 min) Finish
(2 min)

Participant Information Sheet

Study Title: The evaluation of a human-swarm interaction interface
Researcher(s): Ayodeji O. Abioye, William Hunt, Mohammad D. Soorati, and Sarvapali D. Ramchurn.
University email: a.o.abioye@soton.ac.uk
Ethics/ERGO no: ERGO/FEPS/85523
Version and date: Version 1.2 / 19 July 2023

What is the research about?
My name is Ayodeji, and I am a Research Fellow at the School of Electronics and Computer Science at the University of Southampton in the United Kingdom.

I am inviting you to participate in a study regarding the evaluation of a human-swarm interaction interface. A swarm is a collection of robots working together. In this study, we use drones (aerial robots) to perform parcel delivery to target locations in the Human And Robot Interactive Swarm (HARIS) simulator. The HARIS simulator is an online human swarm teaming tool used to test Unmanned Aerial Vehicle (UAV) swarm control algorithms and to study the human-swarm interaction. We have added an estimation feature to our HARIS simulation software to inform the operator of the predicted completion time of the parcel delivery task using the current number of drones. You can add a drone to the mission if the currently available number of drones would not be sufficient. But this is done at an extra cost to the mission. You are also able to remove a drone if the number of drones on the mission are sufficient to reduce the mission cost. We are interested in how the completion time estimation helps the swarm operator to minimise cost while maximising mission success.

This study aims to determine if this feature improves the performance and efficiency of the human swarm team. Our objective is to determine it limitations, usability, acceptability, workload, and performance gain before deploying this in a real-world setting. Our main research question focuses on whether the system improves task completion efficiency by minimising mission cost and optimising completion time in human swarm interaction. This study was approved by the Faculty Research Ethics Committee (FREC) at the University of Southampton (Ethics/ERGO Number: ERGO/FEPS/85523).

What will I be asked to do if I take part?
This study involves completing multiple simulation scenarios using our HARIS simulator and completing an anonymous questionnaire for each scenario, which should take approximately 60 minutes of your time. You participation would start with a short questionnaire for demographic data followed by a tutorial scenario to help you become familiar with the HARIS simulator. Subsequently, the first scenario and its questionnaire, followed by the second scenario and its questionnaire are presented. Lastly, we will inquire a few final questions before completing the study. If you are happy to participate in the study, proceed to the next page and tick (check) the box to show your consent. As this survey is anonymous, the research team will not be able to know whether you have participated, or what answers you provided.

Why have I been asked to participate?
You have been asked to take part because you are signed up to the Prolific crowdsourcing platform for research participants. You have been randomly invited to take part in this study. You must be aged at least 18 years to participate. You would be required to watch a short study briefing/tutorial video and asked to answer three simple questions to ensure you have understood this to be included in the study. There would also be validation check questions to exclude anyone who randomly click through survey questions.

I am aiming to recruit around 120 participants for this study.

What information will be collected?
The questions in this survey ask for information in relation to standardised questionnaires such as the NASA Task Load Index, Usability, and Acceptability survey. The HARIS simulator would collect performance data as log files which would consist of your start and finish time, when agents are added or remove, whether mission was completed successfully or not, and the mission cost.

Demographic data such as gender, education level, computer expertise, and UAV or Swarm robotics familiarity would also be collected in order to determine how representative of the population and generalisable our results are. This is also used to find any correlations or patterns, as well as determine the diversity of the data set.

Some of the survey questions contain textboxes where you will be asked to type in your own answers. Please note that in order for this survey to be anonymous, you should not include in your answers any information from which you, or other people, could be identified.

You have to answer all the questions in the survey in order to complete this study.

What are the possible benefits of taking part?
If you decide to take part in this study, you will receive a monetary compensation via the Prolific platform upon successfully completing the study and your participation will contribute to knowledge in this area of research. Please note that if you failed the attention check questions, you would not be rewarded, and your submission would be discarded.

Are there any risks involved?
It is expected that taking part in this study will not cause you any psychological discomfort and/or distress, however, should you feel uncomfortable you can leave the survey at any time.

What will happen to the information collected?
If you decide to take part in this study, you will not receive any direct benefits; however, your participation will contribute to knowledge in this area of research.

Your participation will help me understand the way humans react to scenarios where the swarms are controlled autonomously or by humans. It will help me analyse the approaches needed for a trustworthy human-robot interaction system. You will also get a reward for the time that you have dedicated to the study. But you will only get the reward on the successful completion of the study. The payment will be in the form of prolific credits.

What are the possible benefits of taking part?
All information collected for this study will be stored securely on a password protected computer and backed up on a secure server. In addition, all data will be pooled and only compiled into data summaries or summary reports. Only the research team will have access to this information.

The information collected will be analysed and written up for presentation at conferences and/or publication in a journal.

The University of Southampton conducts research to the highest standards of ethics and research integrity. In accordance with our Research Data Management Policy, data will be held for 10 years after the study has finished when it will be securely destroyed.

What happens if there is a problem?
If you are unhappy about any aspect of this study and would like to make a formal complaint, you can contact the Head of Research Integrity and Governance, University of Southampton, on the following contact details:
Email: rgoinfo@soton.ac.uk
Phone: +44 2380 595058

Please quote the Ethics/ERGO number above. Please note that by making a complaint you might be no longer anonymous.

More information on your rights as a study participant is available via this link:
https://www.southampton.ac.uk/about/governance/participant-information.page

Thank you for reading this information sheet and considering taking part in this research.

Participant Consent and Information

Thank you for choosing to take part in this study. Please complete this consent form before taking part in the study. It is required for your participation. This study has been approved by the Faculty Research Ethics Committee (FREC) at the University of Southampton (Ethics/ERGO Number: ERGO/FEPS/85523).

Participant Consent

I have read and understood the participant information sheet.

I am aged 18 or over and agree to take part in this study.

I understand that my participation is voluntary and I may withdraw (at any time) for any reason without my participation rights being affected.

Demographic Information

What is your gender?

Which of the following age group do you belong?

What level of education do you hold?

What is your level of computer expertise?

How familiar are you with Unmanned Aerial Vehicles (UAVs) and/or Swarm robotics?

Study Briefing

In this study, you would play the role of an Unmanned Aerial Vehicle (UAV) swarm operator performing a drone delivery mission.

  1. You would be able to add UAVs or remove UAVs from the mission.
  2. The more UAVs you have, the faster you complete the delivery mission. However, adding UAVs increases the mission cost while removing UAVs reduces the mission cost.
  3. Your aim is to maximise performance (finish before time runs out) and minimise total mission cost.

After this, you would be required to complete a questionnaire to document your experience. You need to perform this experiment on a laptop or desktop computer, using an updated version of the Google Chrome web browser.

Please watch the following Youtube video which demonstrates what is expected of you in this experiment (preferably in Fullscreen, 1080p HD). This should take about 7 minutes to complete. When finished, answer the validation check questions to confirm you understood the video, and click Next to continue.

Validation Check

Please answer the following questions based on the briefing video above.

In the tutorial scenario, how many minutes were given by the timer in the top left corner?

Please complete validation check to proceed.

How many minutes were given to complete the first study scenario (the control condition)?

What is the maximum number of agent you can add to a mission?

HARIS Simulator Tutorial

This is the Tutorial study scenario. In this scenario, you would play the role of an Unmanned Aerial Vehicle (UAV) swarm operator performing a drone delivery mission:

  1. Your mission is to complete 8 drone deliveries in 2 minutes with a limited number of UAVs (mininum - 3, maximum - 10).
  2. You have a budget of £ 2000 for hiring and maintaining UAVs for the mission.
  3. You would be able to add UAVs or remove UAVs from the mission at any point during the experiment.
  4. The more UAVs you deploy, the faster your completion time and the higher your mission cost. The fewer UAVs you deploy, the longer your completion time and the lower your mission cost.
  5. Your aim is to maximise performance (finish before time runs out) and minimise total mission cost.

Can you complete this mission incuring the minimum possible cost? Click the Blue "Start Tutorial scenario" button to start the Tutorial study scenario. When finished, tick the "I have completed the Tutorial scenario" checkbox and click Next to continue.

Start Tutorial
Please confirm you have completed scenario to proceed.

Study Scenario I

This is the SNV study scenario. In this scenario, you would play the role of an Unmanned Aerial Vehicle (UAV) swarm operator performing a drone delivery mission:

  1. Your mission is to complete 40 drone deliveries in 6 minutes with a limited number of UAVs (mininum - 3, maximum - 10).
  2. You have a budget of £ 2,000 for hiring and maintaining UAVs for the mission.
  3. You would be able to add UAVs or remove UAVs from the mission at any point during the experiment.
  4. The more UAVs you deploy, the faster your completion time and the higher your mission cost. The fewer UAVs you deploy, the longer your completion time and the lower your mission cost.
  5. Your aim is to maximise performance (finish before time runs out) and minimise total mission cost.

Can you complete this mission incuring the minimum possible cost? Click the Blue "Start SNV scenario" button to start the SNV study scenario. When finished, tick the "I have completed the SNV scenario" checkbox and click Next to continue.

Start SNV Scenario
Please confirm you have completed scenario to proceed.

Study Scenario I Questionnaire

This is the SNV study scenario questionnaire. Please complete the form below and click next to continue.

Task Index

How mentally demanding was the task?

How physically demanding was the task?

How hurried or rushed was the pace of the task?

How successful were you in accomplishing what you were asked to do?

How hard did you have to work to accomplish your level of performance?

How insecure, discouraged, irritated, stressed, and annoyed were you?

Usability Index

I think that I would like to use the system more frequently.

I found the system unnecessarily complex.

I thought the system was easy to use.

I think that I would need the support of a technical person to be able to use this system.

I found the various functions in this system were well integrated.

I thought there was too much inconsistency in this system.

I would imagine that most people would learn to use this system very quickly.

I found the system very cumbersome to use.

I felt very confident using the system.

I needed to learn a lot of things before I could get going with this system.

Acceptance Index

The system is

The system is

This is an attention check. Please select number 3.

The system is

The system is

The system is

The system is

The system is

The system is

The system is

Trust Index

The system is deceptive.

The system behaves in an underhanded manner.

I am suspicious of the system's intent, action, or outputs.

I am wary of the system.

The system's actions will have a harmful or injurious outcome.

I am confident in the system.

This is an attention check. Please select number 4.

The system provides security.

The system has integrity.

The system is dependable.

The system is reliable.

I can trust the system.

I am familiar with the system.

Study Scenario II

This is the SFV study scenario. In this scenario, you would play the role of an Unmanned Aerial Vehicle (UAV) swarm operator performing a drone delivery mission:

  1. Your mission is to complete 40 drone deliveries in 6 minutes with a limited number of UAVs (mininum - 3, maximum - 10).
  2. You have a budget of £ 2,000 for hiring and maintaining UAVs for the mission.
  3. You would be able to add UAVs or remove UAVs from the mission at any point during the experiment.
  4. The more UAVs you deploy, the faster your completion time and the higher your mission cost. The fewer UAVs you deploy, the longer your completion time and the lower your mission cost.
  5. Your aim is to maximise performance (finish before time runs out) and minimise total mission cost.

Can you complete this mission incuring the minimum possible cost? Click the Blue "Start SFV scenario" button to start the SFV study scenario. When finished, tick the "I have completed the SFV scenario" checkbox and click Next to continue.

Start SFV Scenario
Please confirm you have completed scenario to proceed.

Study Scenario II Questionnaire

This is the SFV study scenario questionnaire. Please complete the form below and click next to continue.

Task Index

How mentally demanding was the task?

How physically demanding was the task?

How hurried or rushed was the pace of the task?

How successful were you in accomplishing what you were asked to do?

How hard did you have to work to accomplish your level of performance?

How insecure, discouraged, irritated, stressed, and annoyed were you?

Usability Index

I think that I would like to use the system more frequently.

I found the system unnecessarily complex.

I thought the system was easy to use.

I think that I would need the support of a technical person to be able to use this system.

I found the various functions in this system were well integrated.

I thought there was too much inconsistency in this system.

I would imagine that most people would learn to use this system very quickly.

I found the system very cumbersome to use.

I felt very confident using the system.

I needed to learn a lot of things before I could get going with this system.

Acceptance Index

The system is

The system is

This is an attention check. Please select number 3.

The system is

The system is

The system is

The system is

The system is

The system is

The system is

Trust Index

The system is deceptive.

The system behaves in an underhanded manner.

I am suspicious of the system's intent, action, or outputs.

I am wary of the system.

The system's actions will have a harmful or injurious outcome.

I am confident in the system.

This is an attention check. Please select number 4.

The system provides security.

The system has integrity.

The system is dependable.

The system is reliable.

I can trust the system.

I am familiar with the system.

Feature Index

How accurate was the time estimation in this scenario?

Study Conclusion

Thank you for participating in our study. Please complete the final survey below and click submit to finish and confirm study completion. You would be automatically redirected back to prolific with a study completion code.

Relative Comparison

Which scenario did you think you were more successful at?

Why did you think you were more successful in the scenario choosen above? (Select all that apply)

Which of these reasons selected above was the most important? (Select only one)

What do you think can be improved in this study?

Did you use the estimated completion time feature?

Please provide comments or feedback in the textbox below (optional).