Sue Hill | Chair Examination Committee Part I and OLA Subcommittee
The third full version of the On-Line Assessment (OLA) in anaesthesiology was held across Europe on 17 April 2015. For the second year running, European participants were joined by colleagues in Buenos Aires, Argentina. The assessment takes place in real-time using the web-based platform Ortrac, which was designed by Orzone, our Swedish partners. 654 candidates registered for the assessment and 588 attended. 20 countries provided access to this invigilated assessment with 60 centres in total receiving candidate registrations. The number of registered candidates per country varied from 1 in Armenia, to 146 in the Netherlands; Buenos Aires was the largest individual centre.
The OLA now has an established place in the Examinations Calendar, taking place in April. It grew from a partnership between Orzone, UEMS and ESA in 2011, with anaesthesiology the first specialty to run a small pilot assessment the same year. Since then it has grown in popularity: in 2013 we had just 203 candidates, this rose to 303 in 2014 and this year, 2015, we had almost 600 candidates. The OLA is a formative assessment designed to help anaesthesiologists identify areas where their knowledge needs improving and up-dating. No pass score is set by the OLA Subcommittee, although pass scores for the last five years for EDAIC Papers A and B are given in the introductory material for reference for candidates preparing for the EDAIC Part I. The OLA is not just used for examination preparation; the Netherlands use the assessment as an interim examination, setting their own pass scores and other countries are suggesting that senior colleagues can use this assessment as part of their evidence for revalidation. We feel this latter role may become more relevant as revalidation becomes essential for any doctor wishing to continue their practice.
This year, the OLA was offered in 9 languages: English as the master version, French, German, Italian, Polish, Portuguese, Russian, Spanish, Turkish. Portuguese and Italian were offered for the first time this year. As usual, the assessment consisted of 2 sections each containing 60 questions. The first section corresponded to Paper A, Basic Science, and the second to Paper B, Clinical Anaesthesiology. A candidate has a maximum of 90 minutes to complete each section – a total of 3 hours maximum for the complete assessment. It is possible to take less than 90 minutes for section A and move onto section B but this does not increase the time available for section B and it is not possible to re-visit the first section once completed.
The majority of the questions were specially constructed for the OLA by a dedicated question writing subcommittee of the Examinations Committee, the OLA Subcommittee. All questions map to the European Curriculum and covered all possible examinable domains. A small number of well-established questions from the EDAIC database were included to assess cohort ability compared with those sitting the EDAIC. Each question takes the same multiple true-false (MTF) format as the EDAIC with 5 options associated with each question stem. The questions in each section are presented to individual candidates in a random order with the 5 associated options also randomised to minimise the temptation to look at a neighbour’s screen!
Once the candidate has completed the assessment they are given their overall scores on the 2 sections and 20 minutes to go through the questions to see where they had answered incorrectly. A more detailed report of their performance, broken down into the sections comprising the EDAIC report for Papers A and B, is sent by email after the assessment.
The OLA results show that this year the overall mean score for section A was 65.2% and for section B was 69.9%; for comparison, the mean Part I scores for Papers A and B in the last 5 years are 73% and 75.7% respectively, with average pass scores of 70.7% and 73.8% for Papers A and B respectively. Performance on discriminator questions confirmed that the cohort of candidates sitting this assessment did not perform as well as those sitting the EDAIC. This is to be expected as many OLA candidates are not yet preparing for the EDAIC or are more senior clinicians who are using the OLA to see where their knowledge is a little rusty! Comparison of question performance indicators show that the question difficulty of the OLA was very comparable to an EDAIC examination and is therefore a good guide to potential success at EDAIC Part 1.
The table below shows: Countries, number of centres and number of registered OLA candidates.
Candidates routinely feedback that they would prefer the EDAIC to be on-line. As real-time access needs to be maintained across the world during on-line examinations, ensuring connectivity is key. Maximising security is also a priority for the ESA Examinations Committee as we prepare to move the Part I Examination to this platform. For a formative assessment if candidates cheat, they are only deluding themselves; for a summative assessment careers depend on passing the examination, so security is essential. There are still discussions to be had over such issues before the EDAIC Part I becomes an on-line examination.
Our thanks go to all the hosts who supervised the On-Line Assessment, it is now a permanent feature in the Examinations Calendar and we hope to increase the number of languages in which the OLA is offered to reach the current 11 languages in which the EDAIC Part I is offered. We need to add just Romanian and Hungarian.
Thanks also go to all the OLA Committee members, Dr Marcin Sicinski, Prof Wolfram Engelhardt, Prof Zekeriyya Alanoglu, Dr Armen Varosyan, Dr David Nagore Setién, Dr Paulo Sá Rodrigues, Dr Nicolas Brogly and our current Examinations Committee Chairman Dr Zeev Goldik as well as the ESA office staff who have had to absorb all the additional work this project has brought, including three telephone lines for support for potential issues with logging into the system. Orzone have provided an excellent platform for these assessments and thanks go to David Hedlund who has designed a very attractive interface to our question database and the Orzone team who were available for technical support for the entire time of the assessment. Without the contributions and cooperation of these people the project would not continue to thrive and expand.