Report to NHS Education for Scotland on work to develop a benchmark assessment in numeracy for nursing


Sponsor:
Michael Sabin, RGN BSc (Hons) MSc (Education) PG Cert. Learning Teaching and Assessment, Programme Director, NHS Education for Scotland, currently on secondment as Nursing Officer Workforce Development, Chief Nursing Officer’s Directorate, Scottish Government

Research team:

  • Professor Diana Coben, BA, DipACE, PhD, FRSA, FHEA, Professor of Adult Numeracy, King’s College London

  • Dr Carol Hall, RGN, RSCN, PhD, BSc (Hons) PG Dip Ad/Ed PG Dip ResM, FHEA, Associate Professor, School of Nursing, University of Nottingham

  • Dr Meriel Hutton, RGN, RNT, PhD, BA, CertEd, DN, Senior Visiting Research Fellow, King’s College London

  • Dr David Rowe, Phd, MSc, BEd (Hons), FACSM, FRC, Reader in the Department of Sport, Culture and the Arts, University of Strathclyde

  • Dr Keith Weeks, RN, PhD, BSc (Hons), BEd, DipN, RNT, FHEA, Reader of Health Professional Education, Faculty of Health, Sport and Science, University of Glamorgan

  • Norman Woolley, MSc, MN, RGN, DipN, PGCE, RNT, Head of Learning and Teaching; Associate Head Dept of Professional Education and Service Delivery, Faculty of Health, Sport and Science, University of Glamorgan



Background


The development of competence in numeracy by healthcare staff and students of healthcare programmes within higher education is a key area for concern, as shown by successive studies (Jukes, 2006). The NPSA, in partnership with the British Medical Journal (BMJ) Publishing Group, has provided a standardised online educational package for junior doctors (NPSA, 2006) but as yet nothing has been instigated for nurses and other healthcare professionals.

Medication calculations are an important part of the numeracy required for healthcare. Medication errors are an aspect of clinical governance highlighted recently by the National Patient Safety Agency (NPSA) in England and Wales and have been targeted for remedial action over recent years (NPSA, 2006a; NPSA, 2006b). The number of injuries and deaths attributable to medication error in the NHS is unknown but 9% of incidents reported to the NPSA in its annual audit involved medicines (NPSA, 2003); and this is consistent with historical data. While miscalculation is not separately identified as an aspect of medication error, the NPSA do identify that

Miscalculation, failure to titrate the dose to the patient’s needs, miscommunication and failure on the part of all team members to check the dose before dispensing, preparing or administering a dose are the most common factors contributing to dosing errors. (NPSA, 2009, p.19)

Education and training is identified as contributory both positively and negatively. The Department of Health report on
Improving Medication Safety (Smith, 2004) put some of the blame for medication errors on inadequacies in the education and training of both doctors and nurses.

In Scotland,
Safe Today, Safer Tomorrow (NHS QIS, 2006) focused on patient safety and risk management, including medication error and recommended that NHS Quality Improvement Scotland (NHS QIS) should take responsibility for co-ordinating the development of a structured action plan, including costs and timescales, in order to take this work forward (NHS QIS, 2006, p.16). NHS QIS has a lead role in the promotion of patient safety and works in partnership with the National Patient Safety Agency (NPSA) and other patient safety organisations internationally. NHS QIS is leading and co-ordinating the Scottish Patient Safety Programme (SPSP) within which a key initial goal was to drive improvements in Medicines Management.

In the context of this concern about calculation competence in the healthcare professions, there have been calls both for benchmarks on requirements for entry to programmes of preparation and for some absolute measure of achievement and performance at the point where the student or trainee formally becomes a practising professional. Indeed, from September 2008 the body regulating the Nursing profession in the UK, the Nursing and Midwifery Council (NMC), has required students to achieve 100% in a test of numeracy in practice (NMC, 2007a) before they are allowed to register as nurses. In addition to testing at the point of registration, the NMC requires universities to judge nursing students’ mathematics ability at Entry to Programme and, more recently, also at Entry to Branch (Adult, Child, Mental Health, Learning Disability) (NMC, 2007b).

Unfortunately, there are currently no national standards for teaching or assessment of numeracy during pre-registration nurse education and, in the absence of a robust evidence-based standard (a benchmark), a relativistic position has emerged with a multiplicity of tests, processes and criteria being developed and deployed locally, all ostensibly offering fixed points against which competence could be determined or predicted.

We argue that without such a benchmark, any measure of numerical competence is:

... in the eye of the recipient of evidence of that competence, be it higher education institutions, regulators, employers or service users. (Hutton, 2004)


There are several linked issues here:

  1. The a priori or foundation mathematical skills which a prospective nurse should have before commencing professional training;
  2. Progressive competence development towards a fixed point at which competence must be shown;
  3. The requirement for a benchmark;
  4. The placement of the benchmark;
  5. The nature of such a benchmark.

The first point proposes that without a particular set of (usually foundational) mathematical skills the prospective student will not reach the level of numerical competence which will be required to meet the collective (public and professional) expectations and requirements of the registered professional. Such markers of future capability are common within academic study and constitute the minimum grades sought for programme entry. Many Higher Education Institutions (HEIs) set specific grades in mathematics within their requirements for entry to their pre-registration Nursing programmes on the assumption that a particular level of school mathematics qualification can act as a reliable proxy for future academic capability. However, the results of a study by Hutton suggest that qualifications in mathematics, other than those of above GCSE grade C or equivalent, should not be relied upon to predict performance in a test of calculations required in nursing (Hutton, 1998a). In any case, without a reliable benchmark for competence it is not possible to link entry criteria to subsequent competence development and this may result in either unnecessarily restrictive entry criteria or a failure to identify additional learning needs in our student body.

The second point recognises that, as a core outcome of the educational process, both general conceptual and operationally specific learning will occur and be measurable if appropriate learning, teaching and assessment processes are in place. The duration of educational programmes are generally set not just to allow us to fill students with all the knowledge and skills we believe they will need, but also to allow a step-by-step approach to the development of understanding and competence over time. Thus, an understanding of the key skills required and the students’ existing capabilities allows appropriate support and intervention across the duration of their programme of professional preparation.

The third point focuses on why such a benchmark might be needed. This gets to the heart of the rationale for professional, as opposed to purely academic, preparation. Professional competence is articulated not just in the quality of the individual’s thinking, but in the way that such thinking is put to the service of those who are in receipt of the professional’s services. Notwithstanding an individual professional’s decision to undertake or withhold any action, if the professional act includes technical skills, then it is this technical competence which becomes the embodiment of the required and expected competence. In this way, the recent focus upon safety in healthcare has been the stimulus for both education to enhance competence and for measures which might demonstrate and govern high quality clinical performance. A benchmark therefore seeks to provide both a target towards which competence development can be directed and a robust measure against which its achievement can be governed and assured.

If such a benchmark can be set, at what point should it be deployed? If professional competence is to be characterised in public protection and safety terms (and the NMC clearly sets its stall in this domain), then the logical point is that at which the student transforms into the professional. The point of registration is, by its nature, the point at which the individual professional accepts their accountability for their acts and omissions as measured against a set of (sometimes specific, sometimes rather general) standards. Up until that point, the individual’s learning is transitional from the novice state to that which meets the agreed expectations.

Given the link between the required competence and its public consumption, the nature of any benchmark necessarily requires not just an assessment method which is reliable and valid in educational terms, but one which directly and authentically represents its purpose and the context in which it will be performed. Such a process does not undermine the transferability of learning but firmly cements the relationship between the desired expectation of competence and the governance of its development and subsequent performance.

Once established, a robust assessment benchmark provides not just some assurance of baseline professional standards but for the first time an opportunity to move away from relativistic interpretations of numerical competence and explore the relationship between entry qualifications, in-programme preparation, placement experience, remedial support and subsequent achievement. Thus, it would be possible to work backwards from this fixed criterion to look at key stage indicators and the probability of achieving the required standard. Such data would be extremely powerful in supporting subsequent educational intervention. Further, it would also allow us to look forward to subsequent stages in the individual’s career where either higher level skills require to be built on, or a refreshing of core skills can be facilitated by recourse to the core standard.

Against this background, following on from a review of relevant literature (Sabin, 2001) and recommendations from a consultation on healthcare numeracy (NES Numeracy Working Group, 2006; Sabin, 2006a; Sabin, 2006b) and in line with a strategy developed by NES (Sabin, 2006a; Sabin, 2006b), NHS Education for Scotland (NES) brought together an interdisciplinary group of subject experts from across the UK to explore the key issues associated with determining the achievement of competence in numeracy for professional practice in nursing.

The overall aim of this work is to develop a proposed benchmark assessment for numeracy for nursing in Scotland. In the first instance we propose the establishment of a benchmark at entry to the NMC Register because this is the point at which students become registered nurses, accountable for their professional practice within the NMC’s regulatory framework.

Our work towards this aim is outlined in the remaining sections of this report.


Research Report


Benchmark assessment of numeracy for nursing: Medication dosage calculation at point of registration


Aim
The aim of the study was to evaluate empirical evidence of the reliability and convergent validity of a computer-based learning and assessment tool of medicine dosage calculations by comparing its outcomes with the outcomes of a practical activity requiring the same calculations and to determine learner-perceived acceptability of the assessment tools in relation to authenticity, relevance, fidelity and value. This was intended as a first step towards creating a benchmark for numeracy for nursing.

Research questions

  1. What is the internal consistency reliability of the computer-based assessment and the practice based assessment?
  2. What is the criterion-related validity of the computer-based assessment and the practice-based assessment?
  3. How acceptable to nursing students are the assessments in terms of authenticity, relevance, fidelity and value?

Experimental protocol and methods
The research began with preliminary work undertaken between November 2006 and April 2007 (reported in Coben, 2008). This preliminary work comprised:

  • adoption of a definition of numeracy applicable to nursing (Coben, 2000, p.35);

  • development of evidence-based principles for numeracy for nursing;

  • development of criteria for a benchmark assessment of numeracy for nursing;

  • development of an evidence-based prototype benchmark assessment tool covering practical knowledge in medicine dosage calculations


Instrument Design
Having reviewed the literature, we recognised that an authentic numeracy assessment tool should be:

  • Realistic: Evidence-based literature in the field of nursing numeracy (Hutton, 1997; Weeks, 2001) strongly supports a realistic approach to the teaching and learning of calculation skills, which in turn deserve to be tested in an authentic environment. Questions should be derived from authentic settings.

  • Appropriate: The assessment tool should determine competence in the key elements of the required competence (OECD, 2005; Sabin, 2001).

  • Differentiated: There should be an element of differentiation between the requirements for each of the branches of nursing (Hutton, 1997).

  • Consistent with adult numeracy principles: The assessment should be consistent with the principles of adult numeracy learning teaching and assessment, having an enablement focus (Coben, 2000).

  • Diagnostic: The assessment tool should provide a diagnostic element, identifying which area of competence has been achieved, and which requires further intervention (Black & Wiliam, 1998).

  • Transparent: The assessment should be able to demonstrate a clear relationship between ‘test’ achievement and performance in the practice context (Weeks, Lyne, Mosely, & Torrance, 2001).

  • Well-structured: The assessment tool should provide a unique set of questions with a consistent level of difficulty and a structured range of complexity. (Hodgen & William, 2006)

  • Easy to administer: the assessment should provide the opportunity for rapid collation of results, error determination, diagnosis and feedback (Black & William, 1998).

(Coben et al., 2008, pp 96-97)


In response to these requirements, the evidence-based benchmark computer assessment tool was designed by Weeks and Woolley to develop nurses’ medication dosage calculation skills using a framework derived from the Authentic World® computer programme. The assessment tool uses graphics and participant interaction to create close proximity to real world practice. It covers the full range of complexity within the hierarchy of dosage calculation problems which Adult Branch nurses are likely to meet at the point of registration. This includes unit dose, sub- and multiple-unit dose, complex problems and conversion of Systeme Internationale (SI) units (Weeks, Lyne, Mosely, & Torrance, 2001). The 28 practical simulation items were a subset of 50 computer simulation items selected on the basis of their high internal consistency reliability in an earlier study with students from Case Western Reserve University, USA, and the University of Glamorgan (Clochesy, 2008; Weeks & Woolley, 2007).


The practical simulation assessment was designed and developed to mirror the computer-based assessment.


Design and development of the medication dosage calculation problem-solving competency model

The generic numeracy definition adopted by the NES expert numeracy reference group was that defined by Coben as follows:

To be numerate means to be competent, confident, and comfortable with one’s judgements on whether to use mathematics in a particular situation and if so, what mathematics to use, how to do it, what degree of accuracy is appropriate, and what the answer means in relation to the context.
(Coben, 2000, p. 35, emphasis in the original)

For the purposes of extrapolating and operationalising a definition of dosage calculation competence from this definition of numeracy, the reference group adapted and crystallized a competency model derived from an initial premise described by Weeks, Lyne & Torrance
(2000) and elaborated by Authentic World Ltd. (Figure 2, below).

Healthcare numeracy is a subset of a wider numeracy and medication dosage calculation is both a subset of numeracy and of a wider competency in medicines management (Figure 1).

fig1

Figure 1. Medication dosage calculation in the context of Numeracy and Medicines Management (Sabin, et al., 2008)


Medication dosage calculation was chosen because this is the most commonly cited exemplar for nursing numeracy. The public and professional focus on medication-related calculation is linked to patient safety since incorrect calculation of medication dosage may harm patients and, by default, the reputation of the profession and professional education.

While acknowledging the existence of wider medicines management factors that permeate the prescription, dispensing, preparation and administration of medicines, this competency model developed in the project focused specifically on medication dosage calculation problem-solving (Figure 2). However, once such a benchmark is established, it should be possible to create further benchmarks in numeracy for nursing at other key points in nurse training and to extend these to include other areas of numeracy for nursing, such as fluid balance, nutrition, management of nursing provision, etc..

fig2

Figure 2: Medication dosage calculation problem-solving competency model



This model articulates the inter-relationship between the three sub-elements that combine to form competence in medication dosage calculation problem-solving:

  1. Conceptual competence: The need to identify, interpret, extract and understand the relationship between the medication nomenclature, dosage and numerical information embedded in medication prescriptions and medication container labeling/monographs; and to set up an accurate and appropriate dosage or rate equation that illustrates the relationship between the essential elements of the dosage problem.
  2. Calculation competence: The need to undertake appropriate arithmetical operations and computations to calculate a numerical value that falls within an appropriate degree of accuracy for the required dose or rate.
  3. Technical measurement competence: The need to select an appropriate measurement and delivery vehicle or vehicles and to transfer and measure the calculated numerical value for the correct dose or rate to be administered.

The confluence (central white interface) of the model highlights how all three sub- elements must be present in order to achieve a correct dosage or rate solution. Conversely, if an error is made within any one or more of the sub-elements then a medication dosage calculation error will result.

Design and development of the computer-based authentic assessment data collection tool
The computer based authentic assessment data collection tool designed for use in the NES research project was derived from an original constructivist-based program (Weeks 2001; Weeks et al., 2001) and Authentic World Ltd. Figures 3 – 7 illustrate the original Authentic World Ltd design features of the constructivist based program and how this reflects iconic representations of the three sub-competency elements of the competency model:

fig3

Figure 3: Tablet and Capsule Dosage Calculation Competence

fig4

Figure 4: Oral Liquid Medicine Dosage Calculation Competence

fig5

Figure 5: Injection Dosage Calculation Competence

fig6

Figure 6: Intravenous Infusion (mL per Hour) Calculation Competence

fig7

Figure 7: Intravenous Infusion (drops per minute) Calculation Competence


The original constructivist-based Authentic World Ltd program reflected a diagnostic assessment of cognitive competence (European Commission for Education & Culture 2008) within the medication dosage calculation problem-solving competency domain. This was subsequently adapted for the purposes of designing an assessment and data collection tool for use in this project. The assessment and data collection tool focused specifically upon the presentation of medication prescriptions, medication containers and labels and the technical measurement element of the competency model. The rationale for this adaptation was to:

  1. Provide a framework within which a prospective competency benchmark in medication dosage calculation at the point of registration could be articulated and evaluated;
  2. Provide an appropriate assessment platform that could be utilised by students who had been exposed to education programmes not based on constructivist pedagogies.

Figures 8 and 9 illustrate an injection problem from the adapted model developed as an exemplar assessment tool and used for data collection purposes in the project.


fig8

Figure 8: Stage 1 of the assessment exemplar tool

fig9

Figure 9: Answer representation phase of the assessment exemplar tool

Design and development of the Practical Simulation Assessment data collection tool
Alongside the computer-based authentic assessment tool a practical simulation model was developed to mirror the key assessment elements. The equipment purchased to allow testing of these items in simulated practice in a maximum of two centres at any one time consisted of:

  • 10 medicine cabinets containing:
- appropriately labelled bottles for placebo tablets (6 per cabinet);
- bottles for placebo liquid medicines (6 in each);
- 6 x 10 mL ampoules of water for injection (appropriately labelled as placebo drugs);
  • syringes in a range of sizes;
  • needles for drawing up injections;
  • plastic quills for drawing up liquid medicines;
  • plastic medicine tots (10 ml and 30 ml capacity);
  • placebo tablets;
  • food colouring to mix with water for placebo liquid medicine;
  • 10 mL plastic ampoules of water for injection for placebo drugs.

In both types of assessment, the medication dosage problem-solving questions consisted of:

  • tablet and capsule, liquid medicine and injection problems presented in a hierarchy of unit dose, sub- and multiple unit dose, complex problems and conversion of SI unit problems;
  • intravenous (IV) infusion problems focused on millilitre per hour calculations for administration of IV fluid solutions via volumetric pump delivery devices; and drop per minute calculations for administration of IV fluid solutions via a variety of IV administration set devices.

Questions were generated from a bank of 600 authentic medication dosage and IV infusion problems identified and extracted from clinical practice settings. The sample of questions used in the programme of research was selected on the basis of pre-assessed reliability and internal consistency as calculated by the statistical advisor to the project from data reported by Clochesy (2008): r = .98 and Weeks & Woolley (2007): r = .92.



Methods and Methodology



The methods and methodology utilized in the project may be summarized as follows in relation to the research questions:

1. What is the internal consistency reliability of the computer-based assessment and the practice based assessment?

This was evaluated with Adult Branch nursing students in their 3rd year by administering a set of 40 computer-based assessment items and 28 matched practice-based assessment items. Cronbach’s alpha was then calculated for the total set of dosage calculation items, and on subsets of items (e.g., tablets and capsules, etc.), for both forms of test (computer-based and practice-based)

2. What is the criterion-related validity of the computer-based assessment and the practice-based assessment?

This was evaluated with Adult Branch nursing students in their 3rd year by comparing their performance on the computer-based assessment tool with a practice-based assessment using a modified Objective Structured Clinical Examination (OSCE) format with the students undertaking the same calculations.

3. How acceptable to nursing students are the assessments in terms of authenticity, relevance, fidelity and value?

This was investigated by reviewing students’ online evaluation of the assessment tools they have experienced.

The project commenced investigation of these questions with a Pilot Study in a university in England; this is reported on in Annexe 1. In the main study, all universities in Scotland with pre-registration nurse training programmes were invited to participate. Assessment tools were refined in the light of experience on the pilot study and the practical equipment was moved to Scotland for the main study. Practical testing equipment was set up similarly in each site and students were assessed according to a generic practice schedule provided for the data collectors. Students had access to the practice schedule before participating in the assessment as part of the information given at prior presentations.

Stages of the research study
The research process was structured as follows:

  • Pilot study (March-April, 2008)
- Stage 1: Materials testing
- Stage 2: Piloting of the assessment of 3rd year Nursing students in a university in England

  • Main Study
- Stage 1 (April – September 2008)
- Stage 2 (September 2008 – June 2009)
- Stage 3 (July – December 2009)


Pilot study (March - April, 2008)
The Pilot Study is summarised below (full details are in the Pilot Study report, attached below as Annexe 1 – see also Hutton et al., pub. pending).

The Pilot Study enabled a practical evaluation of both methods and materials with a typical sample group of final year student nurses within a large HEI similar to those planned for the main study. The university’s research ethics committee gave ethical approval for the study. By undertaking the Pilot in England we ensured that Pilot data would not contaminate the main study.

Student performance in 28 medication calculation tasks was assessed using computer-based and practice-based formats. Both formats endeavoured to simulate clinical reality. The computer assessment was based on the Authentic World computer programme which had already been extensively tested for face and content validity (Weeks, Lyne, Mosely & Torrance, 2001; Weeks, 2001) and research item internal reliability and the practice simulation exercise involved materials similar to those used in the local NHS Trust.

A computerised 50-item assessment (10 tablet & capsule; 10 liquid medicine; 10 injection; 10 two-part IVI = 50 items)
was undertaken by all volunteering Adult Branch nursing students at commencement of their third year of study. Twenty eight of these were selected (a random stratified sample) to complete the final stage: a further computer assessment of the twenty eight most reliable set of items (drawn from the results of the baseline computer assessment) matched to the practice-based assessment.

Main Study – Stage 1 (April – September, 2008)
The main study took place in Scotland where six HEIs expressed interest in participating. These universities all included departments with students of nursing in their third year of studies, and all agreed that students could be approached to volunteer to participate in the study, subject to ethical clearance. In the event, student volunteers from only four HEIs participated, due to logistical reasons. Stage 1 entailed adoption of the agreed ethical approval by interested Scottish universities, with extension where required, and recruitment and initial baseline assessment of students, from which a sample undertook Stage 2.

Main Study – Stage 2 (September 2008 - June 2009)
Stage 2 of the Main Study comprised cross-over assessment of the sample established in Stage 1 (n = 63) in which half the students undertook the final computer assessment followed by the practical test and the other half undertook the assessments in reverse order.

Main Study – Stage 3 (July – December 2009)
Stage 3 comprised analysis and interpretation of results.

Sample Frame for the Main Study
The total population of new third year Adult Branch students at the volunteering HEIs in Scotland during the data collection period was estimated at 1000 across both Autumn and Spring entry cohorts.

Sample for the Main Study
The numbers of students involved in the Main Study were as follows:

  • estimated number of eligible participants: 500;
  • number who undertook the initial online assessment and were invited to participate: 80;
  • number who participated: 64;
  • number who completed the assessment process (i.e., both online and practice): 63.

There may be several reasons for the small number of students who participated; this requires further investigation. The difference between those participating and those completing is explained by one student who felt unable to continue following the online assessment.

Ethics and Access

Ethics
The plan for attaining ethical approval was discussed extensively within the research team and also directly with research leads from the university nursing departments who were considering participation. A major difficulty to be resolved was that while the NHS has cross-boundary approval arrangements, this is not the situation between nor even within universities. Because the study was a multi-site project, ethical approval would theoretically be needed individually from every site used. We decided to base the project plan for ethical approval on work undertaken by Lauder et al (2007) as that required a similar multi-site arrangement. Initial contact was made to HEIs through introductory workshops which showcased the intended project within Scotland, and through telephone correspondence with directors of HEI pre-registration Nursing programmes. It was agreed that full ethical approval could be applied for through the English University. Once this had been granted, each individual school was contacted with details of the submission and the initial acceptance letter; the Scottish universities then determined what type of submission for approval was needed for their institution. This took time but worked well, with shortened submission arrangements successfully negotiated.

In designing the project, ethical concerns were specifically identified within three areas. The first related to the outcome of the assessments undertaken, if the tool subsequently demonstrated validity. There was a risk that student nurses who thought that they had performed badly might experience feelings of inadequacy or low self esteem. Secondly, there might be individuals who performed sufficiently badly for their potential professional practice to be brought into question. A final issue related to the imposition of an assessment which might be perceived as increasing the learners’ existing workload or compromising their summative assessment schedule.


In order to resolve these issues the following mechanisms were incorporated into the project.


  1. The outcome of the assessment: Students were advised that the assessment was deliberately timed to allow them to address any shortfalls in their knowledge identified by the assessment. Students were offered feedback and support after the assessment through access to the HEIs’ own facilities and, in negotiation with the Schools, through additional access to the computer learning package Authentic World® from which the online assessment was originally derived. Information given to participating HEIs included the possibility that some students would need support in relation to drug dosage calculation. This was reiterated at presentations to the lecturers in each School. They were also advised that all data would be confidential, i.e., known only to the researchers. Students were offered opportunities to discuss the project at any stage with the project team.
  2. Poor achievement in the assessment: Assessment data from this project would not contribute to any summative assessment within each School. The students’ existing curricula had already been validated and reviewed conjointly by the NMC and the HEIs as an effective means for the preparation of student nurses. The project sought to illuminate methods and potential for future development which may, if deemed significantly valid, be adopted by each HEI as part of its curriculum. While trends in the data would be fed back to schools with the option of inclusion in the development of their curriculum, this decision would remain that of the HEI within the overall context of its governing drivers. In this context it would not be possible to identify final outcomes in relation to professional practice for individual students who have not completed their programme of study, merely to identify the needs of the cohort in relation to further development prior to completion. This anonymised information was thus offered to the participating HEIs.
  3. Issues around workload: Participation in the project would necessarily take up some student time in order to be completed effectively. However, we recognised that students would have to learn about medicine dosage calculation and practise this as part of their existing curriculum. It was expected that learning outcomes from participation in the project would match existing provision designed to meet NMC (2007) requirements. It was anticipated that in some circumstances students would have the opportunity to accredit learning from the study in place of existing preparation for the calculation and administration of medicines. Further, as the project ran at different times from students’ existing study sessions, they would not be disadvantaged.


Access

Once the work had received full ethical approval, we needed to access students for the following purposes:

  • A 1 hour introductory session to outline the project and recruit volunteers. In this meeting students were given a presentation about the project and also invited to take a volunteers’ information form advising them about the project and their potential role within it. They were also invited to take a consent form to sign and return.
  • A period of time for students to reflect on the original presentation before committing themselves to participation.
  • A 2 hour online baseline assessment of all volunteers’ performance on the online assessment developed for the project.
  • A 2 hour further computer assessment and a 2 hour practical assessment for a sample of students selected on the basis of their scores.
  • A short online questionnaire to evaluate students’ perceptions of the usefulness of the two activities.

Research protocol

In each of the main study sites we assessed:

  • All student volunteers using the computerised assessment programme.
  • Up to 10 students (depending upon number volunteering) via practical assessment (Day 1) followed by a computerised assessment (Day 2). 
  • Up to 10 students (depending upon number volunteering) via the computerised assessment (Day 1) followed by the practical assessment (Day 2).

In some cases the model was slightly adapted, within the limits of ethical approval, in order to suit the needs of individual HEIs.


Data Collection
Data collection took place at each of the Main Study HEIs (14 days data collection in total). Students received feedback on their work after all data were collected and were offered access to the full Authentic World® computer programme for the remainder of their course of study.


Practical Issues
In line with ethical clearance, the member of the team working for NES was included in the data collection only where the results of the assessment could not be seen. This meant he could participate only in the introductory presentations and in the computer assessments (where data were submitted centrally online).


We negotiated access to the computer facilities and skills laboratories which would be needed at each site. We compiled a set of equipment necessary for the practical assessment and this was sent to each site by courier in time for the data collection activity. This meant that all materials and documentation were the same at each site. The one exception was the use of local infusion pumps (a) to avoid transporting these and (b) to provide students with familiar equipment.


The Assessment Process
The assessment process comprised computerised assessment of numeracy items associated with administering medication and an assessment of identical items in a simulated practice setting. Students completed the assessments in a counterbalanced order (i.e., some completed the computer simulation first, then completed the practical simulation test, and other participants completed the tests in the opposite order). A break was given between tests.

The assessment items encompassed:

  • Tablets and Capsules,
  • Liquid Medicines
  • Injections
  • IV Infusions

Within each section, items ranged in difficulty, namely

  • unit dose,
  • sub and multiple unit dose,
  • complex calculations
  • SI unit conversion


new_rubric

Table 1: Problem rubric


IV infusions were subdivided into responses given in mL/hr and drops/minute (i.e., the same problem was answered in different units).

Participants responded to 50 items in the computer-based assessment and 28 items in the practical assessment; the latter were a subset of the 50 items tested via computer simulation. The computer-based assessment problem rubric illustrating medication domains, levels of complexity and numbers of questions within each is illustrated in Table 1.


The Computer-based Assessment Process
In our introductory presentation we demonstrated the computer tasks (please see the screen slide prints (figs. 8 and 9 as included within the previous section) of the computerised assessment for examples of questions). Students were then given the opportunity to volunteer and contact details of the research teams were made available to students in case they had any questions about the proposed assessment.

Prior to the assessment participating students were given access to personal computers in computer laboratories and password access to demonstration materials, together with support in learning how to use the computer package.

Students were asked to undertake the computer-based assessment alone. One hour was allowed for the initial computer assessment although some students finished more quickly.

During the online assessment the students were observed by the researchers who were also available to clarify the process. Responses were submitted by the students to a central database where they were collated and marked automatically by the computer program.

The Practice Based Assessment Process
The practical assessment activities were also outlined in the introductory presentation. The medication dosage problem-solving tasks consisted of a total of 23 prescriptions requiring 28 calculations, which comprised:


  • 6 each of: tablet and capsule; liquid medicine; and injection problems. These included unit dose, sub- and multiple-unit dose, complex calculations and conversion of SI units (18 calculations in total).
  • 5 IV infusion prescriptions each requiring millilitre per hour calculations for administration via volumetric pump and drop per minute calculations for administration via a variety of IV administration drip sets (10 calculations in total).


Over two hours, students were assessed at two workstations, one for IV infusion calculations and one for liquid medicines, tablets and capsules, and injections. Each workstation had all the equipment required for completion of the assessment.

We designed the layout for the practical assessment of tablets and capsules, liquid medicines and injections so that we could see all the students but each student was unable to see what other students were doing (see Figure 10). In this position students could act as they would in practice and help from the researchers would be available quickly should they require it. This arrangement also allowed researchers to remove completed medication trays for marking without disturbing other students.

room_layout

Figure 10: Layout of room (accommodating 8 workstations)

Medicine dosage calculations Identical workstations with a medicine cabinet and a selection of medicine pots, syringes and needles at each were arranged on tables. The tables were arranged so that the open medicine cabinet acted as a screen between adjacent students. Paper and pencil were provided for any working out needed. Also on the table were three colour-coded dinner trays, each holding six small papier-maché trays and a prescription sheet containing six prescriptions (formatted similarly to those used in the NHS). One sheet was of prescriptions for tablets and capsules, one liquid medicines and one injections. For each, students had to select the correct preparation from the medicine cabinet and prepare it as for presentation to a patient.


fig11
Figure 11: Colour-coded dinner tray - at start
fig12

Figure 12: Completed tray (for medicines and capsules task)


When the student had prepared the prescriptions from one sheet, the completed tray was removed for checking, leaving the student with space to work through the next prescription sheet. We recorded students’ answers on a results sheet. One researcher acted as invigilator for the whole room while up to two others were required to collect and record students’ results.

Intravenous fluid rates
For calculations involving IV fluid rates, a prescription sheet with five fluid prescriptions was provided to each student. Students were asked to calculate two rates per prescription – one in millilitres per hour for entry into the volumetric pump and the other for drops per minute. They then set the rate by adjusting the manual roller on a giving set and inputting the correct numbers on a volumetric pump. Following pre-pilot materials testing, it was considered impractical for the researcher to count the drops each time and so students were asked to tell the researcher how many drops they were aiming for and this was recorded. This was done by the student noting their intended answer on a piece of paper and showing the examiner, thus avoiding the risk that other students might overhear the answers given. Many students lacked experience in setting rates on volumetric pumps and so again the researcher keyed in the numbers as dictated by the student, using the same approach as above. Students’ answers were recorded on their results sheet as they finished each task.

Calculators
Students were not permitted to use calculators. We do not object in principle to nurses using calculators (see Hutton, 1998b), but disallowing calculators enabled us to standardize the testing procedures across all students.


The Evaluation Instrument
After the completion of the Main Study students were emailed a questionnaire regarding their perception of their experience of the assessment process. The questionnaire design was informed by Gulikers’ 5 dimensional framework for authentic assessment (Gulikers, 2004). This framework was developed to inform and evaluate the design of assessment authenticity, with professional practice as the starting point. The five dimensions: task; physical context; social context; result/form; and criteria; subsequently provided a practice-focused framework for evaluating students’ perceptions of the authenticity and representativeness of the assessment environments. However, in the event, a disappointingly small response rate precluded this element from our final analysis.

Data processing and analysis
Data processing

Following completion of the two forms of assessment, the response of each participant on each item was marked as correct (entered as a 1 in the data) or incorrect (entered as 0 in the data) for the purposes of summary data analyses (see explanation of data analyses provided below). Where incorrect responses were given, field notes and observations were consulted to diagnose sources of error for item-level diagnostic analyses (see explanation of data analyses provided below).

Two sets of computer data were considered. The ‘full set’ included all items (50 items total; 10 items per subscale). The ‘reduced set’ included only those items that were common to the computer and practical simulation tests (28 items total; 6 items per subscale for tablets and capsules, liquid medicines, and Injections, 5 items per subscale for IV infusions mL/hr and IV infusions drops/min).

Two sets of practical simulation data were considered. In the first set, responses were marked based on the volume of liquid present, regardless of the position of the syringe plunger. In this scenario, technical error was penalised. For example, if 5mL of liquid was needed in the syringe, a response was marked incorrect if the participant aligned the head of the syringe plunger with the correct volume measurement bar on the syringe barrel (5 mL), but considerable air was introduced into the syringe barrel (resulting in an incorrect volume of liquid in the syringe). This data set is denoted by the abbreviation MA. In the second set, responses were marked based on the numerical answer provided on the instrument, regardless of whether the correct volume of liquid was present. In this scenario, technical error was not penalised. For example, if 5mL of liquid was needed in the syringe, a response was marked correct if the respondent aligned the head of the syringe plunger with the correct volume measurement bar on the syringe barrel (5 mL), even if considerable air was introduced into the syringe barrel. This data set is denoted by the abbreviation NUM.

This differentiated marking system affected responses to the Liquid Medicines and Injections items, in which syringes were used to provide the answer; it did not affect the tablet and capsule or IV infusion items. These two methods of marking the responses were followed to recognise the difference between the arithmetical component of the problem and the technical component of the problem (please see the Discussion section for a fuller explanation of the tie-in with the paradigm of numeracy problem-solving developed during the course of the NES study).

The responses to the two forms of the test were analysed in two fundamentally different ways, namely the ‘traditional’ or ‘summary’ data analyses, and the ‘item-level’ or ‘error-diagnosis level’. The traditional/summary data analyses answered the global questions regarding internal consistency reliability of the tests and subtests, and regarding the extent to which total score on the computer assessment matched total score on the equivalent items administered using the practical simulation (i.e., criterion-related validity). The item-level/error diagnosis analysis involved detailed inspection of the number and nature of errors at the item level, in order to inform how use of the two different test forms might identify problem-solving errors arising from different parts of the competency model as illustrated in Figure 2.


Summary data analyses

Summary data analyses were conducted on combined item scores, either on the complete version of the test (all items), or at the sub-test level (items on the tablets and capsules, liquid medicines, injections and IV infusions subtests).

Analysis Set 1 consisted of various descriptive statistics (minimum score, maximum score, mean, standard deviation, skewness and kurtosis). This gives a general idea of the range of responses, whether the data were relatively normally distributed (indicating whether the use of parametric data analysis is appropriate), whether some sub-tests were more difficult than others, and enables future comparison to performance by other samples.

Analysis Set 2 was the internal consistency reliability analyses. Internal consistency is the degree to which items within a test differentiate consistently between respondents with high or low levels of numerical competency. Cronbach’s alpha (intraclass correlation coefficient from a 2-way ANOVA model) was used, which is equivalent to the Kuder-Richardson (KR-20) coefficient traditionally used with dichotomous (correct/incorrect) knowledge test items.

Analysis Set 3 consisted of Pearson correlations between the various forms of the tests. The most important correlations were those between the computer simulation and practical simulation tests. This analysis is central to the main research question of whether computer simulation items are an accurate substitute for practical simulation questions (the issue of criterion-related validity evidence). Correlations tell us about
relative validity (do the two tests put people in a similar order?)

Analysis set 4 included mean comparison of the two forms of assessment (computer simulation and practicals), using repeated measures
t-tests and Cohen’s D. Similar to the above, this pertains to the criterion-related validity of the computer simulation form of the test, but from the perspective of absolute validity, namely whether the computer simulation format gives similar scores to the practical simulation format, or systematically higher/lower scores. The t-test is an inferential significance test that tells us whether any mean difference is due to chance or is more likely a systematic bias that exists in the population, when responding to the two forms of the test. Because repeated measures t-tests can be overpowered in larger samples, especially if there is a high correlation between the two sets of scores (as was expected in the current study), this can lead to what is called a Type I error (a conclusion that a mean difference is statistically significant, even though it may be of trivial clinical or practical importance). The Cohen’s D estimates the clinical meaningfulness or importance of a mean difference by standardising it against the within-group variability. Cohen provided the following guidelines for interpreting values of D: 0.2 = small effect; 0.5 = medium effect; > 0.8 = large effect (Cohen, 1968).




Results


Demographic information on participants
In the main phase of the study, 63 third year Adult Branch nursing students from 4 different institutions completed both forms of the dosage calculation test.

Summary analyses of subscale and total scale scores
Summary analysis Set 1: Descriptive statistics – total test scores and sub-tests. Descriptive statistics on the total scale and subscales are presented in Table 2. Mean scores were generally high (i.e., close to the maximum possible score), especially for the total test score and the Tablets and Capsules and Liquid Medicines subscales. It is also notable that the standard deviations were lower for these subscales. Together, these two statistical results (means and standard deviations) indicate that these sub-tests (Tablets and Capsules and Liquid Medicines) were easier for the participants, and that their scores were therefore less variable (i.e., most participants answered correctly on most items in the Tablets and Capsules and Liquid Medicines subscales).

The data were normally distributed (skewness and kurtosis < |2.0|), meaning that they met the distributional assumptions necessary for conducting the later parametric analyses (correlations and
t-tests). This also has implications for the use of these items and similar items in future applied research studies that involve the use of parametric data analysis (e.g., t-tests, ANOVAs, Pearson correlation, etc.). Although slightly more non-normally distributed, most of the sub-tests also could be analyzed using parametric data analyses. In comparison to data previously collected on similar students in Wales (used to select items for the practical simulation in the current study), scores in this sample were generally higher (except for the IV subscales) and more homogeneous. This has relevance to later results reported in this section.

Table 2: Descriptive statistics for test performance on main sample (N = 63) Test


# items
Min
Max
Mean
SD
Skewness
Kurtosis

Computer Simulation Full:








Total

50

19
50

40.4

8.9
-0.8
-0.6

T/C

10

7
10

9.6

0.7
-1.7
2.1

L/M

10

5
10

9.1

1.2
-1.2
1.0

INJ

10

6
10

8.8

1.3
-0.8
-0.6

IV-ML

10

0
10

7.3

3.6
-1.2
-0.1

IV-DP

10

0
10

5.6

4.2
-0.4
-1.6

Computer Simulation Reduced:








Total

28

10
28

22.7

4.8
-0.8
-0.3

T/C

6

4
6

5.7

0.6
-1.9
2.4

L/M

6

3
6

5.6

0.7
-1.9
3.5

INJ

6

3
6

5.1

1.0
-0.7
-0.6

IV-ML

5

0
5

3.6

1.9
-1.1
-0.5

IV-DP

5

0
5

2.7

2.1
-0.2
-1.7

Practical Simulation MA








Total

28

9
28

23.1

4.3
-1.0
0.9

T/C

6

4
6

5.8

0.5
-3.0
7.6

L/M

6

1
6

5.1

1.2
-1.4
1.4

INJ

6

0
6

4.6

1.6
-1.0
0.1

IV-ML

5

0
5

4.2

1.4
-2.0
3.2

IV-DP

5

0
5

2.9

2.0
-0.4
-1.4

Practical Simulation NUM








Total

28

9
28

22.6

4.4
-1.2
1.4

T/C

6

4
8

5.9

0.6
-1.4
7.0

L/M

6

1
6

5.3

1.1
-1.8
3.3

INJ

6

0
6

4.9

1.4
-1.5
2.3

IV-ML

5

0
5

4.2

1.4
-2.0
3.2

IV-DP

5

0
5

2.9

2.0

-0.4

-1.4

Notes: T/C = Tablets and capsules, L/M = Liquid medicines; INJ = injections; IV = Intravenous infusions; ML = mL/hr; DP = drops/min

Summary analysis set 2: Internal consistency reliability. Results for the reliability analysis are presented in Table 3. Although reliability was generally high for the total scale scores (for all forms of the test, i.e., Computer Simulation Full, Computer Simulation Reduced, Practical Simulation MA and Practical Simulation NUM), it was low for several of the subscale scores. This was due to two factors. First, several items were automatically excluded from the analyses due to zero variance (i.e., every student answered the item correctly). Calculation of correlation statistics in these circumstances is impossible because of the incorporation of item variance in the mathematical computations (and therefore the impossibility of dividing and multiplying by zero). Second, the variability (standard deviations) of these subscales was consequently reduced (and reliability tends to be lower in tests with a lower number of items). For subscales that had a complete item set in the analysis (and for the total scale scores), reliability was moderate to high, and similar to the reliabilities calculated on the data previously collected on similar students in Wales. Generally, the IV sub-tests were more reliable than the other sub-tests, and the sub-tests taken by the practical simulation method were slightly more reliable than the same sub-tests taken via computer simulation.

There are two reasons why the internal consistency was better in the Welsh data than in this study. First, there were 10 items per sub-test in the Welsh data, whereas this study had shortened sub-tests of 5-6 items. Second, and more influential, was the effect of the “zero-variance” items in the main study. This did not happen in the Welsh data – every item had several people who got a wrong answer (so there were zeroes in every item, and there were more zeroes overall in the previous data set, creating greater variability and therefore higher correlations (internal consistency reliability estimates).

Table 3: Internal consistency reliability for test performance on main sample (N = 63)

Test

Original # items
Items excluded*
Final # items
α

Computer Simulation Full:





Total

50

1, 2, 3, 4, 11, 13, 15, 21, 22, 23

40

.94

T/C

10

1, 2, 3, 4

6

.36

L/M

10

11, 13, 15

7

.56

INJ

10

21, 22, 23

7

.55

IV-ML

10


10

.95

IV-DP

10


10

.95

Computer Simulation Reduced:





Total

28

1, 3, 11, 22, 23

23

.89

T/C

6

1, 3

4

.45

L/M

6

11

5

.37

INJ

6

22, 23

4

.45

IV-ML

5


5

.90

IV-DP

5


5

.89

Practical Simulation MA





Total

28

1, 3, 5, 6

24

.84

T/C

6

1, 3, 5, 6

2

.83

L/M

6


6

.62

INJ

6


6

.73

IV-ML

5


5

.82

IV-DP

5


5

.87

Practical Simulation NUM





Total

28

1, 3, 5, 6

24

.85

T/C

6

1, 3, 5, 6

2

.68

L/M

6


6

.64

INJ

6


6

.69

IV-ML

5


5

.82

IV-DP

5


5

.87

Notes: α = Cronbach’s coefficient alpha; equivalent to KR-20 in dichotomous data
* Items excluded from reliability analysis due to zero variance

Summary analysis set 3: Intercorrelation between tests. These results are presented in Table 4. Although these results are briefer than the others, they are among the most important results. The significant, and meaningful, correlations between the computer simulation format (full and reduced) and the practical simulation format (MA and NUM-scored versions) indicate that the computer simulation format tends to put testees in a similar order to the practical assessment format (i.e., higher scorers on the computer simulation also tend to be higher scorers on the practical simulation, lower scorers on the computer simulation also tend to be lower scorers on the practical simulation, middle scorers on the computer simulation also tend to be middle scorers on the practical simulation). This supports the proposition of using a computer simulation for testing dosage calculation skills in trainee nurses. Importantly, these intercorrelations (all between
r = .73 and .77) were estimated from a reduced item set of 28 items (because of the limitation on the number of items that could be tested in the practical simulation test), whereas the proposed computer simulation test would have a greater number of items (closer to 50), in which case the “true” correlation between the computer simulation and the practical simulation (if an equivalent item set of 50 items were used for both) would be higher. Using the Spearman-Brown prophecy formula, this would approximate to correlations of between r = .83 and .86.

Although not reported in this section, the intercorrelations between the practical simulation and computer simulation for separate subscales were lower (these are contained in the
t-test results, but range from r = .29 to .76). This is to be expected because these consist of far fewer items.

Table 4: Pearson correlations between different forms of dosage calculation tests on main sample (N = 63)


Computer Simulation (All)
Computer Simulation
(Reduced)
Practical Simulation MA
Practical Simulation NUM

Computer Simulation (All)

1.00




Computer Simulation(Reduced)

.98

1.00


Practical Simulation MA

.73

.74
1.00

Practical Simulation NUM

.76

.77
.97
1.00
Note: All correlations were significant (p < .01)

Summary analysis set 4:
t-tests comparing mean performance. Due to the fact that the full computer simulation test contained 50 items, it would have been inappropriate to compare mean scores of the full computer simulation test to the practical simulation test, which had a reduced set of items, therefore only computer simulation scores from the reduced version were compared to the practical simulation. This was done for the total test score and each subscale score. These results are presented in Tables 5a through Table 5f.

Table 5a: Mean comparison of Computer Simulation (MA and NUM scoring) vs. Practical Simulation, total score (N = 63)

table4a

Results statement: Mean total scores from the Computer Simulation test were not significantly different (p > .05) from mean total scores on the Practical Simulation test. Differences were clinically trivial (Cohen’s D ≈ 0.02-0.09)

Table 5b: Mean comparison of Computer Simulation (MA and NUM scoring) vs. Practical Simulation tablets and Capsules sub-test score (N = 63)

table4b

Results statement: Mean scores on the Tablets and Capsules questions were significantly (p < .05) higher on the Practical Simulation than on the Computer Simulation. However, this corresponded to a clinically trivial to moderately meaningful difference (Cohen’s D ≈ 0.2-0.4)

Table 5c: Mean comparison of Computer Simulation (MA and NUM scoring) vs. Practical Simulation, Liquid Medicines sub-test score (N = 63)

table4c

Results statement: Mean scores on the Liquid Medicines questions were significantly (p < .05) lower on the Practical Simulation than on the Computer Simulation. This corresponded to a clinically moderately meaningful difference (Cohen’s D ≈ 0.3-0.5)

Table 5d: Mean comparison of Computer Simulation (MA and NUM scoring) vs. Practical Simulation, Injections sub-test score (N = 63)

table4d

Results statement: Mean scores on the Injections questions were significantly (p < .05) lower on the Practical Simulation MA than on the Computer Simulation. This corresponded to a clinically moderately meaningful difference (Cohen’s D ≈ 0.4). Mean scores on the Injections questions were not significantly (p > .05) different between the Practical Simulation NUM and the Computer Simulation.

Table 5e: Mean comparison of Computer Simulation (MA and NUM scoring) vs. Practical Simulation, IV sub-test score, mL/hr (N = 63)

table4e

Results statement: Mean scores on the IV (mL/hr) questions were significantly (p < .05) higher on the Practical Simulation than on the Computer Simulation. This corresponded to a clinically moderately meaningful difference (Cohen’s D ≈ 0.4)

Table 5f: Mean comparison of Computer Simulation (MA and NUM scoring) vs. Practical Simulation, IV sub-test score, drops/min (N = 63)

table4f

Results statement: Mean scores on the IV (drops/min) questions were not significantly different (p > .05) between the Computer Simulation and Practical Simulation versions. The mean difference was clinically trivial (Cohen’s D ≈ 0.1)

Similar to the previous results, these also are very important results, as they assess the
absolute validity between the two forms of the test – namely, on average, do participants get similar scores from the computer simulation and practical simulation (or alternatively, does one test result in higher scores, indicating a systematic bias)?

The results here are again very promising for the total test score. Average score over all 28 items is essentially the same regardless of whether the questions are answered using the practical simulation or computer simulation format. They are statistically not significantly different and clinically/practically not meaningfully different (as indicated by trivial effect size). The same is not always true for the separate sub-tests. In some cases (e.g., Tablets and Capsules), the practical simulation format resulted in a higher mean score (though the difference was relatively minor). In other cases (e.g., Liquid Medicines), the computer simulation format resulted in a higher mean score than the practical simulation format. In other cases (e.g., IV drops/min), there was no difference between mean scores on the two test formats. The irregular pattern of results, in addition to the often relatively small differences (even though sometimes statistically significant), combined with the often homogeneous data (small standard deviations increase statistical power) lead to the conclusion that any differences between formats (computer simulation vs. practical simulation) on the sub-tests is coincidental, or a function of sampling variability (random differences).

Item-level data analyses

The following section provides:

  1. Summary tables which compare congruence (criterion-related validity) between participant performance on assessment items within the computer simulation and the practical assessment. Operational definitions for congruence were manifestation of a CORRECT performance on an item in the computer assessment and on its counterpart item in the practical assessment AND manifestation of an INCORRECT performance on an item in the computer assessment and on its counterpart item in the practical assessment (denoted in green in the tables). Operational definitions for incongruence were manifestation of a CORRECT performance on an item in the computer assessment and an INCORRECT performance on its counterpart item in the practical assessment, and vice-versa (denoted in red in the tables). As described previously, within the liquid medicines and injections sections a further level of evaluation of performance was undertaken and variation denoted using the abbreviations MA and NUM. This is reflected in the inclusion of two tables for both these sections.
  2. Item by item histograms, analysis and commentary on individual assessment items. Individual item analyses includes the complexity level of the dosage/rate problem, the correct solution to the problem, the congruence level between the computer and practice based assessments expressed as a histogram and a percentage value, and the range of incorrect solution values manifested by participants.

This data and item-by-item representation provides a detailed discrete and comparative analysis of within section participant performance and the congruence (criterion-related validity) between the two assessment environments.

Tablets and Capsules Section Summary

Table 5: Tablet and Capsule Summary Results

 

Congruent

Outcome

Incongruent

Outcome

 

Question

% Computer & practice performance correct

% Computer & practice performance incorrect

% Computer performance correct/ Practice performance incorrect

% Computer performance incorrect/ Practice performance correct

Total

1 (1)

63 (100%)

0 (0%)

0 (0%)

0 (0%)

63 100%

2 (3)

63 (100%)

0 (0%)

0 (0%)

0 (0%)

63 100%

3 (5)

62 (98.4%)

0 (0%)

0 (0%)

1 (1.6%)

63 100%

4 (6)

61 (96.8%)

0 (0%)

0 (0%)

2 (3.2%)

63 100%

5 (9)

53 (84.1%)

4 (6.4%)

0 (0%)

6 (9.5%)

63 100%

6 (10)

48 (88.9%)

4 (7.4%)

2 (3.7%)

0 (0%)

54 100%

Sub-Total

94.7%

2.3%

0.6%

2.4%

100%

Congruence Totals

97%

3%

100%

 

Participants: Problems 1 – 5 (n = 63); Problem 6 (n = 54)

Review of results:

Question 1

image001

  • Problem: unit dose: Prescribed 20mg / Dispensed 20mg: 1 Tablet
  • Correct Solution: 1 Tablet
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 100%
  • Computer environment errors: 0
  • Quasi-Practice environment errors: 0

Question 2

image002

  • Problem: unit dose: Prescribed 7.5mg / Dispensed 7.5mg: 1 Tablet
  • Correct Solution: 1 Tablet
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 100%
  • Computer environment errors: 0
  • Quasi-Practice environment errors: 0

Question 3

image003

  • Problem: multiple unit dose: Prescribed 4mg / Dispensed 1mg: 1 Tablet
  • Correct Solution: 4 Tablets
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 98%
  • Computer environment errors: 1 (1 tab)
  • Quasi-Practice environment errors: 0

Question 4

image004

  • Problem: multiple unit dose: Prescribed 100mg / Dispensed 50mg: 1 Tablet
  • Correct Solution: 2 Tablets
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 97%
  • Computer environment errors: 2 (0.5 tab)
  • Quasi-Practice environment errors: 0

Question 5

image005

  • Problem: SI Unit Conversion: Prescribed 1.5mg / Dispensed 500mcg: 1 Tablet
  • Correct Solution: 3 Tablets
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 90%
  • Computer environment errors: 10 (Did Not Attempt (DNA), 0.5 tab, 1.5 tab, 2 tabs, 2.5 tabs)
  • Quasi-Practice environment errors: 4 (DNA, 1 tab, 1.5 tabs)

Question 6

image006

  • Problem: SI Unit Conversion: Prescribed 0.6gram / Dispensed 200mg: 1 Tablet
  • Correct Solution: 3 Tablets
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 96%
  • Computer environment errors: 4 (DNA)
  • Quasi-Practice environment errors: 6 (DNA, 1 tab, 0.5 tab)

Oral Liquid Medicines Section Summary

Table 6: Oral Liquid Medicines Summary Results

 

Congruent

Outcome

Incongruent

Outcome

 

Question

% Computer & practice performance correct

% Computer & practice performance incorrect

% Computer performance correct/ Practice performance incorrect

% Computer performance incorrect/ Practice performance correct

Total

7 (11)

60 (95.2%)

0 (0%)

3 (4.8%)

0 (0%)

63 100%

8 (12)

53 (84.2%)

0 (0%)

5 (7.9%)

5 (7.9%)

63 100%

9 (14)

56 (88.9%)

0 (0%)

5 (7.9%)

2 (3.2%)

63 100%

10 (16)

54 (85.7%)

0 (0%)

8 (12.7%)

1 (1.6%)

63 100%

11 (19)

57 (90.4%)

2 (3.2%)

3 (4.8%)

1 (1.6%)

63 100%

12 (20)

42 (66.7%)

10 (15.9%)

9 (14,2%)

2 (3.2%)

63 100%

Sub-Total

85.2%

3.2%

8.7%

2.9%

100%

Congruence Totals

88.4%

11.6%

100%

 


Table 7: Oral Liquid Medicines Summary Results (Results adjusted for observed minor technical measurement error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement.)

 

Congruent

Outcome

Incongruent

Outcome

 

Question

% Computer & practice performance correct

% Computer & practice performance incorrect

% Computer performance correct/ Practice performance incorrect

% Computer performance incorrect/ Practice performance correct

Total

7 (11)

58 (92.1%)

0 (0%)

5 (7.9%)

0 (0%)

63 100%

8 (12)

53 (84.2%)

0 (0%)

5 (7.9%)

5 (7.9%)

63 100%

9 (14)

53 (84.2%)

0 (0%)

8 (12.7%)

2 (3.2%)

63 100%

10 (16)

52 (82.6%)

0 (0%)

10 (15.8%)

1 (1.6%)

63 100%

11 (19)

56 (88.8%)

2 (3.2%)

4 (6.4%)

1 (1.6%)

63 100%

12 (20)

41 (65.1%)

10 (15.9%)

10 (15.8%)

2 (3.2%)

63 100%

Sub-Total

82.8%

3.2%

11.1%

2.9%

100%

Congruence Totals

86.0%

14.0%

100%

 

Question 7

image007

  • Problem – unit dose: Prescribed 5mg / Dispensed 5mg: 5mL
  • Correct Solution = 5mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 95%
  • Computer environment errors: 0
  • Quasi-Practice environment errors: 3 (6mL, 2mL, 1mL); two participants failed to displace air bubbles from oral syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Two quasi-practice oral syringe measurement errors observed: 4.8mL & 5.2mL
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 92%

Question 8

image008

  • Problem – unit dose: Prescribed 2.5mg / Dispensed 2.5mg: 5mL
  • Correct Solution = 5mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 84%
  • Computer environment errors: 5 (2.5mL, 4mL)
  • Quasi-Practice environment errors: 5 (1mL, 2.5mL, 4mL); two participants failed to displace air bubbles from oral syringes


Question 9

image009

  • Problem – multiple unit dose: Prescribed 160mg / Dispensed 40mg: 5mL
  • Correct Solution = 20mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 89%
  • Computer environment errors: 2 (15mL)
  • Quasi-Practice environment errors: 5 (6mL, 15mL, 25mL); one participant failed to displace air bubbles from an oral syringe

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Three quasi-practice measurement pot measurement errors observed: 22mL, 22mL & 22mL
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 84%


Question 10

image010

  • Problem – multiple unit dose: Prescribed 300mg / Dispensed 50mg: 5mL
  • Correct Solution = 30mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 86%
  • Computer environment errors: 1 (22mL)
  • Quasi-Practice environment errors: 8 (15mL, 22mL, 27mL, 33mL, 34mL, 35mL)

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Two quasi-practice measurement pot measurement errors observed: 31mL & 32mL
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 83%


Question 11

image011

  • Problem – SI Unit Conversion: Prescribed 0.5gram / Dispensed 5mg: 5mL
  • Correct Solution = 5mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 94%
  • Computer environment errors: 3 (DNA, 0.5mL, 1mL)
  • Quasi-Practice environment errors: 5 (DNA, 3mL, 10mL, 12mL, 18mL); four participants failed to displace air bubbles from oral syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • One quasi-practice oral syringe measurement error observed: syringe plunger drawn to 5mL volume graduation point with inclusion of 2.5mL air bolus
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 92%


Question 12

image012

  • Problem – SI Unit Conversion: Prescribed 1 gram / Dispensed 200mg: 5mL
  • Correct Solution = 25mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 83%
  • Computer environment errors: 12 (DNA, 2.5mL, 9.5mL, 20mL, 22mL)
  • Quasi-Practice environment errors: 19 (2.5mL, 3mL, 5mL, 15mL, 20mL, 28mL, 29mL, 30mL, 60mL)

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • One quasi-practice measurement pot measurement error observed: 23mL
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 81%

Injections Section Summary

Table 8: Injections Summary Results

 

Congruent

Outcome

Incongruent

Outcome

 

Question

% Computer & practice performance correct

% Computer & practice performance incorrect

% Computer performance correct/ Practice performance incorrect

% Computer performance incorrect/ Practice performance correct

Total

13 (22)

58 (92.1%)

0 (0%)

5 (7.9%)

0 (0%)

63 100%

14 (23)

62 (98.4%)

0 (0%)

1 (1.6%)

0 (0%)

63 100%

15 (24)

53 (84.1%)

2 (3.2%)

6 (9.5%)

2 (3.2%)

63 100%

16 (27)

27 (42.9%)

18 (28.5%)

8 (12.7%)

10 (15.8%)

63 100%

17 (28)

38 (60.3%)

8 (12.7%)

10 (15.8%)

7 (11.2%)

63 100%

18 (29)

50 (79.3%)

4 (6.4%)

5 (7.9%)

4 (6.4%)

63 100%

Sub-Total

76.2%

8.5%

9.2%

6.1%

100%

Congruence Totals

84.7%

15.3%

100%

 


Table 9: Injection Summary Results (Results adjusted for observed minor technical measurement error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement.)

 

Congruent

Outcome

Incongruent

Outcome

 

Question

% Computer & practice performance correct

% Computer & practice performance incorrect

% Computer performance correct/ Practice performance incorrect

% Computer performance incorrect/ Practice performance correct

Total

13 (22)

54 (85.7%)

0 (0%)

9 (14.3%)

0 (0%)

63 100%

14 (23)

58 (92.1%)

0 (0%)

5 (7.9)

0 (0%)

63 100%

15 (24)

49 (77.8%)

2 (3.2%)

10 (15.8%)

2 (3.2%)

63 100%

16 (27)

25 (39.7%)

18 (28.5%)

10 (15.8%)

10 (15.8%)

63 100%

17 (28)

35 (55.5%)

8 (12.7%)

13 (20.6%)

7 (11.2%)

63 100%

18 (29)

47 (74.6%)

4 (6.4%)

8 (12.6%)

4 (6.4%)

63 100%

Sub-Total

70.9%

8.5%

14.5%

6.1%

100%

Congruence Totals

79.4%

20.6%

100%

 

Participants: n = 63

Review of results


Question 13

image013

  • Problem – unit dose: Prescribed 5mg / Dispensed 5mg: 1mL
  • Correct Solution = 1mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 92%
  • Computer environment errors: 0
  • Quasi-Practice environment errors: 5 (0.6mL, 0.8mL, 1.2mL, 5mL, 10mL); seven participants failed to displace air bubbles from injection syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Four quasi-practice syringe measurement errors observed: 3 x 0.9mL (measured in a 5mL syringe); syringe plunger drawn to 1mL volume graduation point with inclusion of 0.98mL air bolus
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 86%


Question 14

image014

  • Problem – unit dose: Prescribed 60mg / Dispensed 60mg: 1mL
  • Correct Solution = 1mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 98%
  • Computer environment errors: 0
  • Quasi-Practice environment errors: 1 (10mL); seven participants failed to displace air bubbles from injection syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Four quasi-practice syringe measurement errors observed: 0.92mL, 0.94mL; syringe plunger drawn to 1mL volume graduation point with inclusion of 0.1mL air bolus & 1mL air bolus
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 92%


Question 15

image015

  • Problem – multiple unit dose: Prescribed 75mg / Dispensed 50mg: 1mL
  • Correct Solution = 1.5mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 87%
  • Computer environment errors: 4 (0.75mL, 1.25mL)
  • Quasi-Practice environment errors: 8 (DNA, 0.75mL, 1mL, 1.3mL, 1.4mL, 1.7mL); eleven participants failed to displace air bubbles from injection syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Four quasi-practice syringe measurement errors observed: 3 x 1.4mL; syringe plunger drawn to 1mL volume graduation point with inclusion of 1mL air bolus
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 81%


Question 16

image016

  • Problem – sub unit dose: Prescribed 1mg / Dispensed 2.5mg: 1mL
  • Correct Solution = 0.4mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 71%
  • Computer environment errors: 28 (DNA, 0.1mL, 0.15mL, 0.2mL, 0.25mL, 0.3mL, 0.45mL, 2.5mL, 4mL, 5mL)
  • Quasi-Practice environment errors: 26 (DNA, 0.25mL, 0.3mL, 0.5mL, 0.6mL, 0.7mL, 0.8mL, 1mL, 1.1mL, 1.25mL, 1.5mL, 2.5mL, 3.8mL); seven participants failed to displace air bubbles from injection syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Two quasi-practice syringe measurement errors observed: syringe plunger drawn to 0.4mL volume graduation point with inclusion of 0.08mL air bolus & 0.4mL air bolus
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 68%



Question 17

image017

  • Problem – sub unit dose: Prescribed 2mg / Dispensed 5mg: 1mL
  • Correct Solution = 0.4mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 73%
  • Computer environment errors: 15 (DNA, 0.075mL, 0.2mL, 0.25mL, 0.3mL, 0.45mL, 1mL, 2mL, 2.5mL, 4mL)
  • Quasi-Practice environment errors: 18 (DNA, 0.1mL, 0.2mL, 0.25mL, 0.3mL, 0.33mL, 0.5mL, 0.6mL, 0.75mL, 1.9mL, 2mL, 2.4mL, 2.5mL); six participants failed to displace air bubbles from injection syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Three quasi-practice syringe measurement errors observed: syringe plunger drawn to 0.4mL volume graduation point with inclusion of 0.07mL air bolus, 0.2mL air bolus & 0.4mL air bolus
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 68%

Question 18

image018

  • Problem– SI Units Conversion: Prescribed 0.25gram / Dispensed 500mg: 10mL
  • Correct Solution = 5mL
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 86%
  • Computer environment errors: 8 (DNA, 0.5mL, 2.5mL, 3mL, 10mL, 20mL)
  • Quasi-Practice environment errors: 9 (DNA, 0.15mL, 2.4mL, 2.5mL, 4.2mL, 4.5mL, 8); six participants failed to displace air bubbles from injection syringes

Technical measurement error adjusted results (observed minor error variation from the correct numerical value; or correct numerical value transferred to the measurement vehicle with inclusion of an air bolus of >10% volume displacement):

  • Three quasi-practice syringe measurement errors observed: 4.6mL; syringe plunger drawn to 5mL volume graduation point with inclusion of 0.5mL air bolus and 5mL air bolus
  • Corrected congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 81%


Intravenous Infusion Section Summary

Table 10: Intravenous Infusions Summary Results

 

Congruent

Outcome

Incongruent

Outcome

 

Question

% Computer & practice performance correct

% Computer & practice performance incorrect

% Computer performance correct/ Practice performance incorrect

% Computer performance incorrect/ Practice performance correct

Total

19 (31) (mL/Hour)

40 (63.5%)

10 (15.8%)

2 (3.2%)

11 (17.5%)

63 100%

20 (31)

(DPM)

27 (42.8%)

21 (33.3%)

8 (12.7%)

7 (11.2%)

63 100%

21 (32)

(mL/Hour)

42 (66.6%)

7 (11.2%)

2 (3.2%)

12 (19.0%)

63 100%

22 (32)

(DPM)

28 (44.5%)

21 (33.3%)

4 (6.4%)

10 (15.8%)

63 100%

23 (33)

(mL/Hour)

49 (77.8%)

6 (9.5%)

3 (4.8%)

5 (7.9%)

63 100%

24 (33)

(DPM)

25 (39.7%)

18 (28.5%)

11 (17.5%)

9 (14.3%)

63 100%

25 (34)

(mL/Hour)

41 (65.1%)

8 (12.7%)

4 (6.4%)

10 (15.8%)

63 100%

26 (34)

(DPM)

28 (44.5%)

18 (28.5%)

6 (9.5%)

11 (17.5%)

63 100%

27 (40)

(mL/Hour)

40 (63.5%)

7 (11.2%)

2 (3.2%)

14 (22.1%)

63 100%

28 (40)

(DPM)

25 (39.7%)

22 (34.9%)

5 (7.9%)

11 (17.5%)

63 100%

Sub-Total

54.8%

21.9%

7.5%

15.8%

100%

Congruence Totals

76.7%

23.3%

100%

 

Participants: n = 63

Review of results:


Question 19

image019

  • Problem: Prescribed volume: 500mL / Infusion duration: 6 hours
  • Correct Solution = 83.33 (83) mL per hour
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 79%
  • Computer environment errors: 21 (DNA, 0mL, 12mL, 20mL, 82mL, 85mL, 87mL, 97.4mL, 123mL, 125mL, 160mL)
  • Quasi-Practice environment errors: 12 (DNA, 16mL, 17mL, 75mL, 76mL, 87mL, 97.4mL, 101.4mL, 123mL, 500mL


Question 20

image020

  • Problem: IV administration set delivers 20 drops per millilitre
  • Correct Solution = 27.77 (28) drops per minute (DPM)
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 76%
  • Computer environment errors: 28 (DNA, 0 DPM, 3 DPM, 4 DPM, 5 DPM, 10 DPM, 14 DPM, 16 DPM, 20 DPM, 22 DPM, 24 DPM, 25 DPM, 29 DPM, 30 DPM, 32 DPM, 33 DPM, 36 DPM, 38 DPM, 60 DPM)
  • Quasi-Practice environment errors: 29 (DNA, 4 DPM, 5 DPM, 6 DPM, 9 DPM, 10 DPM, 12 DPM, 14 DPM, 15 DPM, 17 DPM, 20.18 DPM, 21 DPM, 22 DPM, 25 DPM, 29 DPM, 30 DPM, 31 DPM, 34 DPM, 38 DPM, 50 DPM, 51 DPM)


Question 21

image021

  • Problem: Prescribed volume: 1000mL / Infusion duration: 6 hours
  • Correct Solution = 166.66 (167) mL per hour
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 78%
  • Computer environment errors: 19 (DNA, 8mL, 12mL, 15.5mL, 60mL, 83.2mL, 140mL, 150mL, 168mL, 177.7mL, 183.3mL, 1000mL)
  • Quasi-Practice environment errors: 9 (DNA, 15mL, 165mL, 177.4mL, 267mL, 1000mL)



Question 22

image022

  • Problem: IV administration set delivers 12 drops per millilitre
  • Correct Solution = 33.33 (33) drops per minute (DPM)
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 78%
  • Computer environment errors: 31 (DNA, 0 DPM, 5 DPM, 8 DPM, 9 DPM, 10 DPM, 12 DPM, 16 DPM, 19 DPM, 28 DPM, 29 DPM, 31 DPM, 36 DPM, 48 DPM, 53 DPM, 60 DPM, 72 DPM, 83 DPM)
  • Quasi-Practice environment errors: 25 (DNA, 13 DPM, 15 DPM, 16 DPM, 20 DPM, 24 DPM, 26 DPM, 31 DPM, 32 DPM, 35.4 DPM, 36 DPM, 42 DPM, 43 DPM, 44 DPM, 55 DPM)



Question 23

image023

  • Problem: Prescribed volume: 500mL / Infusion duration: 4 hours
  • Correct Solution = 125 mL per hour
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 87%
  • Computer environment errors: 11 (DNA, 12mL, 48mL, 25mL, 120mL)
  • Quasi-Practice environment errors: 9 (DNA, 83mL, 120mL, 122mL, 185mL, 500mL)



Question 24

image024

  • Problem: IV administration set delivers 12 drops per millilitre
  • Correct Solution = 25 drops per minute (DPM)
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 68%
  • Computer environment errors: 27 (DNA, 0 DPM, 2 DPM, 5 DPM, 7 DPM, 9 DPM, 10 DPM, 12 DPM, 15 DPM, 21 DPM, 24 DPM, 26 DPM, 30 DPM, 55 DPM, 62 DPM)
  • Quasi-Practice environment errors: 29 (DNA, 6 DPM, 10 DPM, 12 DPM, 15 DPM, 17 DPM, 20 DPM, 20.12 DPM, 24 DPM, 29 DPM, 31 DPM, 36 DPM, 37 DPM, 38 DPM, 50 DPM)



Question 25

image025

  • Problem: Prescribed volume: 250mL / Infusion duration: 8 hours
  • Correct Solution = 31.25 (31) mL per hour
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 78%
  • Computer environment errors: 18 (DNA, 0mL, 30mL, 32.2mL, 36mL, 41mL, 60mL, 81.2mL, 125mL, 360mL)
  • Quasi-Practice environment errors: 12 (DNA, 33mL, 41mL, 41.66mL, 42mL, 62.5mL, 63mL, 250mL)


Question 26

image026

  • Problem: IV administration set delivers 60 drops per millilitre
  • Correct Solution = 31.25 (31) drops per minute (DPM)
  • Congruence & equivalence of student medication dosage problem solving performance between computer & practice quasi-environments: 73%
  • Computer environment errors: 29 (DNA, 0 DPM, 1 DPM, 12 DPM, 19 DPM, 25 DPM, 27 DPM, 30 DPM, 34 DPM, 36 DPM, 41 DPM, 48 DPM, 60 DPM, 69 DPM, 81 DPM)
  • Quasi-Practice environment errors: 24 (DNA, 1.7 DPM, 3 DPM, 4 DPM, 6 DPM, 17 DPM, 19 DPM, 20 DPM, 23 DPM, 25 DPM, 60 DPM, 63 DPM)



Question 27

image027

  • Problem: Prescribed volume: 1000mL / Infusion duration: 6 hours
  • Correct Solution = 166.66 (167) mL per hour
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 75%
  • Computer environment errors: 21 (DNA, 6mL, 20mL, 125mL, 140mL, 150mL, 155.5mL, 158mLmL, 168mL, 177.7mL,183.3mL)
  • Quasi-Practice environment errors: 9 (DNA, 83mL, 165mL, 177.4mL, 1000mL)



Question 28

image028

  • Problem: IV administration set delivers 20 drops per millilitre
  • Correct Solution = 55.55 (56) drops per minute (DPM)
  • Congruence & equivalence of student medication dosage problem solving performance between computer & quasi-practice environments: 75%
  • Computer environment errors: 33 (DNA, 0 DPM, 2 DPM, 3 DPM, 8 DPM, 10 DPM, 18 DPM, 20 DPM, 27 DPM, 28 DPM, 31 DPM, 33 DPM, 36 DPM, 50 DPM, 52 DPM, 53 DPM, 54 DPM, 57 DPM, 65 DPM)
  • Quasi-Practice environment errors: 27 (DNA, 3 DPM, 8 DPM, 17 DPM, 18 DPM, 24 DPM, 33 DPM, 44 DPM, 50 DPM, 54 DPM, 59 DPM, 60 DPM, 66 DPM)




Discussion



In seeking to propose an evidence-based benchmark assessment for medication-related calculation in nursing, this project set out to evaluate the empirical evidence regarding the internal consistency reliability of both a computer-based assessment and a practice simulation assessment of medicine calculation competence. Further, it sought to determine the criterion-related validity of the computer-based assessment and the practice-based assessment to determine whether the computer-based assessment might act as a ‘proxy’ for the practice-based assessment. Finally, the study sought to determine whether, alone or in conjunction, these assessment processes could meet;

  • the requirements of the professional regulator, employers and public in providing a reliable and valid measure of medicine-related calculation competence at point of registration
  • the needs of HEIs in providing an educationally robust, fair and easily administered assessment
  • the needs of students and practitioners in supporting accurate assessment of competence in the context within which it will be required.

A key focus of this study was to determine the validity of the computer simulation format of medication dosage calculations by testing it against the gold standard of a practical simulation format. The underlying rationale was that while a practical simulation format is a way of testing calculation skills across the full range of prescription types and complexity which would not be guaranteed in a practice setting, it is not feasible for mass testing. The computer simulation enables assessment of large numbers of people at the same time, in remote locations and with limited costs. If computer simulated assessment of medications calculations could be shown to operate similarly (provide similar results) to practical simulation testing, this would validate the use of computer simulated testing as a relatively inexpensive, less time-consuming assessment method.

From the results presented above, the criterion-related validity of the computer simulation has been supported, both in terms of putting participants in a similar order of competence and in terms of participants obtaining similar absolute results (getting the same number of questions correct on the computer as they would in the practical situation). Different sections of the assessment are discussed below.

Tablets and capsules
The analysis of results in this section indicated that there were no significant differences between performance in the authentic computer environment and performance in the practice environment.

Very few errors were made overall and none at all in the unit dose calculations. The two errors made in calculations involving multiple unit doses were made in the computer model, but not in practice. One of these involved the choice of half a tablet which was possible in the computer model but not in the practical situation where the placebo tablets were unbreakable. The pilot study had identified this discrepancy between the two models of assessment but the reference group decided to leave the placebos as unbreakable since good practice guidelines are against breaking tablets.

Students found more difficulty in calculating between units where milligrams or grams were prescribed but the tablets were labeled in micrograms and milligrams respectively. There were a number of different wrong answers or failure to attempt these particular calculations in both the computer model and the practical situation, suggesting that students had difficulty with the calculation regardless of the method of presenting the problem.

Oral liquid medicine
The results from this section were analysed in two ways as reported under Results to make allowance for minor measurement errors in the practice environment. Nevertheless, we found a higher rate of discrepancy between computer results and those from the practical exercise in this section than in the tablets and capsules section. The unit dose calculations for liquid medicine were correctly done by most students on the computer but we observed a number of technical measurement errors in the practice environment. Several of these involved failure to dispel large air bubbles from the oral syringe used to measure the dose, resulting in a slightly smaller volume of medicine being available for administration than was suggested by the position of the plunger. In the vast majority of cases, this would not have been clinically significant. Other measured doses were incorrect by factors between 0.5mL and 22mL in the computer assessment and between 0.2 mL and 35mL in the practical assessment. Poor choice of delivery vehicle, for example a 30mL medicine pot used for a dose of 5mL or indeed for 30mL made some amounts difficult for the adjudicator to measure accurately, but this did not seem to occur to the students involved.

Multiple unit dose prescriptions were calculated correctly by most when using the computer, but again there were a number of technical measurement errors made in the practice environment, air in oral syringes being the most common cause. The students, even though they were either at the end of second year or in their third year of training, did not appear to have experience in drawing up liquids using syringes and some had very poor judgment of a suitable receptacle in which to measure the calculated amounts. Few were observed to be holding the medicine pot at eye-level to allow for a meniscus.

That fewer errors in the oral liquid medicine section were made on the computer model may be due to a number of factors. First, it was not possible to simulate the introduction of air into the syringe in the computer model. This has since been recognised as an item for design refinement for the future. Second, because the computer screen was at eye-level with clearly marked levels on the simulated medicine pot, filling it to the correct level was relatively easy. In the practice environment, some of the markings on the pots were quite difficult to see and although the pot could be placed at eye-level by putting it on top of the medicine cabinet, few students did this. Consequently, they did not allow for the meniscus when filling the pot to the calculated mark and resulting amounts were slightly inaccurate because of this.

Conversion between SI units again caused difficulty for a number of students and more made errors on the computer in this section than in the unit and multiple unit dosage prescriptions. Similar errors were made in practice however, and congruence between the two assessment methods was relatively high suggesting that the calculations themselves were the main source of error.

Injections

Unit dose calculations were performed on the computer without error. The practical task resulted in a number of technical errors involving air in the syringe and also errors which suggested miscalculation of the amount required. One of these was an error by a factor of 5 and another by a factor of 10.

Multiple-unit dose errors were made in both assessment methods but again technical measurement errors in the practical use of syringes were common. The actual amount of fluid in a few syringes, particularly those of 1mL capacity, was negligible, suggesting that the student(s) in question concentrated on getting the plunger to the correct mark while failing to see that the needle tip was above the level of the fluid in the ampoule. Hence, a correctly calculated numerical value, although appropriately transferred to the correct measurement graduation point on the syringe barrel would in these cases manifest itself as a considerable sub-dose in reality.

The sub-unit dose prescriptions in this section produced the lowest congruence between the two assessment methods, with a high number of wide-ranging errors demonstrated in both environments. This would suggest that the calculation was sufficiently difficult or poorly understood that different calculation errors were made each time. The commonly manifested technical measurement error of air not being displaced from the syringe accounted only for a small number of errors, leaving miscalculation to be the probable cause of the remaining errors.

The slightly lower level of congruence between the two methods of assessment for prescriptions relating to injections is difficult to explain. Lack of dexterity and practice using actual syringes and needles may account for many of the technical measurement errors observed but would not explain all the differences. Limited exposure to calculations for injections involving anything more complicated than a unit dose is a possible explanation which was borne out by students’ informal comments.

Intravenous fluid delivery rates

Participants were required to work out two rates of delivery for each of the fluid prescriptions. Calculating millilitres per hour tested their performance in setting the rate on a standard volumetric pump, while calculating drops per minute tested whether they were able to calculate the rate through a giving set without a pump attached.

The overall results showed fewer correctly calculated rates than the medicines sections, but there was a similar level of congruence between the two methods and a much higher congruency between incorrect answers suggesting that participants had more difficulty with the calculations themselves than the medium of the assessment.

Many students had not been exposed to calculating fluid delivery rates whether by volumetric pump or by IV fluid administration sets. They reported that they were not aware of any formula to help them to calculate the rate and subsequently performed more poorly in this section.

As there was evidence that many students had not been given the opportunity in the clinical area to apply the calculation of fluid administration rates to a practical situation, simulation, whether by computer or in the practice set-up, gave that opportunity and may be recommended as a practice tool.

We detected mathematical errors in drip rate calculations which involved decimal places. While some volumetric pumps permit the operator to enter decimal fractions, we asked the students to use a rounding technique to the nearest whole number. In spite of this instruction, drip rates were often calculated to 2 decimal places and left as this in the final answer. Participants did not recognise that 66.66 would round up to 67 nor that the answer needed to be in whole numbers when this related to drops per minute. The computer model marked such answers as incorrect. If assessed in practice, it could be argued that this type of error would have no clinical significance.
It was apparent that students had had little exposure to this hands-on practice in setting IV fluid rates.


General analysis of error classifications identified within the authentic computer environment and the practice environment

Figure 13 illustrates a technical measurement error analysis model that defines the error classifications observed in the authentic computer environment and the practice environment:

fig13

Figure 13: Technical measurement error analysis model

This error analysis model illustrates that:

  1. Of the four error classifications two were diagnosed in both the authentic computer program and the practice environment (inappropriate selection of a measurement vehicle; and incorrect measurement or setting of a medication dose or rate), and two were only measurable in the practice environment (inappropriate inclusion of minor air bubbles and air boluses in syringes).
  2. If the student consistently provides an accurate technical measurement solution within a given domain of practice and at a given level of a hierarchical rubric then it can be assumed that they have also demonstrated conceptual and calculation competence within that specific domain (tablet and capsule, oral liquid medicine, injection or IV Infusion) and rubric level.
  3. If the student provides an inaccurate technical measurement solution within a given domain of practice and at a given level of a hierarchical rubric then they have failed to demonstrate competence. However, importantly, the error may be conceptual, calculation, technical measurement or any combination of the latter in nature. Subsequently if the individual’s knowledge and skills within this domain are to be advanced then it is necessary at this point to undertake a diagnostic assessment of the underpinning and discrete inappropriate conceptual, calculation and technical measurement competency schemata that the student/practitioner manifests.



Conclusions



High congruence between results from the two methods of assessment in the tablets and capsules section suggests that for determination of calculation competence in management of this type of prescription, an authentic computer assessment is equivalent to an assessment through practice simulation.

In assessing calculation of liquid medicine doses, as illustrated in Figure 13, both the authentic computer environment and the practice environment assessments facilitated detection of technical measurement errors associated with selection of inappropriate measurement vehicles and measurement/ setting of incorrect liquid doses or IV infusion rates. However, the definition of numeracy which we have used includes competence in knowing what the transfer of a calculated answer to a technical measurement vehicle means in practice and the computer model did not allow for technical measurement errors associated with failing to displace air bubbles and air boluses from syringes, error types which were manifested by several students in the practice environment. This was a measure of numeracy competency which has not been widely considered in the literature and was apparent in all the sections of the assessment which involved liquids. Some students made arithmetic or computational errors but the majority of errors were made in practical technical measurement. These errors were apparent in practice regardless of the accuracy of the original calculation of dose. This element was appropriately identified in the practice simulation assessment and, if coupled with the authentic computer assessment, this competence could be assessed as a practical skill with
any prescription requiring liquid medicine without recourse to repeated measures across the range of complexity. The same argument would apply to prescriptions for injections.

We conclude that for calculation of medicines dosage, the major advantage of the authentic computer environment was to provide prescriptions covering the full range of calculations likely to be met in practice. At the same time it allowed easy assessment of the mathematical element of these calculations with large numbers of students in a short time. In addition, given that marking and feedback generation was entirely automated, the process was quick, easy and totally objective. In assessing nurses’ calculation of medicine doses and IV fluid rates, an authentic computer model that presents dosage problems within an agreed rubric is invaluable in providing assessment of the full range of calculations likely to be met in practice as a newly qualified nurse.

In the practice context it would be impossible to ensure that all third year student nurses encountered and were reliably assessed upon the full range of problems supported by an authentic assessment process. However, the assessment of the numeracy element of a nurse’s competence in medicines management must include assessment of both the full range of calculations likely to be required (supplied through an authentic model – in this case computerised)
and the measurement vehicle manipulation and measurement skills available in most clinical settings and/or able to be simulated in a practical environment.

We propose that the assessment tools and processes identified within this report provide a robust form of assessment that meets the needs of regulators, educators, employers, practitioners, students and public in reliably identifying conceptual, calculation and technical measurement competence in the context of medicines administration.

Further, we propose that this research provides a benchmark against which other researchers and interested stakeholders can measure the impact of other innovations in learning, teaching and assessment strategies, and of recruitment, development and support/retraining strategies.

Summary

The main overall focus of this study was to determine the validity of the computer simulation format of delivering dosage calculation problems. The validity of the computer simulation format was tested against the gold standard practical simulation format. The underlying rationale was that the practical simulation format is not feasible for mass testing, particularly across the full range of question type (tablets, liquid medicines, injections, etc.) and complexity (single unit, multiple unit etc) whereas computer simulation enables testing across the range of question type and complexity of large numbers of people at the same time, in remote locations, with limited costs. If computer simulated testing could be shown to operate similarly (provide similar results) to practical simulation testing, this would validate the use of computer simulated testing for future research.

From the results presented above, the criterion-related validity of the computer simulation format has been supported, both in terms of putting participants in a similar order of competence and in terms of participants obtaining similar absolute results (getting the same number of questions correct on the computer simulation as they would on the practical simulation). These results supplement the more detailed item-by-item comparisons that Weeks has conducted (Weeks 2001), and which produced similar results in terms of confidence in the computer simulated testing format as a substitute method for practical assessment.

Some caveats remain, however:

  • computer simulation does not test certain elements of the real-world dosage calculation problem (e.g., technical competency)
  • these conclusions should only be applied to similar situations, populations, and constructs.


References



  • Black, P., & Wiliam, D. (1998). Inside the Black Box: Raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148.
  • Clochesy, J. M. (2008). Improving medication dosage calculation skills among undergraduate Nursing students. Simulation in Healthcare, 3(4 ).
  • Coben, D. (2008). Numeracy for nursing: The scope for international collaboration. Paper presented at the A Declaration of Numeracy: Empowering Adults through Mathematics Education. Adults Learning Mathematics – A Research Forum (ALM) 15th International Conference, Chestnut Hill College, Philadelphia, PA.
  • Coben, D., Hall, C., Hutton, B. M., Rowe, D., Sabin, M., Weeks, K., et al. (in prep.). Numeracy for nursing - Creating a benchmark. In D. Coben, J. O'Donoghue, G. Knijnik, M. Villavicencio & M. d. C. Fonseca (Eds.), Adult Mathematics Education: Selected papers presented at the ICME11 Topic Study Group 8, Monterrey, Mexico, 6-13 July 2008. Limerick: University of Limerick.
  • Coben, D., Hall, C., Hutton, B. M., Rowe, D., Sabin, M., Weeks, K., et al. (2008). Numeracy for nursing: The case for a benchmark. In T. Maguire, N. Colleran, O. Gill & J. O’Donoghue (Eds.), The Changing Face of Adults Mathematics Education: Learning from the past, planning for the future. Proceedings of ALM-14, the 14th International Conference of Adults Learning Mathematics - A Research Forum (ALM), held at the University of Limerick, Ireland, 26-29 June, 2007 (pp. 88-102). Limerick: University of Limerick in association with ALM.
  • Coben, D. (2000). Numeracy, mathematics and adult learning. In I. Gal (Ed.), Adult Numeracy Development: Theory, research, practice (pp. 33-50). Cresskill, NJ: Hampton Press.
  • European Commission for Education & Culture (2008) The European Qualifications Framework for Lifelong Learning. Euoropean Commission for Education & Culture. Brussels. GLOSSARY, from http://www.eucen.org/EQFpro/GeneralDocs/FilesFeb09/GLOSSARY.pdf
  • Gulikers, J. T. J., Bastiaens, T. J., & Kirschner, P. A. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52(3), 67-85.
  • Hall, C., Coben, D., Hutton, B. M., Rowe, D., Sabin, M., Weeks, K., et al. (2008). Nursing by numbers or nursing with numbers?? Paper presented at the Nursing Education on the Move: What about educators’ competences? 7th European Conference of Nurse Educators, 9-10 October, 2008, Plovdiv, Bulgaria.
  • HEA (2008). Innovations in Numeracy Support for Healthcare Science and Practice Workshop, Higher Education Academy (HEA) in association with the Universities of Glamorgan and Nottingham, King’s College London and NHS Education for Scotland (NES), 6th June, 2008, University of Glamorgan Pontypridd.
  • Hodgen, J., & Wiliam, D. (2006). Mathematics Inside the Black Box: Assessment for learning in the mathematics classroom. London: nferNelson.
  • Hutton, B. M., Coben, D., Hall, C., Rowe, D., Sabin, M., Weeks, K., et al. (pub. pending). Numeracy for nursing – creating a benchmark. Report of a pilot study to compare outcomes of two medications dosage assessment methods using simulated reality. Nurse Education Today.
  • Hutton, B. M. (2004). What do we mean by competency in calculation? Paper presented at the ‘A Calculated Risk: Numeracy needs in Healthcare’ conference, Edinburgh. Retrieved 20th March, 2009, from http://www.nes.scot.nhs.uk/multi/numeracy/documents/Meriel.ppt
  • Hutton, B. M. (1998a). Do school qualifications predict competency in nursing calculations? Nurse Education Today, 18(1), 25-31.
  • Hutton, B. M. (1998b). Should nurses carry calculators? In D. Coben & J. O'Donoghue (Eds.), Adults Learning Maths-4: Proceedings of ALM-4, the Fourth International Conference of Adults Learning Maths - A Research Forum held at University of Limerick, Ireland, July 4-6 1997 (pp. 164-172). London: Goldsmiths College, University of London, in association with ALM.
  • Hutton, B. M. (1997). The Acquisition of Competency in Nursing Mathematics. Unpublished PhD, University of Birmingham, Birmingham.
  • Jukes, L., & Gilchrist, M. (2006). Concerns about numeracy skills of nursing students. Nurse Education in Practice, 6(4), 192-198.
  • NES Numeracy Working Group (2006). Identifying and Supporting the Numeracy Needs of Healthcare Staff in Scotland (Consultation). Edinburgh: NES.
  • NHS QIS (2006). Safe Today - Safer Tomorrow. Patient Safety: Review of Incident and Near-Miss Reporting. Full Report. Edinburgh: National Health Service Quality Improvement Scotland.
  • NMC (2007a). Essential Skills Clusters (ESCs) for Pre-registration Nursing Programmes. Annexe 2 to NMC Circular 07/2007. London: Nursing and Midwifery Council.
  • NMC (2007b). Introduction of Essential Skills Clusters for Pre-registration Nursing Programmes Retrieved 8 October, 2007, from http://www.nmc-uk.org/aArticle.aspx?ArticleID=2558#skills
  • NPSA (2009). Safety in Doses: Improving the use of medicines in the NHS. London: National Patient Safety Agency.
  • NPSA (2006a). Safe Foundations: Patient safety education modules for junior doctors, retrieved 30th Jan., 2010 from http://www.npsa.nhs.uk/press/display?contentId=5186
  • NPSA (2006b). Safety in Doses: Medication safety incidents in the NHS. The fourth report from the Patient Safety Observatory (No. PSO/4). London: National Patient Safety Agency.
  • NPSA (2003). A report on the Pilot Data Audit undertaken by the NPSA NPSA Business Plan 2003-04. London: National Patient Safety Agency.
  • OECD (2005). The Definition and Selection of Key Competencies. Executive Summary. Paris: Organisation for Economic Cooperation and Development.
  • Sabin, M., Hall, C., Coben, D., Rowe, D., Weeks, K., Woolley, N. & Hutton, M. (2008a). Establishing an assessment benchmark in nursing: The example of medication dosage calculation. Paper presented at the RCN Partners in Practice 2008, 22nd February, Blackpool.
  • Sabin, M., Hall, C., Hutton, M., Rowe, D., Weeks, K. W., Coben, D., et al. (2008b). To err is to be human: Reducing medication errors and improving safety in health care… Establishing an assessment benchmark in nursing. Paper presented April 28th at the RCN Congress 2008, Bournemouth.
  • Sabin, M., Coben, D., Hutton, B. M., Hall, C., Rowe, D., Weeks, K. W., et al. (2008c). Establishing a numeracy assessment benchmark in nursing: The focus and design of a NES commissioned research study. Paper presented 6th June at the Innovations in Numeracy Support for Healthcare Workshop, University of Glamorgan.
  • Sabin, M. (2006a). NHS Education for Scotland: Supporting Numeracy Skills in the Scottish Healthcare workforce Retrieved 25 August, 2007, from http://www.nes.scot.nhs.uk/multi/documents/Numeracy_NES_Update.pdf
  • Sabin, M. (2006b). Identifying and Supporting the Numeracy Needs of Healthcare Staff in Scotland. A Strategy Document Edinburgh: NHS Education for Scotland.
  • Sabin, M. (2001). Competence in Practice-based Calculation: Issues for nursing education. A critical review of the literature. Occasional Paper 3. London: Learning and Teaching Support Network (LTSN) Centre for Health Sciences and Practice.
  • Smith, J. (2004). Building a Safer NHS for Patients – Improving medication safety. London: Department of Health Publications.
  • Weeks, K. W., & Woolley, N. N. (2007). Innovative and Authentic Assessment of Medication Dosage Calculation Problem Solving Skills, via Engagement and Interaction with an Authentic World Learning Environment. LTO Innovation Grant 2005-06 Report. Pontypridd: University of Glamorgan.
  • Weeks, K. W. (2001). Setting a foundation for the development of medication dosage calculation problem solving skills among novice nursing students. The role of constructivist learning approaches and a computer based ’Authentic World’ learning environment. Unpublished PhD, University of Glamorgan, Pontypridd.
  • Weeks, K. W., Lyne, P., Mosely, L., & Torrance, C. (2001). The strive for clinical effectiveness in medication dosage calculation problem solving skills: The role of constructivist theory in the design of a computer-based ‘authentic world’ learning environment. Clinical Effectiveness in Nursing, 5(1), 18-25.
  • Weeks, K. W., Lyne, P., & Torrance, C. (2000). Written drug dosage errors made by students: The threat to clinical effectiveness and the need for a new approach. Clinical Effectiveness in Nursing, 4(1), 20-29.





Annexe 1: Report of the pilot study


Numeracy for Nursing – creating a benchmark


Report of a Pilot study to compare outcomes of two medications dosage assessment methods using simulated reality


Diana Coben, King’s College London
Carol Hall, University of Nottingham
B. Meriel Hutton, King’s College London
David Rowe, University of Strathclyde
Mike Sabin, NHS Education Scotland/Scottish Government
Keith Weeks & Norman Woolley, Authentic World
®, University of Glamorgan

Abstract

This is the report of a pilot study for the first stage of an inter-disciplinary project to establish a benchmark for numeracy in nursing by exploring the key issues in determining the achievement of competence in nursing numeracy. The project is funded by NHS Education for Scotland (NES).

Objective: To test the efficacy of a computer-based assessment of pre-registration nursing students’ medication calculation skills by comparing outcomes from this with calculations presented in a practical setting. The pilot also acted as a test of the research design, and the participants’ evaluation of content will inform the suitability of items of assessment for inclusion within a benchmark standard.

Design: A quasi-experimental cross-over design in which half the participants were exposed to the computer-based assessment prior to undertaking the practical tasks, while the other half were exposed to the practical tasks before undertaking the computer-based assessment.

Setting: A large school of nursing in England with modern IT facilities to meet the specifications of the computer assessment and a large classroom for the practical activity.

Participants: Fifty volunteer early 3
rd year (2008) students on the adult branch of a pre-registration nursing programme.

Main Outcome Measures:
Numerical scores from the computerised assessment compared with results from the practical exercise.

Results: Results showed a reasonable to very high level of congruence between the two methods (44%-100% congruence; mean congruence for n = 252 test item opportunities for error = 80% congruence).


Conclusions: Computerised assessment using this particular platform is likely to closely mirror medication calculations done in a practical setting.


Introduction

Medication errors are an aspect of clinical governance which has been highlighted recently by the National Patient Safety Agency (NPSA) and this issue is currently targeted for remedial action (NPSA, 2006a, 2006b).While medication errors are multi-factorial, lack of competence in numerical calculation is often cited as a key area of concern, especially with respect to medication dosage calculation (Weeks, Lyne, & Torrance, 2000). Numeracy is recognized as a key competence for professional practice in nursing, yet successive studies indicate a lack of proficiency within both the student population and amongst registered nurses (Hutton, 1997; Sabin, 2001). To inform the study, the following definition of numeracy was adopted:


To be numerate means to be competent, confident, and comfortable with one’s judgements on whether to use mathematics in a particular situation and if so, what mathematics to use, how to do it, what degree of accuracy is appropriate, and what the answer means in relation to the context.

(Coben, 2000, p. 35)


Numeracy in the healthcare context is much broader than medication calculation, but this is its most visible and commonly cited example, and greater consistency in assessment would strengthen support for the enhancement of learning and teaching approaches. From September 2008 the body regulating the profession in the UK, the Nursing and Midwifery Council (NMC) requires students to achieve 100% in a test of “numeracy in practice” (NMC, 2007) before being allowed to register as nurses, yet there are currently no national standards for what that test should include. Higher Education Institutions (HEIs) and employers are accordingly incorporating numeracy into their pre-registration nursing programmes and devising and setting their own tests of numeracy. Whilst these tests may be constructed on the basis of considerable professional expertise and with the best of intentions, as long as there is no recognised benchmark standard, such tests will remain variable both in level and quality; at worst, it is suggested, some may be professionally or mathematically inappropriate. Whilst it is accepted that in some instances, the establishment of benchmark standards may be controversial in terms of achieving minimum or best practice, the above considerations indicate that the development of a benchmark for nursing numeracy calculation is desirable in order to determine and debate which skills should be taught, and to ascertain when competence has been achieved.

The research team has made the case for such a benchmark fully elsewhere through international conference presentation and publication (Coben et al., 2008). The aim in this stage of the study is to address issues of parity and level in assessing competence in numeracy for successful calculation of medication dosages by nurses at the point of qualification. We focussed on medication dosage calculation because it is probably the most widely-recognised and sensitive area of numeracy for nursing. Medication dosage calculation is one aspect (‘Calculation’) of ‘Medicines Management’ in the NMC’s ‘Essential Skills Clusters’ for pre-registration nurse education (NMC, 2007). There was no intention to assess ‘Medicines Management’ as a whole.

We believe that to assess such competence in practice with every student covering the full range of medication dosage calculations would be an extremely difficult task. Not only are students allocated to placements with very different dosage calculation opportunities, but it would be too complex a task to ensure that each student had been tested in a similar way. One solution would be to set up objective structured clinical examinations (OSCEs). However, these are extremely costly in terms of assessor/supervisor time and there is no guarantee that they would cover a full range of calculations.

From an extensive analysis of the literature, the following research-based criteria for an
effective numeracy benchmark assessment tool was developed. Such a tool should be reliable, valid and capable of recreating the complexity of nursing numeracy in an authentic assessment environment. Specifically, it should be:

  • Realistic: Evidence-based literature in the field of nursing numeracy (Hutton,1997; Weeks, 2001) strongly supports a realistic approach to the teaching and learning of calculation skills, which in turn deserve to be tested in an authentic environment. Questions should be derived from authentic settings. A computer based programme of simulated practice in drug calculations, formative testing, with feedback on the nature of errors made, has been shown to develop competency in medication dosage calculation, which also can be demonstrated in the clinical areas (Weeks, Lyne, & Torrance, 2000). Exposure of students to real-world situations is recommended (Weeks, 2001).
  • Appropriate: The assessment tool should determine competence in the key elements of the required competence (OECD, 2005; Sabin, 2001).

  • Differentiated: There should be an element of differentiation between the requirements for each of the branches of nursing (Hutton, 1997).

  • Consistent with adult numeracy principles: The assessment should be consistent with the principles of adult numeracy learning, teaching and assessment, having an enablement focus (Coben, 2000, Hodgen, Coben & Rhodes, 2008).

  • Diagnostic: The assessment tool should provide a diagnostic element, identifying which area of competence has been achieved, and which requires further intervention (Black & Wiliam, 1998). Thus it should “provide information to be used by students and teachers that is used to modify the teaching and learning activities in which they are engaged in order better to meet student needs. In other words, assessment is used formatively to ‘keep learning on track’”. (Wiliam, 2006).

  • Transparent: The assessment should be able to demonstrate a clear relationship between ‘test’ achievement and performance in the practice context (Weeks, Lyne, Mosely, & Torrance, 2001).

  • Well-structured: The assessment tool should:

    • provide a unique set of questions with a consistent level of difficulty;

    • provide a structured range of complexity; and

    • take place within a defined framework, at points by which students can be effectively prepared, while allowing time for supportive remediation. (Hodgen & Wiliam, 2006)

  • Easy to administer: the assessment should provide the opportunity for rapid collation of results, error determination, diagnosis and feedback (Black & Wiliam, 1998).


The pilot study described here focuses on establishing the validity and reliability of the research instruments and the robustness of the research design. Medication dosage calculations covering a full range of complexity within the hierarchy, that is: unit dose, sub and multiple unit dose, complex problems and conversion of units, which had been previously tested for face and content validity by a team of pharmacists, nurse clinicians and nurse educators (Weeks, Lyne, Mosely and Torrance, 2001) were used as assessment items. The original assessment tool was designed by Keith Weeks and Norman Woolley as part of the Authentic World
® program. This program is based on a constructivist-centred design drawn from the work of Piaget (1983), Bruner (1975), and von Glasersfeld (1987).

Aim of the study
The pilot study set out to test whether a computer-based assessment using the authentic diagnostic assessment outlined above mirrored the results of identical calculations presented in a (simulated) clinical environment. A pre-pilot was used to test the design of the practical tasks and the practicalities of the cross-over design of the study.

Methods

Study design
A computer-based 40-item baseline assessment using web-based materials from Authentic World® was undertaken by all students who volunteered for the project. A representative sample of 20 students was then selected from these on the basis of results. Two weeks later, selected students’ performance in a further computer-based assessment of medication calculations was compared with their performance in a simulated practice situation involving the same calculations. A cross-over design took learning effects into account.

Participants
Following ethical approval by the pilot-site University and agreement from the course directors, 88 adult branch students from two cohorts commencing their 3rd year of Diploma/BSc Nursing Studies were approached. The cohorts included traditional 3-year programme students and a small number of graduate entry students who had received a shortened first year programme. An initial hour-long introductory session by two members of the research team outlined the project and explained volunteers’ participation in it. All students were promised confidential feedback on their performance and a period of free access to the full web-based programme of instruction. Students were then invited to sign up as participants. These volunteers were given information sheets and asked to attend the preliminary computer baseline assessment on a given date three weeks later.

Fifty students attended for the baseline assessment. Because of the large number of volunteers, these were divided into two groups and allocated to either a morning or afternoon session. From the original 50, a representative sample of 20 was selected on the basis of their results in the baseline test to continue with the study. Of these, only nine students presented for the second stage.

Tools

a) Computer-based assessment
A computer-based assessment tool was devised from an existing programme of medication calculation instruction for nurses in which the necessary calculation skills are developed and assessed in an authentic context using high fidelity imagery of the elements of the dosage calculation problem (e.g. prescription charts, medication containers, syringes etc.). See Fig.1. This evidence-based programme (Authentic World®) has been refined over a number of years and is already used by several schools of nursing in the UK. It has been tested extensively for validity (Weeks, Lyne, Mosely, & Torrance, 2001). The 40-item baseline assessment drawn from the programme covered the full range of medication dosage calculations which the development team believed likely to be met by a student nurse in practice. (A further phase of the master study is planned to evaluate this assumption.)

pilot_fig1a

Problem Presentation and Delivery Vehicle Selection


pilot_fig1b

Answer Representation in Chosen Delivery Vehicle


pilot_fig1c


Presentation of Feedback on Performance


Fig. 1: Examples of Authentic World© imagery


To accommodate the assessment of all individuals in the study, irrespective of the medication dosage calculation and problem-solving education process they had previously been exposed to, the assessment programme had been stripped of its constructivist-based modelling, coaching, scaffolding and detailed diagnostic feedback processes (although for participating in the research programme, all students were given access to the full programme, including detailed diagnostic feedback). On volunteering to participate in the study, students were issued with a log-in code which allowed them access to the Authentic World web-site and a specially prepared set of practice questions so that they could get used to the technology. The prepared programme covered competence in calculating doses of tablets/capsules, liquid medicines and injections, all at unit-dose, sub and multiple unit-dose, complex problem and conversion of SI unit levels. The final section included intravenous infusion rates for both volumetric pumps and manually controlled giving sets.


The assessment took place in a computer laboratory with one computer per student and two supervisors. After a short practice with the technology, students were asked to complete the 40-item baseline assessment. Two hours were allowed with an optional break between sections.
A representative sample of 20 participants were selected from the results of this baseline assessment and given a date to attend the next stage of the pilot. Of these, nine students presented for the second stage when they undertook a further computer assessment and a practical activity requiring the same calculations. To randomise sequence effects, half the students were allocated to the computer test in the morning and the practical in the afternoon, and vice versa. Each student was asked to use their ID from the original computer assessment on any paper used for working out and to identify their results.

The second computer assessment consisted of a 28-item test. Statistical work undertaken in the preparatory stage of the project confirmed the reliability of the selected Authentic World® test items; the internal consistency reliability of the selected items were R = ∙92. Six items were calculations involving tablets and capsules; six were liquid medicines and six injections. A further five calculations were of intravenous (IV) fluid rates, each requiring a calculation for both volumetric pumps and manually controlled giving sets. Participants were supplied with paper and pencil for any working out they needed to do and they were given two hours in which to complete the assessment. An optional break was allowed during this time. Student participants were also required to evaluate the items of assessment for suitability as a benchmark standard, using an on-line evaluation tool.

b) The practical assessment activity

The practical activity had been designed for 10 students at a time. Five would be seated at tables arranged so that they could work independently but also be observed by a central invigilator in a similar manner to the computer assessment and the upright medicine cabinets would act as screens between the participants. The other five would be seated at tables where they would be presented with a labelled bag of fluid and giving set for each numbered ‘prescription’. As some students had notified withdrawal from the study prior to Stage 2, not all 20 students were expected and so the practical room was set up to accommodate eight students with a minimum of three supervisors. The four medicine cabinet ‘stations’ were arranged with the cabinets upright and open on the corner of each table thus acting as screens between students. (See Fig.2) Paper and pencils were provided at each ‘station’ and if used, collected afterwards.


room_layout

Figure 2: Layout of room (accommodating 8 ‘stations’)


The practical activity required students to do exactly the same medication calculations as in the computer assessment. These were presented in two sections: medicines calculations covering tablets and capsules, liquid medicines and injections; and calculations of IV rates. The medicine section consisted of three colour-coded prescription sheets (formatted similarly to those used in the NHS) to be worked through one at a time. One sheet was of prescriptions for tablets and capsules, one liquid medicines and one injections. Each sheet contained six numbered prescriptions and was presented on a colour-coded dinner tray (labelled with the student’s project ID number). Six small papier-maché trays (also colour-coded and numbered 1–6) were provided per dinner tray/prescription sheet (See Fig. 3). The dinner trays were stacked in random order and students were instructed to tackle them one at a time by selecting the correct medicine from the medicine cabinet, preparing it as for presentation to a patient and putting the medicine pot or syringe relating to each prescription on the appropriately numbered papier-maché tray and placing this onto the dinner tray (see Fig. 4). When they had completed the six items, or as many as they could, the students signalled for the tray to be collected for ‘marking’. They then dealt with the next tray.

fig11

Fig 3 Colour-coded dinner tray - at start


fig12

Fig.4 Completed tray (for medicines and capsules task)

Half the students did the medicine calculations first and the other half did the IV rates first. Students were given a half-hour break between the different tasks. The process was invigilated by three researchers; one dedicated to the IV section: the others covering the medicines section. Students’ results were recorded as they finished each task or shortly afterwards if trays were ‘stacking up’.
The IV fluid calculations had to be demonstrated using bags of fluid and IV giving sets with differing drop factors (delivering different numbers of drops per millilitre) or attached to a volumetric pump. An IV prescription sheet was provided per student with 5 prescriptions for different fluids to be delivered over differing time periods. As only one type of giving set was available, each IV set-up was labelled with a drip factor relating to the calculation required in the on-line test. Students were required to calculate two rates per prescription – one in millilitres per hour for entry into the volumetric pump and the other for drops per minute. When they were ready to set the rate, they signalled to the invigilator who observed them setting the rate in ‘practice’ by adjusting the manual roller on the giving set and in-putting the correct numbers on the volumetric pump. This model had been tested during the pre-pilot materials testing exercise and it was considered impractical for the invigilator to observe and count the drops each time. Students were therefore asked to tell the invigilator how many drops they were setting the rate at and this was recorded on the ‘mark’ sheet. Another problem was many students’ lack of experience in setting rates on different volumetric pumps and so the invigilator was allowed to key in the numbers dictated by the student.
Calculators

Student participants were not allowed to use calculators. This is not because we object in principle to the use of calculators by nurses or nursing students – see Hutton (1998) for a discussion of this issue, but banning the use of calculators would enable us to standardize the testing procedures across all participants.


Data collection and analysis

The computer-based assessment data were automatically entered into a spreadsheet while the practical scores were entered by hand and transcribed for analysis by the psychometrics specialist member of our team. The transcription was done by 2 people to avoid transcription errors. The same answer given on both computer and in the practical was recorded as congruence (whether correct or not). Any deviation was considered incongruent.

Results

Tablets and Capsules

Calculations were correct and concordance between performance in the computer assessment and in the practical activity was 100% for all but one of the calculations involving tablets and capsules (mean congruence = 98%). The incongruent results involved 2 students using a half tablet which was available in the computer model but not in the practical.

Liquid medicines

Congruence for this section varied from 44–89% (mean congruence = 74%). The majority of students gave the correct answer on the computer but mistakes were made in measuring the actual amount in practice.

Injections

Congruence for these items again varied from 56-89% (mean congruence = 73%). Inaccuracies occurred in both the computer and practical assessments, but most errors were made in measuring the amounts in practice.

IV rates

Congruence for these items ranged 56-100% (mean congruence = 77%). Overall student performance was generally poorer with more incorrect answers. Congruence was demonstrated several times between wrong answers in both models which suggested that the student was making the same error in both situations. This had been demonstrated to a limited extent in the other sections.


pilot_table1

Table 1: Comparison of students’ results by section (n=9)


Discussion


From the original analysis of the data, it was apparent that performance in the two tests of medicines calculations was reasonably well correlated overall. However, if minor errors, such as failing to round as instructed, (e.g., for IV infusion rate per unit time calculations entering 166.6 or 166 instead of 167 mL per hour) were excluded and apparent sloppiness in practical measurement were excused, then a greater degree of congruence was reached. See Table 2. The small number of participants precluded any formal statistical testing at this point.

pilot_table2

Table 2: Summary of Pilot Data Results allowing for factors outside the calculation itself. Congruence between Authentic Computer Assessment & Simulated Practice Assessment

Extraneous factors were observed which demonstrated that although the actual calculation of drug dosage or infusion rate was tested adequately by both models, neither model could capture all the issues around safe drug calculation in practice. Given our original definition of numeracy, we felt that these needed expansion.

1. Drawing up the correct amount into a syringe

In the computer model, the syringe filled as the plunger was drawn down but there was no facility for air to be drawn in as well. During the practical activity, several students drew in air with the ‘drug’ and failed to expel it before presenting the completed task, thereby preparing a wrong volume of drug in spite of a ‘correct’ reading of the level of the plunger. In discussion with the computer programme designers, it was decided that although air entry could be built-in to the program in future, it was not possible to change it for the impending main study and so we decided to record both the plunger level and the actual amount of fluid in a syringe where these differed. Feedback to the student would include this error but if the plunger level was at the same level as the computerised plunger for that calculation, we would record the data as concordant. It was decided that the medicines management issues which these errors illustrated needed to be included in the generalised report to the participating Institution.

2. Amount of drug available

In the practical activity we had supplied an appropriate sized ampoule containing placebo drug for each prescription, but by doing this, the students were given an extra cue to make their answer no more than that amount. The computer model allowed such a mistake to be made. For example, on the computer assessment, a student could provide an answer of 4 mL when the answer should have been 0∙4 mL. In the practice situation, they were only presented with one 2 mL ampoule and so could not make that mistake. We decided that for the main study we would provide 10 mL plastic ampoules of every placebo drug. This would allow students to make this error in practice and at the same time remove the danger of injury from the 2 mL glass ampoules.

The non-concordant result for tablets and capsules involved students using the half tablet facility on the computer model. There was a reluctance to provide placebos which could be broken, as this was considered poor practice, but again the computer programme reflected the situation in reality where some tablets are scored and half tablets may on occasion be given. After some discussion, it was decided not to alter this for the main study as the correct answer did not require a half tablet.

Evaluation from the students involved in the pilot was in most cases very positive and agreed that these were the sorts of calculations which they had encountered in practice. Negative comments were associated with the drip rate calculations which some students thought were irrelevant to their practice because they had not yet been given responsibility for intravenous infusions.

Conclusion

The pilot study was invaluable in resolving practical issues such as those discussed above. It also provided early indications that the two methods of assessing medicine dosage calculations produced similar results (mean congruence between student performance within the two environments = 80%). This would suggest that this particular computer assessment of drug calculations should give a good indication of a student’s performance in practice. What it also showed was that while the computer model was able to assess conceptual and calculation skills and to an extent technical measurement skills, it could not assess all of the wider numeracy issues involved with safety in medicines measurement and medicines management (for example dispelling air from a syringe).

The pilot tested a small number of students from one school of nursing and so the evaluative comments, although informative gave a limited view. The main study will be carried out on a larger sample from a number of different schools of nursing and will provide a much larger data set, both from the assessments and from the evaluations.

References

Black, P., & Wiliam, D. (1998). Inside the Black Box: Raising standards through classroom assessment.
Phi Delta Kappan, 80(2), 139-148.

Bruner, J. S. (1975). Toward a Theory of Instruction. Cambridge: Belknap/Harvard.

Coben, D. (2000). Numeracy, mathematics and adult learning. In I. Gal (Ed.), Adult Numeracy Development: Theory, research, practice (pp. 33-50). Cresskill, NJ: Hampton Press.
Coben, D., Hall, C., Hutton, B. M., Rowe, D., Sabin, M., Weeks, K., et al. (2008). Numeracy for nursing: The case for a benchmark. In J. O'Donoghue (Ed.),
The Changing Face of Adults Mathematics Education: Learning from the past, planning for the future. Proceedings of ALM-14, the 14th International Conference of Adults Learning Mathematics - A Research Forum (ALM), held at the University of Limerick, Ireland, 26-29 June, 2007. Limerick: University of Limerick in association with ALM.
Hodgen, J., Coben, D., & Rhodes, V. (2008, in press).
Formative Assessment in Adult Numeracy. London: NRDC.
Hodgen, J., & Wiliam, D. (2006).
Mathematics Inside the Black Box: Assessment for learning in the mathematics classroom. London: nferNelson.
Hutton, B. M. (1997).
The Acquisition of Competency in Nursing Mathematics. Unpublished PhD, University of Birmingham, Birmingham.UK
Hutton, B. M. (1998). Should nurses carry calculators? In D. Coben & J. O'Donoghue (Eds.),
Adults Learning Maths-4: Proceedings of ALM-4, the Fourth International Conference of Adults Learning Maths - A Research Forum held at University of Limerick, Ireland, July 4-6 1997 (pp. 164-172). London: Goldsmiths College, University of London, in association with ALM.
NMC. (2007). Essential Skills Clusters (ESCs) for Pre-registration Nursing Programmes. Annexe 2 to NMC Circular 07/2007. London: Nursing and Midwifery Council.

NPSA. (2006a). Safe Foundations: Patient safety education modules for junior doctors. from http://www.npsa.nhs.uk/press/display?contentId=5186. London: National Patient Safety Agency.
NPSA. (2006b). Safety in Doses: Medication safety incidents in the NHS. The fourth report from the Patient Safety Observatory (No. PSO/4). London: National Patient Safety Agency.

OECD. (2005). The Definition and Selection of Key Competencies. Executive Summary. Paris: Organisation for Economic Cooperation and Development.

Piaget, J. (1983). Piaget's theory. In W. Kessen (Ed.), Handbook of Child Psychology (Vol. 1). New York: Wiley.

Sabin, M. (2001). Competence in practice-based calculation: Issues for nursing education. A critical review of the literature. Occasional Paper 3. London: Learning and Teaching Support Network (LTSN) Centre for Health Sciences and Practice.

von Glasersfeld, E. (1987). The Construction of Knowledge: Contributions to Conceptual Semantics. New York: Intersystems Publications.

Weeks, K. W. (2001). Setting a foundation for the development of medication dosage calculation problem solving skills among novice nursing students. The role of constructivist learning approaches and a computer based ’Authentic World’ learning environment. Unpublished PhD, University of Glamorgan, Pontypridd.

Weeks, Lyne, & Torrance, (2000) Written drug dosage errors made by students: The threat to clinical effectiveness and the need for a new approach. Clinical Effectiveness in Nursing, 4, 20-29.

Weeks, K. W., Lyne, P., Mosely, L., & Torrance, C. (2001). The strive for clinical effectiveness in medication dosage calculation problem solving skills: The role of constructivist theory in the design of a computer-based ‘authentic world’ learning environment. Clinical Effectiveness in Nursing, 5(1), 18-25.
Wiliam, D. (2006). Does Assessment Hinder Learning? [Electronic Version] from
http://www.uk.etseurope.org/home-corpo-uk/news- home/?print=1&news=136&view=detail&no_cache=1

© 2010 NES Project Team Contact Us