“Our skills are perishable; if we don’t use them, we lose them.” (Pennington, 2013)
As you will recall, JIBC evaluators conduct criterion-referenced evaluation, which involves measuring student performance against set NFPA® standards. As an evaluator, you are not assessing student performance according to how you feel a given skill should be performed, or how well a student performs relative to others in a class or department. The emphasis needs to be on evaluating learning as objectively as possible.
In order to support test validity and reliability, FSD practical evaluations are conducted with the assistance of practical skills sheets. These are designed to help you measure student proficiency in performing a specific job or evolution. Each skill sheet (i.e., skills scenario) is developed by a validation team of Subject Matter Experts (SMEs) and a trial is conducted prior to skill sheet launch in order to uncover any challenges and increase test validity.
According to Clausing and Snyder (2012), validity is the “extent to which a test or other assessment technique measures the learner qualities (knowledge or skills) that it is meant to measure” (p. 483).
Reliability, on the other hand, is “a condition of validity; the extent to which a test or test item consistently and accurately produces the same results or scores when given to a set of learners on different occasions, marked by different assessors, or marked by the same assessors on different occasions” (Clausing & Snyder, 2012, p. 483).
Currently, the FSD’s approach to practical skill sheets vary somewhat based on the program area and delivery method and can be categorized into critical and noncritical components. Within each skill sheet, specific skills are designated as critical and must be achieved with a mastery level of 1 (100%). If a candidate does not succeed on their first attempt at a critical skill, they are granted a second attempt. On the second attempt of a critical skill, the highest achievable grade is 0.76 (76%) provided the candidate successfully masters the skill. Failure on the second attempt results in an unacceptable completion status for the entire skills sheet.
Detailed grading criteria are outlined on relevant skill sheets, with grades ranging from 1 (100%) for exemplary performance, 0.76 (76%) for proficient performance, 0.50 (50%) for adequate performance with minor errors, to 0.25 (25%) for poor level performance.(Sample Yes/No Skill Sheet).
The FSD also currently utilizes a couple of different approaches to skill sheet completion. Specifically, in some program areas, skill sheets are completed electronically via a system called Questionmark. In many other areas, however, skill sheet completion is paper-based. Since the process for submitting completed practical skill sheets varies somewhat, the JIBC Manager or Coordinator responsible will be able advise you as to the correct process.