“Our skills are perishable; if we don’t use them, we lose them.” (Pennington, 2013)
As you will recall, JIBC evaluators conduct criterion-referenced evaluation, which involves measuring student performance against set NFPA® standards. As an evaluator, you are not assessing student performance according to how you feel a given skill should be performed, or how well a student performs relative to others in a class or department. The emphasis needs to be on evaluating learning as objectively as possible.
In order to support test validity and reliability, FSD practical evaluations are conducted with the assistance of practical skills sheets. These are designed to help you measure student proficiency in performing a specific job or evolution. Each skill sheet (i.e., skills scenario) is developed by a validation team of Subject Matter Experts (SMEs) and a trial is conducted prior to skill sheet launch in order to uncover any challenges and increase test validity.
According to Clausing and Snyder (2012), validity is the “extent to which a test or other assessment technique measures the learner qualities (knowledge or skills) that it is meant to measure” (p. 483).
Reliability, on the other hand, is “a condition of validity; the extent to which a test or test item consistently and accurately produces the same results or scores when given to a set of learners on different occasions, marked by different assessors, or marked by the same assessors on different occasions” (Clausing & Snyder, 2012, p. 483).
Currently, the FSD’s approach to skill sheet evaluation varies somewhat based on the program area and delivery method. In some cases, students’ competency is deemed either successful or unsuccessful, or pass/fail. Performance is measured by completion of an assignment or lack of completion as described in the individual skill sheet/performance checklist (Sample Yes/No Skill Sheet).
In other program areas, individual practical skills have been weighted as being Critical, Major or General, and the passing score is set at 75%. If a candidate is unsuccessful in completing a Critical skill, it will result in a deduction of 26%, which by default means that the candidate has failed the evaluation. Deductions for failure to satisfactorily complete a Major skill will result in a deduction of 13%, while an error in completing a General Skill will result in a 6% deduction (Sample Weighted Skill Sheet).
The FSD also currently utilizes a couple of different approaches to skill sheet completion. Specifically, in some program areas, skill sheets are completed electronically via a system called Questionmark. In many other areas, however, skill sheet completion is paper-based. Since the process for submitting completed practical skill sheets varies somewhat, the JIBC Manager or Coordinator responsible will be able advise you as to the correct process.