Test Validity & Reliability

In addition to the large number of practical evaluations conducted annually by the FSD, many of our courses also require assessment of students’ cognitive understanding. To that end, the FSD maintains test question banks for many levels of accreditation. Each question within these banks has been carefully assessed for validity and reliability by FSD staff and a team of local Subject Matter Experts (SMEs). Questions are also reviewed to ensure that the content is relevant to our jurisdiction and reflects Canadian terminology and best practices.

In keeping with accreditation criteria, the FSD ensures that each test bank is of sufficient size to allow several different test instruments (i.e., exams) to be created. At a minimum, there are at least two questions correlated to each Job Performance Requirement (JPR) and requisite knowledge area of the given NFPA® standard. When it is time to generate a student exam from the relevant Questionmark database, this is typically done through a random selection process. Test items are randomly selected from clearly identified subgroups of correlated questions. Selection ensures complete coverage of the applicable standard while weighting the importance of certain topics over others, which places evaluative focus where it is appropriately due.

Perhaps one of the most important things for you to remember as a JIBC instructor or evaluator is the reality that we teach—and evaluate—to a set standard. For courses leading to certification, the respective NFPA® standard is used as a guiding document in the development of our offerings, our course outlines and intended learning outcomes. It is to the standard and to our articulated learning objectives that we must teach and evaluate. We should not be teaching to an exam.

Test scores and test items are reviewed on a regular basis. In order to analyze test questions, FSD staff may generate an analysis sheet (i.e., Coaching Report) following a written examination, either upon request or for audit purposes. As explained in our Accreditation Operational Guidelines, “The Report will identify each question by registration number and examination sequential number, candidate identification number, subject, and identify by answer all questions answered incorrectly. All questions will have a record of the number of times it was answered correctly and incorrectly, and provide a percentage figure for the correct responses.” In the event a Program Manager or Coordinator identifies a problem with a given question, or if a question is consistently answered incorrectly more than 70% of the time, the concern will be brought to the attention of the FSD’s Deputy Director for appropriate action.

As a JIBC evaluator, if you note any concerns with a testing instrument or test item, please bring it to the attention of your Program Manager or Coordinator. You are an important part of our testing and certification processes and your feedback is always welcome.