Interobserver Agreement (Ioa)

The idea that practicing behavioral analysts should collect and report reliability or interobserver compliance (IOA) in behavioral assessments is illustrated by the Behavior Analyst Certification Board`s (BACB) assertion that behavioral analysts are proficient in using “different methods of evaluating the results of measurement methods, such as.B. compliance between observers, accuracy and reliability” (BACB, 2005). In addition, Vollmer, Sloman and St. Peter Pipkin (2008) argue that the exclusion of these data significantly limits the interpretation of the effectiveness of a behaviour modification procedure. Therefore, the inclusion of reliability data for validity claims in any study that includes behavioural assessment should be the inclusion of reliability data (Friman, 2009). Given these considerations, it is not surprising that a recent review of journaled journals in Applied Behavior Analysis (JABA) from 1995 to 2005 (Mudford, Taylor & Martin, 2009) found that 100% of articles reporting continuously recorded dependent variables contained IOA calculations. These data, along with previously published reports on JABA`s reliability procedures (Kelly, 1977), indicate that the inclusion of IOA is in fact a trademark – if not a standard – of behavioural assessment. This is not a very strict agreement procedure, as a total number of 100% IOA could result from two observers recording totally different case observers of targeted responses under the same 15m observation. Observe 1 records three instances of the target response during the first 3 meters (one per minute) of its observation, two instances during minute 4 and misses all other instances for the remaining 12 m.

During the same hypothetical observation, Observer 2 missed all three instances during minutes 1 to 3, records one response instance at minute 4, but records four instances at minute 15. Although these are totally different events, the total number of IOA that results from them would still be 100%. Average number per interval: 1) Divide time into intervals, 2) Observers record the frequency of behaviors per interval, 3) calculate the concordance by interval (similar to the total number), 4) the interval IOA, 5) divide by n intervals (calculate the average) A method of improving the credibility of data comparing independent observations of two or more people of the same event. The IOA is calculated by diverging the number of agreements between independent observers by the total number of agreements plus disagreements. The coefficient is then multiplied by 100 to calculate the percentage (%) of compliance. Interval-based IOA algorithms evaluate the concordance between data based on the interval of two observers (including time samples). These measures consist of (a) Interval-by-Interval, b) Scored Interval and (c) intervalized IOA algorithms. After a brief overview of each interval-based algorithm, Table 2 summarizes the strengths of the three interval-based algorithms.

Consider as a common example of interval-based IOA the hypothetical data stream described in Figure 2, in which two independent observers record the occurrence and absence of a target response at seven consecutive intervals. In the first and seventh intervals, observers disagree on the event. However, both observers agree that no response has been given to the second, third and fourth intervals. Finally, the two observers also agree that at least one reaction occurred during the fifth and sixth intervals. . . .