BA420 Unit 4 Interobserver Agreement Paper Example

Eighty-Nine Percent Agreement That Was Not Really Agreement: A BA420 Unit 4 Interobserver Agreement Paper

[Student Name]

Purdue University Global

BA420: Methods of Behavior Analysis

Unit 4 Interobserver Agreement Paper

[Instructor Name]

[Date]

The writer, the school, the student and every figure are composite and illustrative; the articles and book cited are real.

Two Observers, One Class

Reliability checks are a weekly routine in the middle school where I work as a behavior technician. Tyrell, a lively sixth grader, often shouts out answers and remarks before anyone calls on him, and his behavior plan tracks how often. Once a week, the special education teacher sits in and observes with me to check that our data are reliable. In one forty-minute science class in October, I recorded eighteen call-outs and she recorded sixteen. We sat on opposite sides of the room, recorded on separate tablets that timestamped each mark, and did not talk during the period.

Interobserver agreement, the degree to which two independent observers record the same values, is the main way behavior analysts check the reliability of their data. But there are several ways to calculate it, and they do not always give the same answer.

Four Algorithms

Total count agreement divides the smaller total by the larger: 16 divided by 18, or 89 percent. It is simple and quick, but it says nothing about whether the observers recorded the same call-outs. Two observers could each record sixteen call-outs at entirely different moments and still agree perfectly by this method, which makes it the weakest check of the four.

Exact count-per-interval agreement divides the session into short intervals and counts the proportion of intervals in which both observers recorded exactly the same number. With two-minute intervals, we matched exactly in fourteen of twenty intervals, or 70 percent. This algorithm is much stricter, because it requires agreement about when the behavior happened. Looking at the six intervals where we differed, most involved a burst of two or three call-outs within a few seconds, which one of us had counted as fewer separate events.

For partial-interval recording, which the team also tried with ten-second intervals, interval-by-interval agreement divides the number of intervals in which both observers agreed, whether the behavior occurred or not, by the total intervals. That produced 92 percent. But call-outs occurred in only a small fraction of intervals. Most agreements were about intervals in which nothing happened, which inflated the figure. Scored-interval agreement, which counts only intervals in which at least one observer recorded the behavior, produced 75 percent.

Ninety-two percent agreement mostly meant we agreed that Tyrell was quiet; when he called out, we agreed only three times in four.

Why Algorithms Matter

Mudford et al. (2009) reviewed the agreement algorithms reported for continuous recording in a leading applied behavior analysis journal. They found that a variety of algorithms were used and that some common ones, such as total count and certain interval-based methods, can produce high agreement even when observers disagree about individual occurrences. They recommended choosing algorithms that match the data and reporting them clearly so readers can judge reliability.

Our team now uses exact count-per-interval agreement for Tyrell's call-outs and scored-interval agreement for low-rate behaviors recorded with intervals, and reports which algorithm was used alongside every figure in the student's progress reports, so that parents and teachers know what a percentage means.

Artifact and Bias

Kazdin (1977) described several sources of artifact and bias in observational data. Among them are observer drift, a gradual change in how an observer applies a definition, and reactivity, in which observers record differently when they know their accuracy is being checked. He also noted that agreement tends to be higher when observers know who will check them and when they can communicate about their scoring.

My own data showed both. In December, my count of Tyrell's call-outs had crept upward compared with the teacher's. Reviewing a recording, we found I had started counting quiet comments to a neighbor as call-outs, which the definition excluded. That was drift. We met, reviewed the definition with examples and non-examples, and practiced on video until our counts matched again. I had not noticed the change in my own scoring at all, which is exactly why drift is dangerous. Reactivity appeared too: on days when the district behavior analyst announced she would compare our records, our agreement was consistently higher than on unannounced days. The team now schedules some checks without notice, and the district analyst compares announced and unannounced agreement each quarter to see whether the gap persists.

Agreement for Other Dimensions

Agreement has to be adapted to the measurement dimension. For Jaylen's out-of-seat behavior, which I time continuously, the team calculates total duration agreement, dividing the shorter total duration by the longer, and mean duration-per-occurrence agreement, which compares each episode's length. For Sofia's latency to start work, we compare each latency the two observers recorded. Cooper et al. (2020) describe these variations and note that the same principle applies throughout: the stricter the comparison of individual instances, the more meaningful the agreement. Total duration agreement for Jaylen was 94 percent in one session, but mean duration-per-occurrence agreement was 81 percent, because we had split one long episode differently.

Agreement Is Not Accuracy

High agreement shows that two observers recorded the same thing. It does not show that what they recorded was correct. If both of us had misread the definition in the same way, our agreement could have been perfect while our data were wrong. Accuracy requires comparison with a true value, such as a carefully scored video. The team now scores one video per month against a master record created by the district behavior analyst, which checks accuracy as well as agreement. In January, both observers were undercounting very quiet call-outs, an error agreement alone would never have revealed.

What Good Agreement Requires

The comparison taught me that reliability depends on three things: a clear definition, trained observers who are checked regularly, and an algorithm that measures agreement about the behavior that matters. A high agreement figure is meaningful only if the algorithm is strict enough to catch real disagreements and the checks are frequent and unannounced enough to catch drift and reactivity.

Conclusion

The same class of Tyrell's calling out produced agreement between 70 and 92 percent depending on the algorithm. Total count and interval-by-interval agreement inflated the figure, while exact count-per-interval and scored-interval agreement revealed real disagreements. Kazdin's account of drift and reactivity explained changes in our own records, and Mudford and colleagues' review supported the team's new reporting standard.

References

Cooper, J. O., Heron, T. E., & Heward, W. L. (2020). Applied behavior analysis (3rd ed.). Pearson.

Kazdin, A. E. (1977). Artifact, bias, and complexity of assessment: The ABCs of reliability. Journal of Applied Behavior Analysis, 10(1), 141-150. https://doi.org/10.1901/jaba.1977.10-141

Mudford, O. C., Taylor, S. A., & Martin, N. T. (2009). Continuous recording and interobserver agreement algorithms reported in the Journal of Applied Behavior Analysis (1995-2005). Journal of Applied Behavior Analysis, 42(1), 165-169. https://doi.org/10.1901/jaba.2009.42-165