In this brief blog article, we will examine the difference between accuracy, reliability and validity.
This article is useful for registered behavior technicians (RBT) or students who are studying to become board certified behavior analysts (BCBAs). Understanding (Applied Behavior Analysis) ABA terms is critical for both being an effective ABA therapist and passing your BCBA exam.
In an effort to help you study for your BCBA exam more effectively, this post is written in a “study note” form rather than as a long form blog post. Happy Studying!
Definition of accuracy
Accuracy measures whether or not the value of a behavior being recorded represents the true value of the behavior. Simply put, it measures if the ABA therapist scored the behavior correctly.
How to measure accuracy
The biggest threat to accuracy is human error. The best way to measure accuracy is recording a situation such as an ABA session and compare the data that was taken against the true value.
For example, if you are recording the number of spontaneous tacts in an ABA session and you record 72 during the session, but when you watch a video of the session and you record data without any distractions it comes up as 92, then the data is not accurate.
In another example, if you are recording head banging and you record 1 in the session and then when watching the video it occurs 1 time, then that data is accurate.
Another way to measure accuracy is to use interrater reliability. This is simply to have two or more more observers measure the same behavior and compare the calculations to determine whether they are the same. When using this procedure, if they are the same, it is likely the data is accurate (although that is not 100 percent guaranteed). If the data is different than it is certain at least one (and possibly both) are inaccurate.
Definition of reliability
Reliability refers to a measurement procedure producing the same result. In simple terms, data is reliable when it occurs consistently.
Examples of reliability
Example 1.) You weigh yourself on a scale 5 times in one day. Each time, the scale shows 150 lbs. (* It is important to note in this example, the data is reliable but it may not be accurate. The scale could be broken and could be consistently measured incorrectly.)
Example 2.) When asked his name, Dan responds correctly 100 percent of the time when probed with five trials over the course of three days.
Examples of data that is not reliable
During three ABA sessions, a learner engaged in 50, 8, and 22 independent tacts across the same activity.
When data is taken on a child’s ability to independently tie his shoes from start to finish, it was marked as correct three days and incorrect three days.
How to measure reliability
It is important that before mastering out a target in an ABA repertoire or establishing a baseline that a behavior occurs consistently. If a behavior occurs just once, it could be a fluke. A child could get a high score because he or she guessed correctly or could demonstrate high instances of a problem behavior due to an outside factor such as being tired, not feeling well or being upset by an event earlier in the day. For this reason, always take data measurements on multiple occasions to ensure that they are reliable.
Definition of valid
Validity is whether or not you measured what was intended to be measured. It has nothing to do with how you measure something but simply that you measured what you were supposed to. Therefore data could be valid but not accurate. If you are scoring the correct thing but miss some instances of the behavior, it is valid but not accurate.
The best way to ensure data is valid as a behavior analyst is to ensure that the definition of a behavior is clear:
For example, when recording vocal stereotypy, if you write the definition as, “Joe will engage in vocal stereotypy,” the therapist will mark down all instances of vocal stereotypy including singing, humming, talking to himself and other sounds that Joe might make that does not have communication intent. If you actually wanted to record the behavior of Joe emitting a high pitched vocal sound, then the data would be invalid because it did not measure what you intended it to measure. In this situation, you can solve the discrepancy by making the definition more clear. “Joe will emit a high pitch vocal utterance that he will hold for more than three seconds.”
Let’s example some more scenarios in which data is valid and invalid.
Examples of valid data
Example 1.) The BCBA on the case asks the therapists to record pure mands. The therapist records every time the learner spontaneously asks for something in the session. The data is accurate because the correct thing is being recorded.
Example 2.) The BCBA asks the ABA therapist to include sharing with his or her peers in a preschool setting. The ABA therapist takes data when she is at preschool and the learner is interacting with peers. The data is accurate because the correct thing is being recorded.
Example 3.) The BCBA asks the therapist to record how many times a child spits out food during breakfast. The therapist works with the child during breakfast and records how many time the child spits out food. The data is accurate because the correct thing is being recorded.
Examples of invalid data
Example 1.) The BCBA on the case asks the therapists to record pure mands. The therapist records every time the learner says any word including tacting and echoics, The data is invalid because the wrong thing was being recorded.
Example 2.) The BCBA asks the ABA therapist to include sharing with his or her peers in a preschool setting. The ABA therapist takes data on sharing blocks with her in her home session. The data is invalid because the wrong thing was recorded.
Example 3.) The BCBA asks the therapist to record how many times a child spits out food during breakfast. The therapist records and escapes spitting behavior in the session. Again, the data is invalid because the wrong thing is being recorded.
References
Cooper, J. O., Heron, T. E., & Heward, W. L. (2019). Applied Behavior Analysis (3rd Edition). Hoboken, NJ: Pearson Education.
