Understanding the Impact of Range Restriction on LSAT Correlation Studies

Several empirical studies have found a link between LSAT score and bar exam performance. AccessLex is currently seeking to gain a better understanding of factors, both LSAT score and others, that influence bar passage through the Bar Exam Success Initiative in partnership with LSSSE. So far, our work suggests that there may be a tenuous relationship between the LSAT and bar exam outcome. At 5 of the 19 schools we studied, LSAT scores had some predictive relationship to bar exam performance. These relationships usually faded by the second year of law school, being overshadowed by law school grades. At the other schools, no relationships between LSAT scores and bar exam performance were found. However, there is a reason why the relationship between LSAT and bar passage may be difficult to detect—that being the influence of range restriction.

What is range restriction?

Range restriction occurs when “observed sample data are not available across the entire range of interest.”¹ In simpler terms, range restriction occurs when there is a large range of values, but we only observe a smaller subset. While possible LSAT scores range from 120 to 180, most researchers at law schools have a limited range of student LSAT scores (perhaps 150 – 165).

Why does it matter if range restriction is present? When looking at correlations between variables (e.g., LSAT score and law school GPA), the correlation will be substantially reduced if one variable is restricted. This is particularly problematic if one of those variables is used as a method to select individuals, such as using the LSAT in admissions. This means that even if there is truly a relationship between LSAT and law school GPA, you may not be able to detect it because the range of LSAT scores is necessarily limited.

Range restriction is a common problem for college admissions. A number of studies look at the influence of range restriction on higher education, including one looking at the GRE as a predictor of academic success in law school.

Looking at the influence of range restriction with simulated LSAT scores and law school GPA

To give a better example of how this works, I have created a simulated dataset of 5,000 students. Their LSAT scores range from 120 to 180 with an average score of 150. Their first-year GPA ranges from 0.50 to 4.00 with an average GPA of 2.93 (don’t worry that these numbers are unlikely to ever occur, that is part of the point). Figure 1 shows the relationship between the two variables. As expected, there is a positive relationship between LSAT score and first-year GPA. As LSAT score goes up, we expect that first-year GPA also increases. The correlation between the two variables is 0.45, which is relatively high.

Figure 1: Full dataset

What happens, however, when your school decides to admit only students with a 155 LSAT or higher? Basically, what happens if we get rid of everyone to the left of 155 on the x-axis? Figure 2 shows this exact situation. The red line is much flatter, and the correlation is only 0.19. This is a much weaker relationship, even though the underlying data is the same. By selecting students based on LSAT score, we are fundamentally weakening our ability to determine if the LSAT is a meaningful predictor.

Figure 2: When you only accept students with a 155 or higher

Now imagine one final scenario. As before, you only accept students with a 155 or higher. But you have never been able to yield those students above 165. You can guess what happens—the correlation drops even further (though very small changes do not necessarily lead to changes lower correlations). Figure 3 shows the new graph, which results in a correlation of 0.14. Remember that this is the same exact dataset in Figure 1, just a snapshot of the data.

Figure 3: Only enrolling students between 155-165 LSAT scores

What to do?

Several of the studies I have read have found a weak relationship between LSAT and some dependent variable and spend a few pages talking about why the LSAT should be given less weight in admissions and the U.S. News rankings. Then, at the very end, the paper may have a sentence or two explaining that the weak relationship may be because they are looking at a limited band of LSAT scores. For many, a recognition of the limitations is all that is feasible and/or appropriate. However, there is another method for responding to range restriction. For those who want to take a further analytical step, the most common statistical “fix” is to use a variation of Thorndike’s Case II method. Using a few assumptions about the selection criteria (how you admit students), one can easily estimate the corrected correlation by knowing the standard deviations of the independent variable (e.g., LSAT score), standard deviation of the dependent variable (e.g., first-year GPA) and correlation between the two restricted variables (correlation between LSAT score and first-year GPA). Most statistical software programs have functions that can do the calculations for you.  The result may not give the true correlation, but the method is likely to get you closer to the true correlation. 

Want to see more?

I have built an interactive web application where you can filter that same 5,000 simulated individuals. You can view it here: https://accesslex.shinyapps.io/range_restriction/. All graphs and the app were built using R. If you have any questions, contact us at research@accesslex.org

¹ http://methods.sagepub.com/reference/encyc-of-research-design/n388.xml