AccessLex
Not So Fast: Predicting Law School Bar Success Is More Complicated Than You Might Think
With a decline in bar passage rates over the past decade and the recent adoption of the ABA Standard 316—requiring that, to stay in good standing with the ABA, at least 75 percent of a law school’s graduates who sat for a bar exam have passed within two years of their graduation—bar success has become high stakes for law graduates and law schools alike. Amidst these recent developments, studies applying statistical methods to model and predict law schools’ bar passage rates have captured the attention of the legal academy. The reason for their attraction is obvious; these empirical methods can be useful tools to assist a law school in anticipating rough seas ahead as well as operationalizing a measure of the value that a law school can provide to its students. But these methods are nearly as complex as the matter to which they have been applied—to wit, bar exam success—and require skill and care in their use.
One recent application of these statistical methods to tackle the bar success conundrum comes from a 2021 report by Professor Jeffrey Kinsler,1 estimating under- and over-performance among law schools on two key outcome metrics: bar passage rate (the overall percentage of successful bar exam takers in a given year at a given law school) and bar passage rate differential (the difference between the percentage of successful bar exam takers from a given law school in a given year in a given jurisdiction and that jurisdiction’s overall passage rate). 
To estimate under- and over-performance, Kinsler uses an entering cohort’s median Law School Admission Test (LSAT) score and undergraduate grade point average (UGPA), separately, to predict a value for the two bar passage outcomes he employs. He then compares this predicted value to the actual value of a law school’s bar success and ranks law schools based on the difference between these values.
While Kinsler’s approach is novel, his is not the first to use statistical methods to predict and measure this construct of bar success.2 Because this approach has received much attention, we think it is important to bring to light the difficulties in performing this type of analysis and the efforts currently underway to more rigorously estimate under- and over-performance and, ultimately, better inform the legal education community. Our goal is not to discredit Professor Kinsler or his work; rather, we wish to express employing these regression methods necessitates that a set of assumptions be satisfied. Failing to satisfy these assumptions can yield flawed results.
In early 2021, Professor Rory Bahadur, Kevin Ruth, and Professor Katie Tolliver Jones published a study centering the problems associated with applying linear regression to heteroscedastic, non-linear data—in effect, a critique of Kinsler’s 2021 study, which was based on his 2018 study. In Kinsler’s original study, he employed either median LSAT scores or median UGPA for a given law school to predict a single outcome—either bar passage rate or bar differential rate.3 Conclusions from underspecified models like these should be avoided because they cannot meaningfully control for bias; nor can they satisfy the underlying assumptions for using linear regression in the first place. Our research, independently, has found that Kinsler’s models are misspecified and, consequently, his rankings are inaccurate. Consequently, we agree with Bahadur, et al., that the Kinsler methodology is problematic, because it reduces the complex problem of bar success to a single measure of a single input against a single outcome.
Many factors contribute to a law school’s bar
passage rate, ranging from the entering characteristics of its students to school characteristics including class size, student-faculty ratio, and attrition and transfer rates. Furthermore, school-level bar passage rates change annually—and not uncommonly substantially—which means that results from a model using only one year of data will also be volatile from year to year. For example, Seton Hall Law School’s bar passage rate increased from 82 percent in 2018 to 90 percent in 2019. Other factors often vary widely year-over-year, such as attrition and transfer rates. And although a school’s median LSAT score and class size are relatively stable over time, it is apparent that just as no single factor should be used to predict bar passage rates, so too should no single year be used.
We appreciate the rigor that Professor Bahadur, et al., have used in evaluating the use of statistical methods to model and predict bar success. We are currently applying the same rigor in two separate projects to improve upon the work that has been previously completed and are taking great care to avoid the issues that Bahadur, et al., outline in their working paper.
In the first, a project led by the first two authors, we are investigating the predictive relationship of student credentials and background characteristics on bar passage. This approach departs from previous studies of bar exam success in that we quantitatively and qualitatively consider the factors that precipitate success or failure on the bar exam. Our objective is different from the purpose of prior studies of bar examination success in that we seek to identify the relationship between student characteristics and first-time bar exam passage rates. In the first phase of our research, completed in June 2020, we identified law schools that have consistently higher and lower first-time bar exam passage rates than their measures of entering student credentials would predict for a 6-year period spanning 2014-2019. Using these results, we surveyed law school administrators from across this distribution to highlight programs that overperforming law schools employ in order for average or underperforming law schools to develop similar programs that redound to their students’ bar exam success. Our study is forthcoming later this year, and it is our hope that it yields a greater understanding of the bar success initiatives and curricular programs engaged by overperforming law schools, thereby increasing the value proposition for all law schools.
In the second, a project led by the third author, we are using data reported by law schools for those students that graduated between 2016 and 2019 and attempting to create a model that reliably predicts a given school’s bar passage rate differential, taking into account factors such as its median LSAT score, median UGPA, attrition and transfer rates, and other characteristics. In addition, recognizing that schools themselves are often quite unique (e.g., size, geographic location, demographic composition of student body, programmatic differences), we account for these between-schools differences by employing both fixed effects and between-within models. Essentially, this approach allows us to compare each school to itself and then average the results across all of the schools. By comparing the predicted values that we produce to the actual values that schools report on their 509 Standard Disclosure Statements, we hope to provide a measure of the “value added” by each active ABA-approved law school. Ultimately, the final model should be updateable, allowing us to calculate new value-added rankings each year based on the most recent data. The report and its findings should be available this summer.
Both of these approaches take great care to avoid the concerns Bahadur et al. describe in their working paper; that is, those surrounding linearity and heteroskedasticity. Below, we briefly describe each concern and how our approaches mitigate the problem.
Linearity: The relationship between an input and an outcome variable must be linear to satisfy a core assumption of ordinary least squares (OLS) regression. In general, the issues of nonlinearity that Bahadur et al. describe affect regressions that use bar passage rates only (this is because bar passage rates are fixed within a 0-100 range, with a substantial majority above the 75 percent threshold, which results in the inaccurate predictions Bahadur et al. describe). Our approaches rely on bar pass rate differential, which is not restricted by these bounds and thus better lends itself to linear regression. In addition, the first two authors use all three measures (25th, 50th, and 75th percentiles) of LSAT and UGPA to comprise a weighted composite variable and then standardize the composite, with a mean of 0, to comprise a singular, predictive performance index. This transformation makes the composite variables continuous, by design, and linear, satisfying the linearity assumption of OLS.
Heteroskedasticity: The variance of the residuals—or error terms—must be evenly distributed across the data’s underlying input variables. This is tied directly to the linearity assumption above. Our approaches mitigate heteroskedasticity, as noted in the paragraph above, by using each school’s pass rate differential (rather than its actual pass rate) and using a composite LSAT/UGPA variable. The presence of many of the schools with high LSAT scores at the top of our ranking suggests that this criticism does not apply to our study.
Since our approaches address these concerns and satisfy the other three assumptions of linear regression, they are less susceptible to the bias that Bahadur et al. note; that “it is mathematically impossible for schools with the higher entering credentials to be considered overperforming according to Kinsler’s model. Kinsler’s model is strongly biased against the schools with the highest entering credentials because they will almost always be identified as underperformers” (p. 5). By contrast, our methodology does not demonstrate bias against any one particular group of law schools. In fact, among the top 25 law schools in our study, 18 are top 50 schools, as ranked by US News & World Report, and 11 are top 25 law schools in the US News rankings—all of which have among the highest entering credentialed students in the nation.
Regression analysis holds much potential for application to present considerations in the legal education sector. But, when empirical methods are employed even when the data suggests a problem, they can produce unreliable results. We appreciate Bahadur et al.’s collegial and academic approach to review Professor Kinsler’s work, which we do not mean to impugn here. Such critique adds to the rigor of research on legal education and, ultimately, redounds to a better understanding of bar success.
1 Jeffrey Kinsler, Top Law Schools for Passing the Bar Exam, Nat’l Jurist (Jan. /Feb. 2021). This report employs the same methodology that Prof. Kinsler used to produce a study published in 2018 to estimate the same. See Jeffrey Kinsler & Jeffrey Usman, Law Schools, Bar Passage, and Under and Overperforming Expectations, 36 Quinnipiac L. Rev. 183 (2018).
2 Compare Rory D. Bahadur, Kevin Ruth & Katie Tolliver Jones, Reexamining Relative Bar Performance as a Function of Non-Linearity, Heretoscedasticity, and a New Independent Variable, 4 (2021), SSRN Working Paper, available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3784863 (noting that “Kinsler’s publications were really the first to apply science, specifically the science of statistics, to the mystery of bar passage”—a reference to Kinsler’s and Usman’s 2018 study published in the Quinnipiac Law Review, first posted to SSRN on October 22, 2018) with Christopher J. Ryan, Jr., A Value-Added Ranking of Law Schools, 29 U. Fla. J. L. & Pub. Pol’y 285 (2019) (applying a predictive modeling technique based, a composite index of LSAT and UGPA, that measures the distance between predicted bar passage rates and actual bar passage rates for the same cohort three years after beginning law school, and posted to SSRN as a working paper on June 28, 2015). Indeed, using LSAT and UGPA to examine bar exam success has a longstanding academic pedigree. See Linda F. Wightman, LSAC National Longitudinal Bar Passage Study , Law School Admissions Council (1998) (analyzing, inter alia, the correlation and predictive power of a host of student-level data on bar passage outcomes in the aggregate).
3 It appears that, in the 2021 study, Kinsler performed the same single-input-single-outcome analysis he used in his earlier (2018) study for each law school, calculated a rank for each law school based on these analysis, and averaged the rankings over a five year span from 2015-2019.