Revisiting Cut Scores in the Wake of COVID-19

Cut scores are integral to many standardized tests. In some cases, they have a profound impact—perhaps nowhere more so than in the case of bar examinations. A bar exam cut score is the primary hurdle that a law school graduate must clear in order to join the legal profession.

In light of the coronavirus pandemic, now is the time to explore bar exam cut scores in greater depth.  Jurisdictions have been forced to confront a totally changed professional licensing landscape, and many states face renewed pressure to revisit their cut scores, re-igniting long-standing debates about minimum passing scores for the bar exam.

Such debates have been especially prominent in California. Critics long opposed the state’s high cut score, arguing that it has a disparate impact on test-takers of color; however, others argued that the exam was not unduly difficult and pointed to rates of lawyer discipline as a reason not to lower the cut score. Similar debates are now playing out nationwide. When the California Supreme Court announced in mid-July that it was moving the state’s bar exam online and instructing the state bar to create a supervised licensure program for 2020 law graduates, it also took the significant step of permanently lowering the state’s bar exam cut score. Will other states follow suit? As bar examinations, and attorney licensure more broadly, face increased scrutiny, such measures—previously thought by many to be unlikely—are entirely possible or even expected.

Before the pandemic, high cut scores had their defenders and their detractors. (The diversity of cut scores across the country—even among jurisdictions that have adopted the UBE—is another area of concern.) Now is the time to consider several questions: Why are cut scores set as they are? How are the decisions that determine cut scores made? How will the pandemic affect these decisions (and how has it already)? This post explores these questions, although it leaves other questions—such as those regarding the relationship between the bar exam and minimum competence to practice law—aside.

Cut scores are the result of value-based decisions

A cut score is simply a point value for a given test, one that is chosen and intended to establish a measurement threshold for the acquisition of knowledge or skills.1 And although many of the most prominent tests utilize a single cut score (a pass/fail threshold), more than one may be used, each indicating a different level of performance. 

Cut scores must be set somewhere. (As the late Supreme Court Justice John Paul Stevens put it, “[e]xaminations are a permissible method of determining qualifications, and lines must be drawn somewhere.”2) But decisions about cut scores are never purely—or even primarily—technical. They are expressions of what a community of stakeholders has decided is the appropriate threshold for licensure. In the words of the prominent psychometrician Michael Zieky, “cut scores are constructed, not found.”3 Setting a cut score is necessarily the work of policymakers, content experts and psychometricians working together.

Even under ideal circumstances, the task of setting a bar examination cut score represents something of a compromise—one that is made jointly by bar regulators and assessment specialists. Consequently, setting cut scores is a normative decision—a decision about the value of what is assessed, about how the assessment is serving a broader community or constituency. Those who are tasked with determining cut scores cannot avoid making a decision about whether it’s more important to avoid false positives (e.g., admitting unprepared attorneys to the bar) or to avoid false negatives (e.g., shutting the door in the face of law graduates from systematically under-served backgrounds or populations).

In the case of the bar exam, these decisions have historically had substantial disparate impacts. The most recent national estimate (for students entering law school in 1991) found a gap of nearly 20 percent for Latinx test takers and 30 percent for Black test takers.4 More recent jurisdictional data from New York and California also show little narrowing.

There are no ideal cut scores

The recognition that cut scores are set through judgment—and that such judgments are inevitably controversial—does not, by itself, provide any guidance as to how cut scores should be set. It simply suggests that there is no such thing as an ideal cut score for any given test. Likewise, current cut scores do not deserve deference simply because they have already been chosen. Those scores merely represent past decisions, not technical processes whereby an ideal cut score was somehow found.

Indeed, existing cut scores will require adjustment in those jurisdictions that are altering their bar exam in response to the pandemic. When an exam changes, a new cut score determination must be made—and it is an essential step when designing a wholly new exam. The time is ripe to consider whether current and proposed cut scores unfairly shut out large proportions of minority test-takers from entering the legal profession (according to the most recent ABA Profile of the Legal Profession, minorities comprise thirty-one percent of law school students but only fourteen percent of lawyers in the U.S.) and, accordingly, to consider how the cut score should be determined.

The cut score decisions that have been and will be made this year will have a meaningful and lasting effect on the legal profession.

¹ Cut scores can be, and frequently are, pre-established threshold values (usually referred to as “criterion-based”), or they can be determined after the test has been administered (and graded), based on the performance of a given group on the test (for example, using the 75th percentile for all test takers in a given administration as the threshold—this is usually referred to as “norm-based”). However, “[i]n practice, the distinction between norm-referenced tests and criterion-referenced tests is often blurred. Criterion-referenced tests are always ‘normed’ in some sense. That is, the criterion cutoff scores are not determined at random. The cutoff score will be associated with a decision criterion based on some standard or expected level of performance… The distinction between criterion- and norm-referenced tests is further blurred when scores from norm-referenced tests are used as cutoff scores.” R. Michael Furr, Psychometrics (3rd edn., 2018), 8.

² Board of Regents v. Tomanio, 446 U.S. 478, at 493.

³ Michael J. Zieky, So Much Has Changed: An Historical Overview of Setting Cut scores, in Setting Performance Standards: Foundations, Methods, and Innovations (Gregory J. Cizek, ed., 2012), at 28. Cf. Kurt F. Geisinger and Carina M. McCormick, Adopting Cut Scores: Post-Standard-Setting Panel Considerations for Decision Makers, 29 Educational Measurement: Issues and Practice 1 (2010).

⁴ Linda R. Wightman, LSAC National Longitudinal Bar Passage Study (1998).