Washington Post columnist Jay Mathews famously invented the so called Challenge Index, which purportedly measures how well high schools challenge all students with coursework. Advanced Placement courses, so the argument goes, gives them a taste of university education. To quote Mathews, the Index is a “measure of how effectively a school prepares its students for college.” To come up with the measure you simply “Divide the number of Advanced Placement, International Baccalaureate or other college-level tests a school gave in [any year] by the number of graduating seniors.”

Of course, ensuring a high ranking is then as simple as pushing large numbers of students to sign up for the so-called “college level” tests. Common sense also dictates that schools pushing students into Advanced Placement (AP) and “other college-level courses,” are most likely to be situated in the Washington metropolitan area where the Washington Post holds sway. Indeed, in 2010 a column exulted “When compared with schools across the country, Washington area schools are overall the most encouraging of college-level testing, as measured by the index.”

Mathews has explained why he chose not to use AP passing rates: “I decided not to count passing rates in this way because I found that many high schools kept those rates artificially high by allowing only top students to take the courses. AP, IB and AICE are important because they give average students a chance to experience the trauma of heavy college reading lists and long, analytical college examinations. Research has found that even low-performing students who got a 2 on an AP test did significantly better in college than similar students who did not take AP.”

Montgomery County Public Schools (MCPS), the largest public school system in Maryland, opens a window into AP performance in high schools with a memorandum on “2014 Advanced Placement Exam Participation and Performance for Students in Montgomery County Public Schools and Public School Students in the State of Maryland and the Nation.”

At first glance the memo highlights the gap in performance between white or Asian students and blacks or Hispanics—the so-called achievement gap. According to the statistics, 78.3% of Asians and 81.2% of whites scored 3 or higher on AP tests. Just 49.8% of blacks and 57.7% of Latinos achieved the same benchmark.

The interesting statistics are buried a little deeper in the memo. For example, 35.2% of MCPS students scored a 5 on Calculus AB; 62.1% scored a 5 on Calculus BC. In contrast, just 19.8% scored a 5 in U.S. History. English Literature and Biology also proved tough for MCPS high school students with just 12.9% and 14.1% scoring a 5 on those two subjects.

When disaggregated by high school, the scores prove to be even more intriguing. For example, at Montgomery Blair High School which houses a math magnet program, just 13.3% scored a 5 on Calculus AB. Poolesville High School, which also houses a math magnet and is a whole school magnet, posted 55% on the same metric, with Quince Orchard reporting 72.6% and Thomas Wootton 77.3%. When it came to Calculus BC Blair did much better with 71.6% scoring a 5; Poolesville with 72.2% also improved, and Quince Orchard proved it was no slouch with 81.8% scoring 5. Winston Churchill with 83.2%, Walter Johnson with 83.3% and Clarksburg with 84.0% were among the top performers. When it came to AP Statistics, both Blair with 50.0% and Poolesville with 52.3% did very well. When it came to AP Biology, it was Poolesville with 35.0% and Walter Johnson with 35% scoring 5 took the crown. Churchill with 60.8% and Wootton with 39.3% led the way in students scoring 5 on AP Chemistry.

Top scorers in AP Physics C were Bethesda-Chevy Chase, Walter Johnson, and Walt Whitman, with Poolseville and Blair following close behind. Calculus powerhouse Quince Orchard didn’t have any reported participants. In case you are wondering, coming in fourth Poolesville was the highest ranked on the 2014 Post Index; Richard Montgomery was 5th, Churchill was 8th, Bethesda-Chevy Chase 11th, Whitman 13th, Walter Johnson, Wootton and Quince Orchard came in 14th, 15th, and 16th, respectively.

With these patterns, AP performance across MCPS high schools alludes to the possibility that all high schools don’t prepare students equally well. Getting a score of 5 in a given AP course, if we are to believe the data in the memorandum, depends on the high school you attend. If one were to believe MCPS data, the Washington Post Challenge Index is hardly a “measure of how effectively a school prepares its students for college.” Nor is it an indication of the quality of instruction. Like most school ranking measures, the Challenge Index doesn’t seem to mean anything.