Look Closely! One Star? Or, an Entire Constellation?

Comparison:

So the best way I've learned to describe it [when an algorithm interprets individual bits of data], and this is almost ten years on, is as constellations, right? Sometimes the stars form something when you see them with other stars. But other times, they're just a star, right?

Kevin DeLiban, founder of TechTonic Justice, helps us understand how an AI algorithm might on one calculation see an individual bit of data as simply singular, and at other times see that same bit of data as in a configuration of meaning with other bits of data taken from extensive interview forms. Such AI interpretive processes were used by the state of Arkansas to evaluate extensive interview data from possible recipients of state aid for low-income, disabled citizens. DeLiban notes that each application of such AI programs to social service always results in decreased services. Seems that the constellations point in one direction only.

Context:

DE LIBAN [founder of TechTonic Justice]: . . . So the first part of the process [for an Arkansas state welfare system process] was a 286-question assessment. So the state nurse would come out and ask you 286 questions, usually taking an hour and a half to two hours to do, which is an exhausting process. At the end of that, the nurse would push a button and the responses from those 286 questions would be run through an algorithm that would group people into one of 23, what they call the acuity categories or severity categories.

And then each of those 23 categories had a fixed number of hours [of home care] attached to them, that really couldn't be deviated from at all.

CHAKRABARTI: It couldn't be deviated from at all. I want to know more about that. Yeah.

DE LIBAN: Sure. So the way the algorithm works is of those 286 questions, it turns out only around 60, give or take a few, ever matter.

DE LIBAN:

But they don't all matter in the same way. They only matter sometimes when aligned with other factors. So the best way I've learned to describe it, and this is almost ten years on, is as constellations, right? Sometimes the stars form something when you see them with other stars. But other times, they're just a star, right?

That's how the algorithm worked with those questions from the assessment that actually mattered. Some of them mattered some of the time, and there was never any explanation about which mattered which of the time. And the state couldn't explain it, let alone expecting beneficiaries who are on the program to suddenly understand how come their care is being cut drastically, when their conditions haven't been improved.

Citation:

Arnold, Willis Ryder and Meghna Chakrabarti. "What Happened When AI Went After Welfare Fraud." On Point, NPR, 13 March 2025. Web.

(Image design by Lee Aigue; base images courtesy of sebastien lebrigand, flickr, 09 Oct. 2013, CC-2.0 and Bing March 2025.)