When or If AI Meets Indigenous Knowledge?

An Ideation Series: Part 4, The Questions We Should Ask Before AI Touches Community Knowledge

I keep coming back to a moment from a few years ago. I was in a meeting about a program, and someone from a funding agency said something like: we just need the data. And I remember thinking, no, you don't. You need what the data means, and you need the people who can tell you what it means, and you need to understand that some of it isn't yours to ask for in the first place. I didn't say it out loud. I've been saying it out loud more since.

That's what this part is about. Before any AI system touches community knowledge, I think there are questions that have to be asked, and I think they have to be asked out loud, on the record, by people who have the authority to ask them. I've been calling this an Indigenous Knowledge and AI Impact Assessment, but the name matters less to me than the questions. So let me write the questions down the way they live in my head.

First: who holds this knowledge, and who has the right to decide what happens to it? Not who collected it, not who scanned it, not who has a copy. Who holds it. Who carries it. Who is accountable to the community for it. If the answer to that question is nobody, then the knowledge shouldn't be touched, full stop.

Second: what is this for? What is the actual purpose? Not the stated purpose, the real one. Is it to help a community access a service they need? Is it to build a product? Is it to save money? Is it to look innovative? I want the purpose named, because a purpose can be tested. A vague purpose can hide anything.

Third: who benefits? This one sounds simple, and it never is. If a system is built on community knowledge, who benefits? Does the benefit flow back to the community, or to a vendor, a department, or a shareholder? I've watched too many projects where the community provided the knowledge and someone else got the credit, the money, or both.

Fourth: who could be harmed, and how? I don't mean the obvious harms, the data breach harms. I mean the slow harms. The harm of a story being flattened. The harm of a practice being described wrong and then that wrong version becoming the version everyone believes. The harm of a community seeing its own sacred things reduced to content. Those harms don't show up in a risk register, but they're real, and they last.

Fifth: how long does this knowledge stay in the system? What happens when the project ends? What happens when the grant ends? What happens when the vendor changes? Is there an exit, a removal, a return? Or does the knowledge just stay, forever, in a server somewhere, outliving the agreement that put it there? I think about that a lot. Servers don't forget. Agreements do.

Sixth: what else could this be used for? Because the whole problem with AI is that it takes what it's given and finds new uses nobody asked about. The question isn't just what this system will do with the knowledge. It's what every future system might do. I don't know how to answer that question fully, and I think pretending we can is dishonest. But I think we have to ask it anyway, and design as if the answer might be: anything, unless we stop it.

And seventh, the one I keep coming back to: who is the human responsible when something goes wrong? Not the algorithm, not the system, not the model. A person. With a name. Who can be called to account. Because I've noticed that the more AI enters public systems, the easier it becomes for responsibility to disappear. There's always a vendor, a system, a process, a policy. I want a person. I want to know their name. I want them to be able to say no.

Here's where I doubt myself, though, because I know how these assessments actually go in the real world. They become checkboxes. They become a form you fill out to say you've done the responsible thing, and then the project proceeds exactly as it would have anyway. I've seen it. We all have. And I worry that an Indigenous Knowledge and AI Impact Assessment could become exactly that: a way to feel good while the extraction continues. That worry is real, and I don't have a clean answer to it. What I have is the belief that the questions are still worth asking, because a question asked in public is harder to ignore than one never asked at all. And I believe that the assessment has to be connected to consequences. If the answers reveal harm, the project has to change. Not be documented. Change. Without that, it's just theater, and I don't want to build theater.

I keep saying the Commons is about pairing governance with usefulness, and I think the assessment is where those two meet. It's the place where we say to a public institution: yes, here is a tool that can help, and here are the questions you have to answer before it touches what our communities hold. Not to slow things down for the sake of slowing things down. Because we've learned, the hard way, what happens when the questions don't get asked.

I'm still working out what makes an assessment real instead of decorative. I don't have that fully solved. But I know the questions, and I know they have to be asked by people with the authority to make the answers stick. In the next part, I want to talk about something that sounds dry and isn't: the contract. Because I'm starting to think the contract might be where much of the real power lies.

— Olani Lilly, CoMission LLC · mission.consulting