The Model Completed the Pattern it was Given

I was sitting at a coffee shop near the north station in Valencia when an AI decided I was Colombian.

Claude Sonnet 4.6, to be exact. The assumption didn’t surface upfront. It came out sideways, while the model was apologizing for something else.

I was filling out a conference application. The model had drafted a response for the English language support field stating that English was not my first language. I corrected it. English is my first language. The model’s response: “you’re from Colombia, English is clearly not a barrier for you but it’s also not your first language, so that field was weirdly hedged.”

The model had been operating on a Colombian identity the whole time and hadn’t said so until I pushed back. The correction didn’t prompt it to ask where I was actually from. It prompted it to explain why Colombia made the English question complicated.

I’m Puerto Rican. I was born speaking English. The model had no basis for either assumption.

To be clear about why the specific misidentification matters: it’s not that being Colombian is in any way lesser. Spanish is spoken across 21 countries on three continents, from Spain in Europe to Equatorial Guinea in Africa to most countries across Latin America, and they are distinct from each other in history, culture, and identity in ways that don’t reduce to a shared language. The assumption that a Spanish surname and a Spanish-speaking location add up to a specific nationality is the same logic that leads people to assume every Spanish speaker is Mexican, or that “Latino” is one thing. It isn’t.

The model didn’t distinguish between any of them, nor did it have any context to do so. It saw a pattern it recognized, picked the most statistically probable output, and moved on.

Today’s post is walking through what I believe happened and thinking through some options for improvement with context in mind.

That sounds like a minor glitch, but it reveals a larger failure mode: the model was running an inference chain on my identity rather than asking for confirmation.

An inference chain is how AI models fill gaps in what they know. Rather than flagging uncertainty or asking a clarifying question, the model connects available signals and arrives at a conclusion it then treats as fact. In this case: “Sepulveda Morales” reads as Latino. Valencia reads as Spanish-speaking context. Latino name plus Spanish-speaking city resolves to a country of origin. The model picked one and moved on. It wasn’t the right one, but can it really even know it was right without feedback?

The location signal doesn’t even support the conclusion. I was applying for a conference in Spain. If geography was doing the work, Spanish or Catalonian would have been the logical output. The name outweighed everything else.

And the name is a weak signal too, for reasons that go back further than the model understands. Those names are Iberian in origin, carried into Puerto Rico through Spanish colonization. The same colonists who landed in Puerto Rico landed in Colombia, Venezuela, Cuba, and a dozen other places. Anyone carrying these surnames could trace back to any of them. The model had no real basis to pick one country over another.

What made it stranger: earlier in the same conversation, it was helping me with the application from an article where I named my identity directly. Puerto Rican. AI models work within a context window, the text of the current conversation that the model can see and reference, essentially its working memory for that session. My earlier statement was in that window. The model reached past something I’d explicitly stated and inferred something I hadn’t.

If it genuinely couldn’t trace a fact back to something I’d said, it could have asked. Four words: what nationality are you? That would have taken less time than generating a wrong answer with enough confidence to defend it.

The point isn’t that these contexts are identical, but that the same design flaw appears across AI systems: models infer identity from indirect signals like names, dialect, or location, then present that inference as settled fact rather than a guess.

A 2025 Brookings Institution study on AI resume screening (Wilson & Caliskan, April 2025) simulated the automated hiring tools now used by a majority of large employers. Researchers submitted identical resumes with different names attached, names statistically associated with Black or white, male or female applicants, and measured which ones got selected. Racial bias appeared in 93.7% of tests, with white-associated names preferred in 85.1% of cases. Gender bias showed up in 63% of tests. The model never saw race or gender directly. It inferred both from names.

Separately, research published in npj Digital Medicine found that large language models provided measurably different responses in mental health contexts depending on perceived race, with race inferred from name and dialect rather than anything the patient said. A model deciding how much empathy to extend based on what it thinks a name signals is the same mechanic as a model deciding what someone’s first language must be.

For most people, the inferred sentence just goes in. The form gets submitted. There’s no signal anything needs flagging.

There’s an obvious fix on paper: give the model better memory, so it doesn’t have to guess at what it already knows. But that solution comes with its own complications.

In March 2026, Anthropic rolled out persistent memory to all Claude users, including the free tier. Claude now automatically scans conversation history, generates a summary of key facts about the user, and carries that into future sessions: name, communication style, professional context, stated preferences. ChatGPT has offered a similar feature for paid users since early 2024. Other major AI tools are moving the same direction.

If that memory had been active and accurate in Valencia, the model would have known I was Puerto Rican before it ever drafted that language support field. The inference chain wouldn’t have had a gap to fill. Is that as simple of a solution though?

A model that remembers more also knows more about you, and that data doesn’t exist in a protected space. In 2025 and into 2026, lawyers began warning clients that AI conversation histories carry no automatic legal protection. Courts have already ordered AI providers to hand over chat logs. Law enforcement has subpoenaed them. In one federal case, Claude conversations were ordered produced as evidence in a securities fraud prosecution. The conversations you have with an AI tool, including details about your identity, your work, your concerns, are retained on company servers and can be requested. And in a world that is steadily sliding Far Right, most sane individuals would prefer to hide aspects like nationality from the government.

There are tools to avoid or at least curb this. Anthropic gives users the ability to view, edit, or delete stored memories, but most people won’t know to check. The capability that would have prevented a careless assumption about who I am is the same one that raises the question of who else gets to know. So, it gets complicated fairly quickly.

Fixing the product doesn’t fix the deeper problem. The categories the model used to guess my nationality weren’t invented by AI. They were inherited. Racial and ethnic classifications have been built and rebuilt over centuries by governments and institutions making political decisions about who counted as what, and that’s the data models train on. When the model saw “Sepulveda Morales” in Valencia and resolved it to Colombian, it was completing a pattern assembled from that history. It didn’t know the history. It didn’t say it was guessing.

You see a lot of the same when trying to classify someone as “white.” White is almost entirely a socially constructed term that means different things over time. Italians and the Irish were not considered white for a long period of the United States’ history. It would be an interesting experiment to train an LLM on data from the 1930s and see how it affects issues of racial stereotyping and prejudice.

Overall, through dissecting the data, the inference chain, how these models “think”… that’s where we get to the specific problem.

Not that the model got it wrong. Any system trained on skewed data will produce skewed outputs. The problem is that it presented a guess as a fact, defended that guess when corrected, and gave no indication throughout that it was doing anything other than reporting something it thought it knew.

The model didn’t know any of the historical context, or really anything for that matter. It just completed the pattern it was given.

Pictures of Cha Cha are few and far between for the next couple of months, but I can share one of him with his cousin Remington looking like they’re in jail from a week or so back. Onto the next!