Is this as unnerving as it sounds?

Imagine watching a master chef create a complex dish, but not knowing exactly how the ingredients blend or why certain flavors emerge. That's kinda like what’s happening with large language models, or LLMs. As /u/reasonablejim2000 points out, we can see how they're trained — step by step, tweak by tweak — but the real mystery is *why* certain circuits or patterns pop up. According to Andrej Karpathy, there's a ton happening under the hood that we don’t fully understand. It’s like the AI is learning in ways that aren’t transparent, and honestly, that’s a red flag. If we don’t grasp why these models develop certain structures, how can we truly trust or control them? As Jim highlights, this hidden complexity isn’t just academic; it’s a real-world problem we need to figure out before rushing ahead. So, get this — what happens if these emergent patterns turn out to be unpredictable? That’s the kind of uncertainty that could define the next phase of AI development.