Humans invent stuff (without realising) it to, so I don’t think that’s enough to disqualify something from being intelligent.

The interesting question is how much of this is due to the training goal basically being “a sufficiently convincing response to satisfy a person” (pretty much the same as on social media) and how much of it is a fundamental flaw in the whole idea.