It destroys meaningful operations it comes into contact with, and requires invisible and growing workarounds to maintain (e.g. “countably” infinite vs “uncountably” infinite) which smells of fantasy, philosophically speaking.

This isn’t always true. The convergent series comes to mind, where an infinite summation can be resolved to a finite number.