LLMs used directly as classifiers have significant limitations including poor calibration, inability to properly incorporate structured data, and lack of interpretability. The solution is to treat LLM outputs as features in traditional ML models like logistic regression, which recovers desirable properties like calibration, information integration, and interpretability.