@Grok Is This True? LLM-Powered Fact-Checking on Social Media

Large language models (LLMs) can both correct inaccurate beliefs and manipulate attitudes. As LLMs are increasingly deployed as fact-checkers on social media, their usage and effectiveness remain unclear. Here, we analyze an exhaustive sample of 1,555,281 English-language fact-check requests sent to Grok and Perplexity bots on X from February through September 2025 and conduct a preregistered survey experiment (N=1,592). We find clear evidence of polarization. Grok requesters skew heavily Republican; familiarity with and trust in specific LLMs vary substantially - such that trust in AI is no longer a unitary construct - and are polarized along partisan lines; and revealing that a fact-check came from Grok polarizes belief updating. Yet polarization does not neutralize LLMs' corrective power. Republicans actively solicit Grok fact-checks (in contrast to Republicans' frequent rejection of professional fact-checking), both parties disproportionately fact-check Republican-authored claims (which are rated less accurate by all models, including Grok), and fact-checks shift beliefs across party lines. Some models can match agreement among professional fact-checkers - yet this technological capacity can be undermined by model designers. For example, Truth Social AI, which uses Perplexity with imposed news source restrictions, performs substantially worse than Perplexity itself. The trajectory of LLM fact-checking will depend less on technological capability than on usage patterns and design choices.