The Food and Drug Administration's (FDA) adoption of the AI tool Elsa has revealed critical reliability issues. According to multiple FDA employees and a CNN investigation, Elsa fabricates studies and misrepresents research, raising concerns about the tool's accuracy in supporting regulatory decisions. While its deployment has been heralded as cost-effective, this AI hallucination phenomenon underscores the need for rigorous human oversight in drug approval workflows to safeguard public health.