A new analysis argues that AI systems can fail in ways that are not captured by aggregate benchmark accuracy, and that hidden risks appear in model behavior when deployed in real workflows. Separate work highlights that adversarial techniques and side-channel attacks can exploit neural systems, including a demonstration against post-quantum cipher BIKE through neural network side-channel leakage. The combined message is that clinical-grade AI reliability depends not only on predictive performance but also on security, privacy and robust evaluation methods that detect failure modes before patients are exposed. As AI tool adoption grows, regulators, health systems and vendors face pressure to align safety and security testing with real-world use, including auditability and resilience against misuse or unintended data leakage.