Back to Case Studies

Real-Time Anomaly Detection for Patient Data Security

Real-Time Anomaly Detection for Patient Data Security

The Challenge

A healthcare AI startup needed to build a robust security monitoring system to protect sensitive electronic health records (EHR). Their core challenge was to analyze massive streams of user activity logs to proactively identify abnormal behavior—such as a hospital employee accessing an unusual number of patient records—that could indicate an internal threat or compromised account. They required a system that could detect both known and unknown patterns of suspicious activity in real time.

Our Solution

ActiveWizards designed and implemented a sophisticated anomaly detection platform that combined a scalable data architecture with a multi-layered machine learning approach.

Architecture for the Real-Time Anomaly Detection Platform

  • Scalable Data Architecture: We engineered a high-performance data pipeline to handle rapidly increasing log data. Raw text logs were processed and stored in Amazon Redshift for structured querying and in Google BigQuery (via Parquet files) for high-speed, ad-hoc analysis with Apache Spark.

  • Multi-Faceted Anomaly Detection: We developed a suite of machine learning models to identify suspicious activity from different angles. This included time-series analysis to spot deviations from normal user behavior, classification algorithms like Isolation Forest and One-Class SVM to flag outliers, and advanced clustering techniques to discover previously unknown patterns of abnormal behavior.

  • Novel Factorization Machine Models: To enhance detection accuracy, our team designed two unique approaches based on Factorization Machines. This allowed the system to model subtle interactions between users, data fields, and actions, identifying complex suspicious patterns that simpler models would miss.

Key Outcomes & Business Impact

  • Proactive Threat Detection: The platform automatically identified and alerted security officers to suspicious actions in near real-time, enabling rapid intervention.

  • High Accuracy with Low False Positives: Our multi-layered ML approach resulted in a highly accurate system with a very low rate of false negatives, ensuring that real threats were caught without overwhelming security staff with noise.

  • Enhanced Investigative Capabilities: The system provided interactive dashboards (built with Plotly and Matplotlib) that visualized user activity, allowing for deep investigation and analysis of security events.

  • Discovery of Unknown Threats: The unsupervised learning components successfully identified new, previously undefined patterns of suspicious behavior, hardening the startup's overall security posture.

Technology Stack

  • Core Languages & Libraries: Python, PySpark, Scikit-learn

  • Machine Learning: Libfm & Libffm (Factorization Machines), Time Series Analysis, Clustering

  • Data Storage & Warehousing: Amazon Redshift, Google BigQuery, Parquet

  • Visualization: Plotly, Matplotlib