Smart Crawlers for Infectious Disease Modelling: Automating Data Acquisition in the Digital Age

To address this, the study introduces Intelligent Interactive Persistent Crawlers (IIPCs), autonomous, cognitive agents designed to continuously collect relevant epidemiological data from global digital sources. These crawlers operate in both automated and human-assisted modes, allowing flexibility and resilience when encountering complex data access issues.

A key contribution is the development of two performance metrics: Data Acquisition Efficiency (DAqE), which measures how effectively required data is obtained, and Data Analytics Efficiency (DAnE), which assesses how efficiently the data is processed. Simulation results show that crawler-based systems significantly outperform traditional manual methods, with a hybrid human–crawler approach achieving the highest efficiency and robustness, even under degraded system conditions.

In real-world terms, this framework enables continuous, real-time data integration into disease models, supporting faster, more accurate decision-making for public health interventions such as vaccination strategies and treatment allocation. Overall, the study demonstrates that integrating intelligent crawlers into epidemiological workflows can substantially improve data pipelines and advance digital epidemiology for more responsive and effective disease control, elimination, and eradication.