Building Scalable Infrastructure for Delphi’s Epidemic Forecasting Tools

Delphi, a research group founded in 2012 at Carnegie Mellon University, focuses on advancing epidemic detection, tracking, and forecasting, using AI. Their flagship project, COVIDCast, which we collaborated on, aims to make epidemic forecasting as useful as weather forecasting.

Delphi is a leader in epidemic research, partnering with CDC, Google, Meta, and UnitedHealth Group/Optum. With an Epidata repository comprising over 500 real-time epidemic signals, they process more than 4 billion records, adding 3 million daily.

Some of Delphi’s notable achievements include:

• Multiple wins in CDC’s “Predict the Flu” challenge since 2014.
• Named a National Center of Excellence for Flu Forecasting (2019-2025).
• Recently designated a National Center for Innovation (2023-2028) by CDC.

Delphi’s primary mission is to make epidemic data widely accessible for public health agencies, the healthcare industry researchers, and the general public. To achieve this, they provide tools like the COVIDCast platform and the Signal Discovery App, which aggregates and filters multiple data sources. They develop ML and statistical models for nowcasting and forecasting infectious diseases. AI-powered anomaly detection algorithms identify unusual patterns or changes in epidemic signals, and trend detection models help public health agencies follow disease spread trajectories.

Delphi required real-time, reliable systems capable of aggregating, processing, and forecasting complex epidemiological data. With over 3 million new records added daily, maintaining data integrity, ensuring secure API access, and scaling the infrastructure posed significant technical challenges. Delphi also needed intuitive interfaces to serve multiple stakeholders—from public health agencies to data journalists.

StartupSoft joined the COVIDCast project in 2022 to help Delphi enhance its epidemic forecasting capabilities. Our collaboration encompassed frontend and backend development, API security improvements, and workflow automation. We also supported the Signal Discovery App, a key tool that aggregates and filters epidemic signals for users.

• Frontend.
  We used Svelte, HTML, and CSS to build a responsive, user-friendly interface.
• Backend.
  Powered by Python, Django, and Flask, we delivered a robust and maintainable backend, ensuring the data remained accessible and accurate.
• Database.
  Our team optimized data handling using MySQL (MariaDB) to manage large-scale, real-time data streams.
• API Integration & Security.
  We introduced API keys to monitor usage and control access, ensuring secure endpoints for authorized users only.
• Workflow Automation.
  Automation of Slack and GitHub scrapers streamlined communication and issue tracking. Additionally, Jira automation reduced manual efforts and improved operational efficiency.

Through our collaboration, Delphi achieved:

• Seamless data management by processing over 4 billion records and adding millions daily, ensuring the information remains up-to-date and accessible.
• Better user experience through a responsive interface designed for public health agencies, researchers, and the general public.
• Improved API security and usability with API keys, enabling controlled access and secure data handling for authorized users.
• Increased operational efficiency through Jira automations and data scraping tools, reducing manual tasks and enabling faster responses and smoother workflows.

We continue working with Delphi to expand their data platform through the Signal Discovery App, which consolidates signals from various providers in one place. This app helps public health professionals to filter data by source, pathogen, or region and integrate insights into COVIDCast for further analysis.