Senior Data Engineer
StartupSoft connects top Ukrainian engineers with world-class startups from Silicon Valley and EU. Our developers work directly on the product as an integral part of the startup team.
About project
Delphi is a research group based out of Carnegie Mellon University dedicated to developing the theory and practice of epidemic tracking and forecasting. Pre-pandemic they worked mostly on influenza, dengue and norovirus; they’ve now picked up a focus on COVID. They procure unique data streams that reflect epidemic (or pandemic) activity, extract relevant indicators, and make these publicly and continuously available. They and others then use these indicators for nowcasting (situational awareness) and short-term forecasting.
The main goal of Delphi Research Group improving the world’s ability to deal with the next pandemic.
Tech stack
MySQL, Python
Team composition
9 full-time engineers (1 DevOps, 1 database, 2 pipeline/statistics/masters, sys admin) + graduate students and professors. The team is mostly in EST
Project stage
Live product
Qualifications and skills
- Proficiency in data pipeline tools
- Experience with Python and SQL
- Strong understanding of data structures, algorithms, and system design
- Familiarity with data storage solutions (e.g., databases, data lakes)
- Excellent problem-solving skills and attention to detail
- Effective communication skills, both synchronous and asynchronous
Responsibilities
- Monitor and maintain the health and efficiency of data pipelines
- Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues
- Communicate with data providers to understand data discrepancies and manage changes in data delivery
- Implement fixes and enhancements to improve data quality and pipeline performance
- Collaborate with data scientists and analysts to understand data needs and implement effective data solutions
- Develop strategies for data validation and quality assurance
- Optimize data flow and collection to improve system efficiency
- Document and manage data pipeline architectures, including maintenance and update protocols
- Use tools such as SQL, version control and CI/CD, containerization, task schedulers, python frameworks, and cloud services for data pipeline management
- Ensure compliance with data governance and security standards
Why StartupSoft?
- Work with top entrepreneurs and startups from Silicon Valley and EU
- Become an integral part of the core team — be more than someone who just writes code
- Potential to receive equity in high-growth startups
- Contribute to well-funded products with potential to impact millions
- Beautiful offices in the best locations with high-end amenities
- Tight-knit team with an inclusive and transparent culture
- Only the latest, modern technologies with no legacy code and management bureaucracy
- Above average salary, benefits & medical insurance
