Senior Data Quality Engineer

• Framework & Automated Pipeline Testing Data Quality Automation: Architect, implement, and extend automated testing frameworks (e.g., Great Expectations, Soda Core, or custom PySpark modules) within ETL/ELT pipelines.
• Data Reconciliation: Build distributed reconciliation jobs in PySpark to verify end-to-end data consistency across MongoDB, MySQL, Elasticsearch, and the Data Lake.
• Anomaly & Drift Detection: Implement real-time profiling to identify schema drift, unexpected NULL values, volume drops, and telemetry anomalies.
• 2.

Data Observability & Monitoring

• Alerting & SLAs: Define SLA/SLO metrics for data freshness, completeness, and accuracy; build real-time monitoring dashboards and alerting integrations.
• Root Cause Analysis (RCA): Lead technical investigations into pipeline failures, duplicate telemetry records, or corrupted time-series data, implementing permanent fixes.
• 3.

Engineering Standards & Governance Lineage & Documentation: Map data flows and maintain automated cataloging to ensure clear visibility from ingestion to analytics downstream.

• CI/CD Integration: Embed data validation gates into continuous integration/deployment (CI/CD) pipelines to prevent bad code or schema changes from hitting production.
• Cross-Team Collaboration: Partner closely with Data Engineering, Analytics, and IoT teams to align validation rules with business requirements.