Our production data pipeline is already live on Azure, but the data-processing layer needs a careful upgrade. I ingest events through Kafka, land them in an Azure Data Lake, then run real-time transformations in Apache Flink. Everything is orchestrated by Airflow on a Kubernetes cluster, with CI/CD handled through our DevOps toolchain. I need a seasoned data engineer who can dive straight into the Flink jobs, refactor the Python code where necessary, tune state management and checkpointing, and release the changes through our existing Kubernetes-based workflow. You will also validate end-to-end data quality in the lake and leave the deployment scripts cleaner than you found them. Deliverables • Refactored Flink job(s) with improved throughput and lower latency • Updated Airflow DAGs reflecting the new logic and resource needs • Helm/K8s manifests and pipelines amended for zero-downtime rollout • Documentation outlining the changes and rollback steps Acceptance criteria • End-to-end tests show parity or better on data accuracy • Average processing latency reduced by at least 20 % under load tests • All code passes our automated CI checks and lints cleanly If you are fluent in Python, Azure services, Kafka, Flink, Airflow, Kubernetes and DevOps best practices, I’d like to start as soon as possible.