Data Engineer with Microsoft Fabric
Gopi NathUnited StatesMIDPosted 5h ago
CI/CDgitautomated deploymentsPR checksrelease gatesisolationrollbackFabric lakehouseswarehousesdelta tablesshortcutsSQL analytics endpointsSparkFabric warehouse behaviorenterprise pipeline engineeringautomated pipelinesretriesincremental loadsrecoverytransparent failure statesPythondbt coredbt CLItestingmonitoringoperationalertscentralized logspipeline run trackingincident investigationanalytics engineeringreconciliationdeduplicationidempotencydata profilingschema contractsSpark performance engineeringDeltapartitioningconcurrencycapacity usageAzure devopsgithub actionsfabric CLIREST APIAzure identitysecurityownershipaccessfabric pipelinesservice principalslogic apps
About the Role
We are looking for a Data Engineer with Microsoft Fabric experience to join our team at Cognizant. This role involves working on a 12+ month project focused on data engineering and pipeline optimization.
Key Responsibilities
• Design and implement CI/CD pipelines with git-based development
• Build automated deployments with PR checks, release gates, isolation, and rollback mechanisms
• Work with Fabric lakehouses and warehouses, understanding delta tables, shortcuts, SQL analytics endpoints, Spark, and Fabric warehouse behavior
• Develop enterprise-level pipeline engineering solutions with automated pipelines, retries, incremental loads, recovery, and transparent failure states
• Develop, debug, and maintain Python code in Fabric ingestion pipelines and dbt deployment pipelines
• Implement monitoring and operation solutions including alerts, centralized logs, and pipeline run tracking
• Apply analytics engineering principles including reconciliation, deduplication, idempotency, data profiling, and schema contracts
Requirements
• Experience with CI/CD and git-based development
• Proven ability to build automated deployments with PR checks, release gates, isolation, and rollback
• Strong understanding of Fabric lakehouses, warehouses, delta tables, shortcuts, SQL analytics endpoints, Spark, and Fabric warehouse behavior
• Experience with enterprise pipeline engineering and automated pipelines
• Proficiency in Python development, debugging, and maintenance in Fabric ingestion pipelines and dbt deployment pipelines
• Understanding of dbt core and dbt CLI
• Experience with project work including testing and CI
• Ability to implement monitoring and operation solutions including alerts, centralized logs, and pipeline run tracking
• Experience with incident investigation
• Knowledge of analytics engineering principles including reconciliation, deduplication, idempotency, data profiling, and schema contracts
• Familiarity with Azure devops or github actions
• Experience with fabric CLI or REST API
• Understanding of Azure identity and security as it applies to fabric pipelines, service principals, and logic apps