Job description
Responsibilities for Data Engineer
- Create and maintain optimal data pipeline architecture,
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SSIS or ADF technologies.
- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
- Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
- Work with data and analytics experts to strive for greater functionality in our data systems.
Job Skills:
-+ Having strong Python scripting experience
-+ Extensive experience connecting to various data sources and structures: APIs, NoSQL, RDBMS, Blob Storage, Data Lake, etc.
-+ Deep understanding of ETL, ELT, data ingestion/cleansing and engineering skills
-+ Having experience building data ingestion pipelines using Azure Data Factory and SSIS to ingest structured and unstructured data with C# programming experience.
-+ Having strong knowledge on Azure Storage schematics such as Gen1 and Gen2
-+ Have experience on CDC tools like Transaction Replication, Debezium or HVR.
-+ Having knowledge of Azure networking, security, key vaults, etc.
-+ Advanced Analytics including Azure Data Bricks, visualization tools as PowerBI, Tableau
-+ Have experience in Data Governance, Data Catalog, Master Data Management
-+ Big Data implementation - using Open Source and Non-SQL technologies such as Databricks, Spark, Spark Streaming, Kafka, Cosmos DB, Snowflake and Python.
-+ Knowledge of Lambda and Kappa architecture patterns.
-+ Knowledge of Master Data Management (MDM) and Data Quality tools and processes.
-+ Experience with messaging/eventing architecture concepts and technology such as, Kafka, Event Hubs, Streams
-+ Strong analytic skills related to working with unstructured datasets.
-+ Build processes supporting data transformation, data structures, metadata, dependency and workload management.
-+ A successful history of manipulating, processing and extracting value from large disconnected datasets.
Compensation
- Competitive salary Total CTC (18,00,000 to 22,00,000) Per Annum
- Training
- Opportunity for advancement
- Fun and exciting environment
Soft Skills:
Ability to understand and adhere to Information security requirements of the organizations and the specific responsibility assigned.