Data Engineer

Overall Purpose of the Job:

The data engineer is responsible for ingesting and transforming a wide range of data across the business into formats that can be used by end-users to develop differentiating business insights. By structuring data requirements into logical data constructs, the Data Engineer enables optimal, physical implementation of the data structure by generating data models. A key deliverable is the development of visual reports, dashboards and KPI scorecards.

The role will involve collaborating with internal and external resources to ensure the data analytics team have the required data from the source systems as well as be responsible for the accuracy of data loaded into the Data Warehouse.

Main Outputs and Responsibilities for this Position:

Data analysis / data modelling/ report building:

• Creates data model designs in accordance with recommended best practice.
• Collaborates with team members and business to translate business requirements into data requirements.
• Analyses and profiles the source data to understand data quality issues, relationships, patterns, and rules in the data.
• Structures data requirements into logical data constructs based on recommended best practice including ERD models and dimensional models to ensure optimal implementation.
• Compiles Source to Target Mapping Specifications with the Business Intelligence Business Analyst (BIBA).
• Ensures complete and correct data by identifying definitive or authoritative sources of data; analysing source data; and identifying gaps.
• Enables physical implementation of the data structure by generating the physical data model from the logical data model.
• Cleans and maintains databases by restructuring and deleting old data/stored procedures.
• Develops visual reports, dashboards and KPI scorecards using Power BI desktop.
• Connects to data sources, importing and transforming data for Business Intelligence.
• Implements row level security on data, demonstrating an understanding of application security layer.
• Writes DAX queries in Power BI.
• Develops Power BI reports as per the business requirements.

ETL and pipeline development:

• Assists in designing and implementing scalable and robust processes for ingesting and transforming complex datasets.
• Develops the work packages as per the design to deliver elements of the data warehouse architecture (data ingestion, ETL, etc.,).
• Provides technical support, troubleshooting and upgrades setup or support on ETL and data warehouse related issues.
• Analyses and maps data from source systems to target data warehouse.
• Participates in all aspects of quality assurance.
• Monitors and maintains ETL pipelines and data warehouse.
• Develops the data dictionary.

Task tracking and Support Resolution:

• Makes use of DevOps to track tasks, effort estimates, effort completed and repos.
• Responds timeously to support calls and tracks issues to completion.

Continuous improvement:

• Owns the development of elements of centralised, automated data improvement initiatives under the guidance of the Team Lead.
• Makes increased contributions by broadening and enhancing individual skills.
• Keeps abreast of industry trends and best practice and makes recommendations to management.
• Investigates new technologies to improve and optimize standard reporting.

Education:

• Diploma in IT or related field

Experience:

• 5 Years of experience as a SQL Developer
• 5 Years of experience as a BI Developer
• 3 Years of experience in Data Engineering
• 3 Years of experience with Synapse pipelines / Data factory / SSIS