Job description
Senior Azure Synapse Data Engineer
Remote
£90,000 - £115,000
Sponsorship Cannot Be Provided
ROLE OVERVIEW
We are looking for a Senior Azure Synapse Data Engineer with exceptional technical, analytical, communication and project management skills to join our team of data and analytics experts, helping organize and leverage arguably the richest, broadest data available on work and people, and drive the build-out of a scalable, distributed data estate and supporting applications.
ABOUT THIS ROLE
The Senior Azure Synapse Data Engineer will support our move to a modern data architecture and build-out of our data estate. Core activities include automating data ingestion from internal and external tools/platforms; identifying and linking individuals across different datasets/platforms for further analytics; creating sophisticated ETL processes to clean and organize our data using Azure Synapse pipelines, and providing Business Intelligence to establish the breadth, depth, and quality of the data.
This data is utilized by our team of data scientists to perform analytics and develop new research, IP, and thought leadership. Our ability to have clean, well-organized data is fundamental to this effort.
RESPONSIBILITIES
Architecture design. Design, develop and maintain the architecture of the Azure data estate (data lake, SQL Analytics store, pipelines, ML/AI environment and network) to meet business requirements.
Data acquisition and integration: ensure feeds from various platforms - primarily internal but increasingly external are established. Use programming skills to develop, customize and manage ingestion tools, storage accounts, and analytical applications and their access.
Perform ETL processes for optimal extraction, transformation, and loading of data from a wide variety of data sources using Azure Synapse tools and technologies.
Data pipeline development/testing. Test the reliability and performance of each part of a system during the development phase. Identify, design, and implement process improvements: automating manual processes, optimizing data delivery, and re-designing infrastructure for greater scalability and data quality.
Track pipeline stability. Monitor the overall performance and stability of the system and ensure automated parts/scheduled tasks are monitored.
Manage structured and unstructured data and meta-data stored in the data estate. Ensure availability of clean, transformed data to meet business needs.
Machine learning algorithm deployment. Deploy Machine learning models (designed by data scientists) into production environments and help manage these environments.
Partner with IT to provide Business Intelligence tools/infrastructure. Set up tools, views, and access permissions for different user groups enabling them to view data, generate reports, and create visuals. Work with data scientists to strive for greater functionality and use of our data.
Data strategy. Contribute to the broader organizational data strategy, including data privacy and governance practices. Develop strong relationships with key stakeholders across the business and help determine critical objectives.
SKILLS/QUALIFICATIONS
- Will likely have a degree in Computer Science, Statistics, Informatics, Information Systems or quantitative field.
Critical skills/experience:
- Microsoft Azure Synapse experience is essential.
- End-to-end Data Warehouse experience: ingestion, ETL, 'big data' pipelines, data architecture, message queuing, stream processing, BI/Reporting, and Data Security.
- Ability to build processes supporting data transformation, data structures, metadata, dependency, and workload management, as well as the ability to manipulate, process, and extract value from large, disconnected structured and unstructured datasets.
- Advanced SQL/relational database knowledge (including SSIS), query authoring (SQL).
- Knowledge of SQL Server Always Encrypted.
Supporting skills:
- Understanding of machine learning and artificial intelligence ML libraries and frameworks (TensorFlow, Spark, etc).
- Experience with Data Governance (Quality, Lineage, Data dictionary, and Security).
