Data Engineer


Summary:

The Data Engineer is responsible for engineering, developing, implementing, and operating stable, scalable, low cost solutions to source data from client systems into the data lake, data warehouse and end-user facing BI applications. The Data Engineer will be responsible for ingestion, transformation and integration of data to provide a platform that supports data analysis, data enrichment and data science as well as making data operationally available for analysis.

Candidate will work with architects, infrastructure team, data analysts, and data insight developers in a fast-paced environment, understanding business requirements, and implementing data integration and calibration across various data sources. Focus should be on reliability, efficiency and quality of data assets and processing/integration pipelines.

Responsibilities:

  • Engineer, build and maintain scalable automated data pipelines. Support critical data pipelines with a highly scalable distributed architecture - including data ingestion (streaming, events and batch), data integration, data curation.
  • Ability to build cloud data solutions and provide domain perspective on storage, big data platform services, serverless architecture, RDBMS, DW/DM, NoSQL databases and security.
  • Deploying, automating, maintaining and managing Azure cloud-based production system, to ensure the availability, performance, scalability and security of productions systems.
  • Good architectural understanding to build and ensure customer success when building new solutions and migrating existing data applications on Azure platform.
  • Conduct full technical discovery, identifying pain points, business and technical requirements, “as is” and “to be” scenarios.
  • Design and arrangement of scalable, highly attainable, and fault tolerant systems on Azure platform.
  • Ownership and responsibility for end-to-end design and development, testing, release of key components.
  • Understand and implement best practices in management of data, including master data, reference data, metadata, data quality and lineage.
  • Support regular ad-hoc data querying and analysis to understand customer behaviors.
  • Experience with code versioning tools and a command of configuration management concepts and tools, CI-CD including DevOps.
  • Other duties as assigned.

Requirements:

  • Bachelor’s Degree in Business or IT.
  • Hands-on knowledge of cloud-based data warehouse solutions like Azure, Snowflake and similar cloud-based product.
  • Experience with cloud architecture and solutions (MS Azure preferred).
  • Experience with cloud services, particularly Azure: Azure Data lake store, Blob Storage, VMs, Data Factory, SQL Data Warehouse, Azure Databricks, HDInsight, etc.
  • Experience with Oracle, Microsoft SQL Server Database system.
  • Moderate skill in business intelligence tools such as Power BI, Tableau.
  • Experience with scripting languages such as R, Python.
  • Data storytelling skills; combine data analysis and visualization to convey complex results in an understandable and impactful way.

To apply, please send your resume and cover letter to HR@trellance.com.