Proactive and results-driven Senior Data Engineer having over 4 years of experience in the field, with a focus on Python, SQL, AWS, Snowflake. Skilled in data modelling , building EDW and recognised for driving end-to-end project delivery. Passionate about applying skills to solve complex data engineering problems.
Led API-driven data integration pipelines to load data from a centralised data lake into Amazon OpenSearch and Amazon Redshift: Designed and developed data transformation jobs using AWS Glue with PySpark, optimising data for ingestion and query performance. Configured VPC Peering to securely connect OpenSearch hosted in a different AWS account to the data lake environment, ensuring seamless cross-account data integration. Developed automated workflows to load data into external portals as CSV files using AWS services: Utilised Lambda for API data retrieval, transformation to CSV, and storage in S3. Orchestrated periodic jobs with AWS Batch and EventBridge scheduler, with SNS notifications for success/failure. Deployed code pipelines via ECR and Jenkins, ensuring scalability and reliability.
Architected real-time streaming pipelines using AWS Gateway, Kinesis, and Glue, reducing processing time to under 1 minute for seamless data ingestion from Amazon S3 to Snowflake, improving operational efficiency. Loaded data from source systems to datalake using AWS DMS and Step Functions. Implemented Enterprise Data warehouse using the DataVault 2.0 methodology. Performed advanced analytics on structured and unstructured data using SQL, Python and DBT. Migrated data from Aurora MySQL (OLTP) to Snowflake (OLAP) Created SnowPipe to load data from SQS to Snowflake External Stage. Transitioned pipelines from Talend to AWS Glue and Apache Airflow infrastructure
Developed software applications to meet customer requirements. Payment Integration with PayPal. Conducted unit tests on developed code to ensure quality assurance standards are met. Collaborated with other developers and stakeholders on software projects. Created user documentation for newly developed software applications. Debugged existing code to identify potential problems or areas of improvement. Researched various techniques for increasing the performance of application programs. Served as a mentor for junior developers, providing guidance when needed.
REST API
AWS
Big Data
Cloud
SQL
ETL
Python
Data Validation
Linux
NOSQL
QWIRK is a brand owned by Gigart Solutions Incorporation, Delaware, USA. The company is a freelance marketplace to help firms find quality professionals when needed, to be a part of your workforce.
11 Seasons Glen Dr, Morris Plains New Jersey 07950 United States of America
Copyright 2023 QWIRK, All Rights Reserved.
