Dave O

data engineer

  • 37 Views

$20.00

(per hour)

Description

As an experienced Data Engineer , I excel in building robust data pipelines on AWS Cloud, enhancing business performance reporting, and developing cutting-edge machine learning products using build & buy strategy according to business challenges and project requirement. Core Competencies: Data Pipelines: Expertise in building and optimizing data pipelines on AWS Cloud. Machine Learning: Development and deployment of machine learning products to solve complex business challenges. Data Solutions: Delivered high-quality solutions in Data Science, Artificial Intelligence, and Data Engineering. Technical Proficiencies: ETL Technologies: AWS Glue Service dbt Databricks Programming & Scripting: Python SQL Infrastructure as Code: -> Terraform -> CloudFormation Version Control & Collaboration: -> GitHub Big Data Processing: -> PySpark


Experience

Data Engineer

  • Integrated software Developers Group
  • October 1, 2025 - September 2, 2024

Led the end-to-end migration of legacy reporting systems to AWS, improving data accessibility and reducing operational costs. Automated ETL processes using AWS Glue, Python, and SQL, enhancing data integration and reducing manual workloads. Implemented data lakes and warehouses for seamless aggregation and analysis of performance data. Collaborated with cross-functional teams to ensure smooth transition and alignment with business objectives. Driven by a passion for cloud technologies and data engineering, I continuously innovate to enhance data-driven decision-making across the organization

Data Engineer

  • Giacom
  • April 17, 2023 - June 30, 2024

Developed data collection pipeline to efficiently gather and process data. Developed CICD pipeline on GitHub Action to automate deployment process across environment Utilized IAC (Infrastructure as Code) Terraform to develop and deploy AWS cloud resources. Created Crawler Jobs to automate data discovery and ingestion into the AWS Glue Data Catalog. Developed Glue Jobs to transform and manipulate data within the AWS Glue environment. Implemented Lambda functions to automate file processing in Amazon S3. Created Lambda functions to organize files into the correct folder structure in Amazon S3. Developed Lambda functions to update CSV table properties and schema in AWS Glue. Ensured data accuracy and integrity within Athena.Developed workflow orchestration to automates ETL process, Created Terraform module. Collaborates with other engineers and data analysts to maintain data integrity and consistency. Work extensively with AWS resources such as S3, Athena, Glue, Crawler, Database Catalog, CloudWatch, EventBridge, S3 event notifications, IAM, and Lake Formation permissions, workflow orchestration. Utilized GitHub Actions for version control and CI/CD workflows development to automate deployment across branches and origin

Data Analyst

  • MTR Group
  • October 3, 2022 - March 1, 2023

Developed sales report using Tableau reporting tool to convey Partner's sales activity to stakeholders

Python Developer

  • Integrated software Developers Group
  • January 5, 2020 - August 7, 2021

Collaborated with cross-functional teams to design and develop scalable python-based applications and solutions. Implemented best practices for code quality, performance, and maintainability, following agile development methodologies.utilized python frameworks such as django and flask to build robust web applications, delivering on time and within scope. Conducted code reviews and provided constructive feedback to team members to improve code quality and ensure adherence to coding standards.worked on database design, optimization, and management, including sql and nosql databases. Participated in the entire software development lifecycle, from requirements gathering to deployment and maintenance.troubleshooted and resolved complex technical issues, ensuring seamless operation of applications. Wrote clear and concise technical documentation to facilitate knowledge sharing among team members. Successfully delivered a mission-critical project, reducing processing time by 30% through optimized Python code. Implemented automated testing, resulting in a 20% decrease in software bugs and improved overall product quality.

Educational Details

Msc Artificial Intelligence and data science

  • University of hull

Report Freelancer

Ready To Get Started

QWIRK is a brand owned by Gigart Solutions Incorporation, Delaware, USA. The company is a freelance marketplace to help firms find quality professionals when needed, to be a part of your workforce.