Hi, I'm Pushpneet Singh.

Data E

Professional Experience

CARS24 | SDE-2 Data Engineer

Jul 2024 - Present

  • Designed and developed a no-code middleware tool for configuring and automating ETL pipelines from sources like Emails, APIs, SAP, PubSub, and Databases.
  • Integrated SAP S/4HANA accounting covering 97% of company accounting volume for real-time financial reporting.
  • Integrated third-party APIs to dynamically generate and trigger Airflow DAGs for seamless data workflows.
  • Used Airflow, Spanner, GCS, GKS, FastAPI, Temporal, and Snowflake to streamline infrastructure.
  • Worked with finance to automate manual tasks including reconciliation via 3-way dashboards on Superset.

Sanius Health | Data Engineer Consultant

Jan 2024 – Jul 2024

  • Wrote Python scripts from scratch to ingest data from different wearable sources and devices to a data warehouse, reducing total processing time from 24 hours to 2 hours.
  • Utilized Apache Kafka, Apache Druid, and Apache Superset to generate customized health reports in real time.
  • Built trigger-based data ingestion pipelines using Azure Functions for automated health device data processing.

SDE-2 Data

Aug 2021 - Dec 2023

  • Worked to migrate a complex on-premise contact center solution for Commonwealth Bank of Australia, involving 100+ tables, to a cloud environment.
  • Developed robust ETL pipelines to process data from 5000+ agents, handling 6M+ inbound calls and 1.5M+ digital chats per month using Hadoop, Hive, SQL, Spark, Zeppelin, and Abinitio.
  • Automated stream job monitoring with scripts, reducing manual effort by 80%.
  • Optimized and maintained pipelines to enhance daily processing efficiency by 75%.

Projects

Neural Style Transfer
Neural Style Transfer

Used pretrained NN to impose artistic styles onto images.

GitHub
State Fiscal Data Explorer
State Fiscal Data Explorer

ETL tool for scraping and analyzing open fiscal data.

GitHub
Fake News Detection
Fake News Detection

Used NLP + ML models to classify news as fake/real.

GitHub
Corona Tracker
Corona Tracker

Vue + ApexCharts app visualizing global COVID data.

Live
Recruitment Bot
Recruitment Bot

Job-seeking assistant built using RASA framework.

GitHub
Connect With Me
© made with by Pushpneet Singh