Pooja

Developed data pipeline using Flume, Sqoop, Pig and Java map reduce and Spark to ingest customer behavioral data and purchase histories into HDFS for analysis, connected to AWS Redshift through Tableau to extract live data for real time analysis, responsible for creating on-demand tables on S3 files using Lambda Functions and AWS Glue using Python and PySpark, and developed ETL framework using Spark and Hive, including daily runs, error handling, and logging.

Extracted transform and load data from sources systems to Azure Data Storage services using a combination of Azure Data Factory, Informatica BDM, T-SQL, Spark SQL, and Azure Data Lake Analytics. Data Ingestion to one or more Azure services and processing data in Azure Databricks, implemented ETL process through Informatica BDM and Python scripting to load data from Denodo to ThoughtSpot, built CI/CD pipeline, and developed features, scenarios, step definitions for BDD and TDD using Cucumber, Gherkin, and Ruby.

Responsible for architecting Hadoop clusters and analysis, migrated Hive UDF’s and queries into Spark SQL for faster requests, configured Spark Streaming to receive real time data from Apache Kafka, stored stream data to HDFS using Scala, hands-on experience in Spark and Spark Streaming creating RDD’s, applying operations, and developed multiple POCs using Scala.