Sathvika Kolisetty - Data Engineering Portfolio

Projects

Scalable Clickstream Data Pipeline

Overview: Built a robust pipeline for ingesting and analyzing massive volumes of user behavior events in real-time.

Technologies: Kafka, Spark Structured Streaming, Delta Lake, Apache Hudi, Hive, Trino, dbt, Airflow

Impact: Enabled unified batch/stream access, 50% faster data discovery, and increased pipeline reliability with dbt tests.

Learn More

Healthcare Data Lakehouse with Real-Time Analytics

Overview: Developed a HIPAA-compliant Azure data lakehouse pipeline for real-time analytics on healthcare claims.

Technologies: Azure Data Factory, Delta Lake, Synapse, Power BI, Azure ML, Terraform

Impact: Achieved 87% predictive accuracy for readmissions and reduced dashboard latency by 40%.

Learn More

Cross-Cloud GenAI Recommendation Engine

Overview: Built a real-time recommendation engine using AWS for ingestion and GCP for AI modeling.

Technologies: AWS Kinesis, AWS Glue, Lambda, Snowflake, Vertex AI, FastAPI, Streamlit

Impact: Improved recommendation accuracy by 30% and cut compute costs by 25% using optimized ELT.

Learn More

Real-Time Streaming Pipeline

Overview: Engineered a high-performance streaming system for real-time analytics.

Technologies: AWS Kinesis, Kafka, Flink, Lambda, FastAPI, QuickSight

Impact: Reduced latency by 60%, improved decision-making by 30%, and enabled real-time APIs.

Learn More

Cloud Data Lake & Migration

Overview: Led a secure, petabyte-scale data lake migration to AWS.

Technologies: AWS S3, Glue, Lake Formation, Snowflake, DMS, Redshift

Impact: Achieved zero-downtime migration, 45% faster queries, and 25% cost reduction.

Learn More

GCP Data Ecosystem & AI Analytics

Overview: Designed a cost-efficient GCP pipeline with AI-driven insights.

Technologies: GCP Dataflow, BigQuery, Vertex AI, Pub/Sub, Looker, Cloud Composer

Impact: Cut processing times by 40%, reduced costs by 30%, and boosted forecast accuracy by 20%.

Learn More

Real-Time Streaming Pipeline

Overview: For a financial services company, I designed and implemented a real-time streaming pipeline to ingest, process, and analyze transaction data with minimal latency, enabling near-instant analytics and actionable insights for business teams.

Technologies Used: AWS Kinesis was used for ingesting streaming data from multiple sources, Apache Kafka for reliable message queuing, and Apache Flink for event-driven processing. I developed FastAPI-based microservices, deployed via AWS API Gateway, to enable real-time data access, handling over 1 million records per day with sub-second response times. AWS Lambda functions facilitated serverless event handling, such as triggering alerts for specific transaction patterns. AWS QuickSight was used to build interactive dashboards for visualizing transaction trends and anomalies. Monitoring and alerting were configured using AWS CloudWatch, Prometheus, and Grafana to ensure system reliability.

Challenges: Ensuring data consistency across distributed systems was a key challenge, as Kafka and Kinesis occasionally faced message duplication. I implemented deduplication logic in Flink and used Kafka’s idempotent producer to mitigate this. High throughput also increased AWS expenses, so I introduced Auto Scaling for Lambda and Kinesis, along with S3 lifecycle policies for archival, reducing costs by 20%.

Impact: The pipeline reduced data processing latency by 60%, from 5 seconds to under 2 seconds, enabling near-instant analytics. The QuickSight dashboards improved decision-making speed by 30%. The FastAPI microservices provided real-time data access to downstream applications, handling 1M+ records daily with 40% faster response times. Cost optimization efforts saved 20% on cloud expenses, and monitoring reduced incident response time by 40%.

Cloud Data Lake & Migration

Overview: For a global consulting firm, I led the migration of petabyte-scale datasets from on-premises databases to a centralized AWS data lake, enabling advanced analytics and ensuring compliance with security standards.

Technologies Used: AWS S3 served as the storage layer for the data lake, with AWS Glue handling ETL processes and schema evolution via the AWS Glue Data Catalog. AWS Lake Formation was used for governance and access control. AWS DMS (Database Migration Service) facilitated the migration of data from on-premises Oracle and SQL Server databases to S3 and Snowflake, using CDC (Change Data Capture) for incremental syncs. Snowflake and Redshift were optimized for analytical workloads with materialized views, partitioning, and result caching. AWS IAM and KMS enforced fine-grained access control and encryption for SOC 2 compliance. Apache Airflow automated ETL workflows, and monitoring was set up with AWS CloudWatch and Datadog.

Challenges: Ensuring zero downtime during migration required careful planning, using DMS with CDC to sync data incrementally and automated scripts for consistency validation. Initial query costs on Redshift and Snowflake were high, so I fine-tuned queries with partitioning and materialized views, reducing compute costs by 25%. Strict access controls for SOC 2 compliance were implemented using IAM roles and KMS encryption.

Impact: The migration was completed with zero downtime, ensuring uninterrupted operations. Query performance improved by 45% due to optimization in Redshift and Snowflake. The centralized data lake enabled advanced analytics, improving reporting efficiency by 50%. Cost savings of 25% were achieved through query optimization and lifecycle policies. Automated ETL workflows with Airflow and Glue reduced manual efforts by 90%.

GCP Data Ecosystem & AI Analytics

Overview: For a technology client, I designed a cost-efficient data ecosystem on GCP to enable real-time analytics and AI-driven forecasting, providing actionable insights for business planning.

Technologies Used: GCP Dataflow handled scalable data processing, ingesting data via GCP Pub/Sub for event-driven streaming. BigQuery served as the data warehouse, optimized for large-scale analytics. Vertex AI was integrated to build and deploy machine learning models for predictive analytics, focusing on demand forecasting. Cloud Composer orchestrated and automated the pipeline, ensuring reliability. Looker dashboards visualized real-time insights and forecasting results. Cost-saving measures included preemptible VMs in Dataflow and BigQuery’s flat-rate pricing. GCP Cloud Build was used for CI/CD, and monitoring was set up with Cloud Logging.

Challenges: Integrating Vertex AI required extensive data preprocessing, which I handled using Dataflow to clean and transform data. Dataflow and BigQuery expenses initially spiked, so I optimized with preemptible VMs and flat-rate pricing, reducing costs by 30%. Pipeline reliability was ensured with Cloud Composer automation and Cloud Logging, reducing failures by 40%.

Impact: The pipeline reduced data processing times by 40%, from over 10 minutes to under 6 minutes, enabling real-time analytics. Vertex AI models improved forecasting accuracy by 20%. Looker dashboards enhanced decision-making efficiency. Cost savings of 30% were achieved, and the automated pipeline reduced manual efforts by 50%.

Scalable Clickstream Data Pipeline

Overview: Built for a retail analytics firm, I developed a real-time data pipeline that processes over 5 million clickstream events daily. The system supports both batch and streaming access by leveraging Delta Lake and Apache Hudi, allowing teams to run historical and real-time queries simultaneously.

Technologies Used: Apache Kafka was used for ingestion, Apache Spark (Structured Streaming) for real-time processing, and AWS S3 for storage. Data was stored in both Delta and Hudi formats to support hybrid access. Hive Metastore integration enabled schema tracking, and Trino provided fast SQL-based access. dbt handled data quality and schema testing, orchestrated through Airflow DAGs on Databricks.

Challenges: Managing schema evolution and consistency across formats was challenging. I implemented strict versioning and partition strategies and automated schema validation using dbt. Scaling Spark jobs to handle peak loads required tuning shuffle partitions and memory configurations.

Impact: Reduced data discovery time by 50% through Trino+Hive integration. Increased data reliability via Airflow-monitored dbt tests. Enabled analytics teams to perform ad-hoc queries with latency under 3 seconds, improving insights turnaround.

Healthcare Data Lakehouse with Real-Time Analytics

Overview: For a healthcare analytics provider, I built a HIPAA-compliant Azure lakehouse pipeline that ingests and transforms patient claims data for real-time reporting and predictive modeling.

Technologies Used: Azure Data Factory handled ingestion, Delta Lake stored transformed data, and Synapse served as the analytics engine. Predictive models were trained in Azure ML to detect high-risk readmissions. Power BI dashboards offered interactive visualizations. All resources were deployed and secured using Terraform.

Challenges: Ensuring HIPAA compliance required encryption, access control, and auditing. I implemented row-level security in Power BI and ensured end-to-end encryption in transit and at rest. Real-time transformation logic was optimized using Delta's schema evolution and merge capabilities.

Impact: Improved analytics speed by 40%, reduced dashboard load times by 35%, and achieved 87% model accuracy. Enabled business teams to identify at-risk patients proactively, reducing readmission costs.

Cross-Cloud GenAI Recommendation Engine

Overview: I developed a real-time recommendation engine leveraging AWS for ingestion and GCP for AI modeling. The system ingests behavior events via AWS Kinesis, transforms them using AWS Glue, and stores in Snowflake for centralized querying.

Technologies Used: The inference engine was built using GCP Vertex AI, where behavioral embeddings were modeled using GenAI. FastAPI served the recommendation endpoint, and a Streamlit frontend was deployed for live demos. The system used partitioned Snowflake tables and materialized views to improve ELT performance.

Challenges: Cross-cloud communication latency was optimized using asynchronous FastAPI calls and minimal payloads. Cost spikes in Snowflake were addressed by implementing clustering and query pruning strategies.

Impact: Achieved 30% higher recommendation accuracy by using real-time GenAI features. Streamlined data pipelines reduced Snowflake compute costs by 25%. The live demo environment supported 1000+ concurrent requests with stable performance.

<Sathvika>

Get to Know Sathvika

Projects

Scalable Clickstream Data Pipeline

Healthcare Data Lakehouse with Real-Time Analytics

Cross-Cloud GenAI Recommendation Engine

Real-Time Streaming Pipeline

Cloud Data Lake & Migration

GCP Data Ecosystem & AI Analytics

Skills

Data Processing & ETL

Databases & Warehousing

Cloud Ecosystem: AWS

Cloud Ecosystem: GCP

Programming & DevOps

Data Visualization

AI & Machine Learning

Big Data Technologies

APIs & Web Frameworks

Experience

Cloud Data Engineer - Swift

Data Engineer - WTW

Certifications

AWS Certified Data Engineer – Associate

AWS Certified Solutions Architect – Professional

Databricks Certified Data Engineer Professional

Contact Me