Google 

Senior Software Engineer - Tech Lead

• Tech lead for the Backup/Restore team in Google cloud responsible for ensuring Google cloud’s data is safely backed up, resilient towards data integrity failures and ready for a quick recovery. The project includes building policies, requirements and tools across Google. 

• Lead the end to end project to replace the existing backend (pull based scheduling structure) for the restore tools with a new backend (push based scheduling framework) to increase reliability and reduce maintenance. 

  • Designed the new backend with estimated milestones and detailed work items. 

  • The new back end removed the race condition and provided a strong guarantee of at least one execution per task. 

  • Used Agile methodology and bi-weekly sprint to assign task items to junior engineers and track project progress.

  • Successfully migrated existing clients to the new backend without requiring any code changes or procedure changes. 

• Increased Google internal databases' resilience to data integrity failures by successfully developing and deploying an automated system for measuring data integrity SLOs. 

  • Developed a dashboard to display time to restore a database and amount of data loss during an data outage for all Google Spanner databases. The dashboard helped many teams to prioritize their efforts on data resilience and reduced mean time to recover services.  

  • Designed and built an automation pipeline to create a data integrity readiness monitoring dashboard for Google internal databases.

• Hosted bi-weekly tech talk to share knowledge and recognize team achievements. It helped create a dedicated space for team members to share their expertise and increased team morale.  

• Coached and mentored new engineers on the team and across Google cloud organization. Provided guidance and support to help them learn the ropes and succeed in their roles.

• Prioritized feature requests and bugs during bug scrub meetings. Successfully ensured that the team was able to focus on the most important tasks.

• Contributed to the development, reporting, and tracking of team-specific OKRs (Objectives and Key Results). Collected and analyzed data to track progress against OKRs.

Software Engineer III

• Received 2022 Google Cloud Tech Impact Award for my work on GCE (Google Compute Engine - VMs) disaster recovery. Lead a project to build a highly available, scalable, low dependency and sharded restore tool. The tool significantly simplified complex restore procedures and allowed teams to build custom restore solutions quickly.  

• Lead the project to shard restore service across all Google cloud regions. Achieved four nine of availability (99.99%) with N+1 redundancy in each region and flexibility to scale horizontally and vertically. Significantly limited possibility of a global service outage.

• Successfully integrated two Google internal restore testing frameworks. Designed and launched an integrated version in the development environment. 

• Built an automated release qualification testing system that automatically tests core use cases on each new version of the binary. The system automatically prevents the rollout of bad binaries and notifies on-call engineers for further debugging.

• Participated in a 24/7 on-call rotation for data disaster related outages in Google Cloud. Handled emergencies, answered questions, replied to emails, and handled failures in release pipelines. Resolved issues quickly and efficiently, minimizing the impact on customers.

Software Engineer II

• Designed and implemented a CLI (Command-Line Interface) tool for restoring database backups during an emergency. The CLI tool was automatically deployed to all Google-issued devices, including laptops, production jobs, and virtual instances. This saved time and effort during an emergency. The release process included creating a staging instance, automated functionality testing, and promotion to production if the tests were successful.

• Designed and built client libraries in Java and C++ to connect to Google's internal restore tool using RPC (Remote Procedure Call). The libraries were designed to simplify interaction with the operational API by providing simpler methods, asynchronous function procedures, and status checking functionality. This project helped to improve the efficiency and effectiveness of the restore process by making it easier for engineers to interact with the API.

• Documented best practices, examples, and a user guide for using the developed client libraries and CLI tool for restoring database backups. Wrote playbooks to handle emergency operations related to CLI outages and debugging.

• Collaborated with the data protection and core infrastructure teams to design and implement MPA (Multi-Party Authorization) for emergency restore operations. MPA requires another engineer to approve the command before it can be executed, which adds an additional layer of protection against human error and malicious intent..

• Built an API server to receive database restore operation requests and enqueue new work items inside the scheduler server. The API server included authentication and authorization of users, early failure detection, load balancing, and N+1 redundancy to achieve high reliability. This project improved the security, reliability, and efficiency of the database restore process. It also made it easier for engineers to request and track database restores.

Software Engineer II - EngRes

• Implemented and launched Explicit Locking feature for Google’s internal Java ORM (Object Relational Mapping) framework. The feature was designed for the advanced user to provide flexibility of locking specific database cells for a given transaction to improve performance by reducing the number of locks. 

• Designed and completed migration of NitroML framework’s benchmark storage from online file to MLMD (ML Metadata) which reduced the load time of machine learning model evaluations from 15 minutes to 30 seconds. 

• Successfully expanded capability of NitroML (www.github.com/google/nitroml) framework to store ML pipeline evaluations to multiple data sources like SQlLite, Google Cloud SQL, and cloud Filestore.