Rishabh Gupta
Engineering SystemsFor The AI Era
Platform Engineering · SRE · Data Platforms · AI Infrastructure
Systems-minded engineering across platform, operations, cloud, data and AI infrastructure with an emphasis on evidence, architecture and production reality.
Profile, flagship systems, selected case studies, resume and verified contact paths.
Engineering Core · conceptual domains
Kubernetes · Docker · Podman · Nginx · Jenkins · Linux
Platform: Building and operating internal engineering platforms that support reliable application, data and operations workloads.
Profile
Building, Operating And Explaining Complex Systems
The portfolio is positioned around platform, reliability, cloud, data and AI infrastructure work rather than a generic full-stack narrative.
Technology and data platform engineer with 10+ years of experience designing, integrating and operating enterprise applications, cloud infrastructure, data platforms, AI-enabled solutions and production services.
Core areas
Current focus
- • Databricks architecture, Spark optimization and Delta workloads
- • Operational tooling for complex production workflows
- • Agentic AI, RAG and enterprise MCP-integrated support systems
- • Observability-driven engineering and incident analysis
- • System design tradeoffs across reliability, cost and speed
Principles
• Keep critical paths observable.
• Treat operations as a product surface.
• Prefer reversible architecture decisions.
• Separate production evidence from experiments.
Technical direction
AI Runtime Control Planes
How retrieval, tools, policies and observability can be composed into safe AI operating surfaces.
Data Observability
Making pipeline correctness and drift easier to reason about before downstream systems suffer.
Operational Product Design
Building internal tools that feel intentionally designed, fast and support-centered instead of purely administrative.
Operating posture
Experience Shaped Around Systems, Not Labels
The emphasis here is on the systems, domains and operational responsibilities that shape the work: platform ownership, data movement, observability, incident response and AI-adjacent infrastructure.
production
Platform And Reliability Operations
Technical ownership across the Databricks layer of the M3 platform, operational tooling, observability, Kafka integration, cloud automation and production reliability.
Responsibilities
- • Serve as technical owner for the Databricks layer across 20+ Spark and Delta workloads.
- • Translate business, operational, platform and data requirements into solution designs, data models and implementation plans.
- • Lead incident investigation and root-cause analysis across Databricks, Spark, Kafka, databases, AWS infrastructure and applications.
- • Provide technical leadership through design reviews, code reviews, prioritization, troubleshooting and mentoring.
Lessons
- • Operational interfaces deserve the same architecture discipline as customer-facing systems.
- • Reliability work is most effective when observability is built into the workflow itself.
production
Data Platform And Performance Engineering
Databricks, Spark, Delta Lake and MSSQL-oriented data engineering focused on throughput, validation, data quality and architectural tradeoffs in large-scale workloads.
Responsibilities
- • Reason about throughput bottlenecks and data skew.
- • Tune data writes, staging patterns and cluster sizing decisions.
- • Keep cost, correctness and operational simplicity in balance.
Lessons
- • Write-path design matters as much as compute-path design.
- • Large-scale data scenarios need observability at every stage of the pipeline.
research
AI Infrastructure And Engineering Lab Work
Exploring knowledge systems, local LLM tooling, embeddings, vector databases, MCP, RAG and agentic workflows as practical infrastructure problems connected to support and incident workflows.
Responsibilities
- • Evaluate local and hosted AI building blocks.
- • Design retrieval pipelines that stay inspectable and reproducible.
- • Treat AI systems as infrastructure with failure modes, not just demos.
Lessons
- • Retrieval quality depends on ingestion and evidence discipline, not model choice alone.
- • Agent systems need strong guardrails around tools, data access and observability.
production
Application Modernization And Infrastructure Migration
Modernization work across enterprise application and data platforms, including deployment automation, NGINX reverse-proxy design and infrastructure migration.
Responsibilities
- • Collaborate with business, application, infrastructure and deployment teams.
- • Translate requirements into implementation plans, ETL workflows and integration changes.
- • Lead migration-related coordination across application, infrastructure, security and release dependencies.
Lessons
- • Modernization succeeds when architecture and release dependencies are mapped explicitly.
production
SQL Server ETL And Reporting Engineering
SSIS ETL pipelines, SSRS reporting, SQL optimization and workload scheduling for enterprise data processing and reporting workloads.
Responsibilities
- • Build SQL Server SSIS ETL pipelines and SSRS reporting solutions.
- • Optimize SQL queries, stored procedures and indexes.
- • Implement workload-scheduler flows with dependency management and failure handling.
Lessons
- • ETL and reporting systems need the same operational discipline as applications.
production
Enterprise Application Support Engineering
Enterprise application support across backend development, deployments, integration support, defect resolution, SQL analysis and production troubleshooting.
Responsibilities
- • Support enterprise applications through backend development and deployments.
- • Investigate defects, integrations and SQL issues in production contexts.
Lessons
- • Production support builds the instincts needed for later reliability and platform work.
Compute
GPU
Storage
Monitoring
Operating model
The focus here is not a decorative career ladder. It is the progression from software delivery into systems thinking: runtime reliability, data movement, platform leverage, observability and AI-adjacent infrastructure.
System emphasis
Experience domains
Domain Expertise Mapped To Evidence
Domains are grouped by evidence and classification, not by self-scored skill percentages. Each one connects to real projects, case studies and technologies from the portfolio graph.
Selected domain
Platform Engineering
Building and operating internal engineering platforms that support reliable application, data and operations workloads.
Classification
PROFESSIONAL
Evidence
5
Related technologies
Related projects
• M3 OPS
• SAM
Related case studies
• M3 Operations Platform
Related experience
• Platform And Reliability Operations
Case studies
Flagship Systems And Engineering Cases
The portfolio moves from flagship systems into progressively deeper case studies, tradeoffs and evidence instead of long project writeups up front.
Flagship system
M3 Operations Platform
Operational platform for health monitoring, data search, workflows and production support paths across data, search and observability surfaces.
M3 · conceptual system model
Operational platform · Workflows · Support systems
Connected to: Data, Search, Observability, Operations
Selected node
M3 Operations Platform
Conceptual system model
Operational platform for health monitoring, data search, workflows and support operations.
Evidence
• Operational tooling
• Health monitoring
• Data search
• Workflow execution
• Incident-adjacent support
Data Platform Performance Scenario
A performance-engineering scenario centered on Databricks, Spark, staging design and MSSQL write patterns, including a scenario involving roughly 700M records.
System path · conceptual
AI Knowledge Base
An engineering-lab system for document ingestion, chunking, embeddings, vector retrieval and agent workflows using local and hosted AI tooling.
System path · conceptual
Observability Platform
A connected view of application signals flowing through search, metrics and tracing systems into alerting and ticketing workflows.
System path · conceptual
Kafka Event Streaming
Streaming-oriented thinking around Kafka, Kafka JDBC Sink and event-driven integration patterns where operational correctness matters as much as throughput.
System path · conceptual
Claims Highway
A data-integration and reporting-oriented experience area focused on SQL Server tooling and modernization paths for established enterprise workflows.
System path · conceptual
SAM
A modernization-oriented application surface where frontend, backend and architecture choices need to balance delivery speed with maintainability.
System path · conceptual
Certificate Management / MHDR
An operations and SRE-oriented problem space around certificate lifecycle visibility, automation and production safety.
System path · conceptual
Case study library
Progressive Disclosure For Systems Work
Summaries stay compact on the page. Each case study opens into targeted views for architecture, evidence, decisions and lessons.
production
7 evidence linksM3 Operations Platform
Operational tooling for workflows, health monitoring, search, analytics, data updates and incident-adjacent support paths.
research
5 evidence linksLarge-Scale Data Platform Performance Engineering
Databricks and Spark write-path engineering for large-scale MSSQL delivery scenarios, including observed runs around 700M records and long-running write windows near 200 minutes for roughly 550M records.
research
6 evidence linksLocal AI Knowledge System
A local-AI and retrieval-focused knowledge system spanning document ingestion, chunking, embeddings, vector search, retrieval and agent workflows.
production
7 evidence linksObservability As An Engineering System
Logs, metrics, traces, alerting and incident workflows treated as one evidence pipeline instead of disconnected tools.
production
4 evidence linksEvent-Driven Data Integration
Kafka-oriented integration thinking around producers, consumers, JDBC sink behavior, replay and failure handling.
production
4 evidence linksClaims Highway Modernization
Enterprise data movement and reporting work across SQL Server, SSIS and SSRS with modernization constraints.
production
4 evidence linksApplication Modernization And Architecture Cleanup
Angular and Node.js modernization work centered on delivery risk, architectural seams and operational clarity.
production
3 evidence linksCertificate Operations And Reliability
Certificate lifecycle visibility, monitoring and automation work framed as reliability engineering.
Engineering lab
Engineering Lab
Active experiments stay clearly separated from production work. The focus is on the question, the environment, the observed behavior and the next engineering move.
AI pipeline · engineering lab · conceptual system model
Vector store
Connected to: Embeddings, Retrieval
Engineering lab
Documents, chunking, embeddings, Qdrant, retrieval, LLM and MCP are shown as a controlled experiment path.
Embedding / Retrieval Experiment
Select Qdrant to inspect vector retrieval, clustering and tool-routed RAG flow.
Active / experimenting / learning
The lab focuses on AI infrastructure, data movement, MCP-based tool integration, observability patterns and operational product design. Every experiment is framed around system behavior, not hype.
Selected experiment
RAG Stack Comparisons
Question
Compare local and hosted retrieval setups without losing transparency.
Experiment
Evaluate chunking, embedding models and retriever behavior across a small knowledge-base workflow.
Result
The ingestion and chunking path proved more important than model switching alone.
Lesson
Retrieval quality is an engineering pipeline problem, not only an LLM choice.
Next
Add stronger citation and evaluation loops.
How I think
Architecture Means Tradeoffs Made Explicit
These are not universal rules. They are the questions and tensions I expect to resolve when designing platform, data and AI-adjacent systems.
Decision surface
Reliability vs Cost
Infrastructure and data systems often need to balance resilience against platform spend and operational complexity.
Tradeoffs
- • Extra redundancy increases safety but can hide inefficient design.
- • Aggressive cost cuts can create brittle operational states.
When to use
- • Increase reliability investment when the blast radius is large or recovery is slow.
- • Optimize cost only after the main failure modes are understood.
Risks
- • False savings from under-instrumented systems
- • Expensive architectures that still lack recovery discipline
Questions I ask
- • What fails first?
- • How quickly can it be diagnosed?
- • What is the cheapest safe shape?
Decision surface
Batch vs Streaming
Data systems rarely need streaming just because the tooling exists.
Tradeoffs
- • Batch keeps operations simpler for many workloads.
- • Streaming improves freshness but multiplies operational edges.
When to use
- • Choose batch when bounded latency is acceptable and correctness dominates.
- • Choose streaming when replay, lag and consumer behavior can be operated safely.
Risks
- • Premature event complexity
- • Slow batch systems that become invisible until they fail
Questions I ask
- • How fresh must the data be?
- • Who operates replays?
- • What is the sink behavior under pressure?
Decision surface
Managed vs Self-Hosted
Platform teams constantly trade direct control against maintenance burden.
Tradeoffs
- • Managed services reduce undifferentiated toil.
- • Self-hosted systems can improve control, locality and customization.
When to use
- • Prefer managed when the workload is standard and team attention is scarce.
- • Prefer self-hosted when data boundaries, performance behavior or platform constraints require it.
Risks
- • Operational sprawl
- • Vendor lock-in without clear exit paths
Questions I ask
- • What control is actually required?
- • Who will operate the custom path?
Decision surface
Local AI vs Cloud AI
AI systems vary widely in privacy requirements, latency budgets and operational ergonomics.
Tradeoffs
- • Local AI increases control and experimentation speed.
- • Cloud AI improves managed availability and model breadth.
When to use
- • Use local stacks when data control, offline work or experimentation matters most.
- • Use cloud stacks when scale, managed services and integration speed matter more.
Risks
- • Operational complexity on local stacks
- • Opaque failure behavior and governance gaps on cloud stacks
Questions I ask
- • What evidence needs to remain local?
- • How do we observe retrieval and tool behavior?
Decision surface
Build vs Buy
Internal engineering products often sit on the border between platform reuse and custom workflow value.
Tradeoffs
- • Buying reduces time to first capability.
- • Building can align the system to operational reality.
When to use
- • Buy when the workflow is standard and integration is straightforward.
- • Build when the operational model itself is the differentiator.
Risks
- • Custom systems with weak ownership
- • Bought systems that operators bypass because they do not fit
Questions I ask
- • What is truly custom here?
- • What would operators still do manually after the purchase?
Knowledge graph
Technologies Connected By Evidence
Every technology is derived from the same experience, project, case-study, experiment and lesson graph. Only real relationships are shown.
Observability · conceptual system model
Event evidence
Connected to: Application, Database, Kafka, OpenSearch
Signal paths
Select logs, metrics or traces to inspect how signals move into OpenSearch, Grafana, Instana, VictoriaMetrics and alerting.
What I build
Select a domain to shift the evidence surface, then inspect a technology node to see where it shows up across production work, research and engineering experiments.
Technology topology
AWS
Infrastructure foundation for compute, networking, data and operational integrations.
Evidence
4
Selected node
AWS
production · production
Cloud infrastructure, automation and operational data integrations in M3 platform work.
Projects
1
Case studies
1
Experience
1
Lessons
1
Evidence
• Operational tooling for M3 workflows
• Health monitoring
• Data search and analytics
• Operational configuration and data-update workflows
• Comparator support paths
• Observability integrations
Projects
• M3 OPS
Case studies
• M3 Operations Platform
Experience
• Platform And Reliability Operations
Lessons
• Good observability reduces decision latency, not just dashboard count.
Future direction
• AI Runtime Control Planes
Tech stack
An Interactive Skill Graph
Platform engineering sits at the center, with cloud, data, observability, AI and application engineering connected around it.
Engineering
Selected domain
Platform Engineering
Operating systems, delivery workflows and runtime environments that make other teams faster and safer.
Evolution
Capability Evolution
This timeline shows how the engineering focus widened over time from application delivery into data, operations, platform, observability and AI-adjacent systems work.
Capability evolution · conceptual system model
Software delivery
Connected to: Data
Capability path
Select a capability stage to inspect how the engineering focus widened over time.
Activated
Software delivery
Capability evolution
Foundation
Core software and systems grounding that led into enterprise engineering work.
• Start from system behavior, not tooling fashion.
Capability evolution
Enterprise Data
Pipeline-oriented engineering around SQL Server and enterprise data movement.
• Data reliability is operational work, not only ETL work.
Capability evolution
Application Modernization
Architecture cleanup and modernization across application stacks.
• Simpler systems are easier to evolve under pressure.
Capability evolution
SRE And Operations
Reliability, supportability and operational tool design became central engineering concerns.
• The best systems shorten the path from issue to action.
Capability evolution
Platform Engineering
Runtime, automation and workflow thinking expanded from isolated services to engineering platforms.
• Platforms succeed when they remove repeated cognitive load.
Capability evolution
Data Platform
Distributed data movement and performance engineering became a larger part of the system picture.
• Data volume changes architecture choices very quickly.
Capability evolution
MLOps And AI Infrastructure
Practical exploration of retrieval systems, vector search and agent tooling as infrastructure problems.
• AI systems need the same operational rigor as any other platform component.
Interactive resume
A Searchable Resume Layer
The resume is embedded into the portfolio instead of hidden behind a stale PDF. The dedicated route stays factual and pulls from the same content model as the rest of the site.
Resume summary
Rishabh Gupta
10+ years of engineering experience across platform, cloud, data and operational systems.
Experience domains
11
Case studies
8
Resume policy
• No unsupported employers, titles, dates, metrics or certifications have been invented.
• Experimental work is separated from production-oriented experience.
• The route is paired with a downloadable static PDF built from the verified resume content.
Contact
Open A Conversation
Platform Engineering, SRE, data platforms, AI infrastructure and architecture conversations are all in scope. The fastest paths are direct email, LinkedIn and the resume route.
Primary contact paths
Open to senior engineering, platform, data and AI infrastructure conversations.
Gurugram, India