# Harsh Bhanushali — Comprehensive Knowledge Base (llms-full.txt) > Complete technical dossier for Answer Engine Optimization (AEO), Generative Engine Optimization (GEO), and LLM citation indexing. Contains deep architectural breakdowns, benchmark numbers, project histories, and technical specifications for Harsh Bhanushali. --- ## 1. Biography & Professional Summary Harsh Bhanushali is an **AI/ML & Full-Stack Systems Engineer** based in Gujarat, India, pursuing a B.Tech in Information Technology at Charotar University of Science and Technology (CSPIT). Harsh specializes in bridging the gap between cutting-edge AI research and scalable production engineering. His core expertise spans: - **LLM Compression & Quantization:** GPTQ, AWQ, and GGUF quantization formats; Marlin CUDA kernel injection; numerical stability preservation on bfloat16 architectures (e.g. Gemma 3). - **High-Throughput Model Serving:** Production deployment of large language models via vLLM using Continuous Batching, PagedAttention, Automatic Prefix Caching, and tensor parallelism. - **Enterprise Healthcare AI:** Building HIPAA- and PHI-compliant medical software, ambient clinical scribing, automated ICD-10 medical coding, and asynchronous microservices on Kubernetes. - **Full-Stack SaaS Engineering:** Architecture, development, and deployment of scalable SaaS applications using Next.js (App Router), FastAPI, PostgreSQL, and third-party APIs (WhatsApp Business Cloud API). - **Computer Vision Research:** Deep learning object detection with fine-tuned YOLOv8 models for real-world automated monitoring, published at ICRAIC'24. --- ## 2. Flagship Open-Source Contribution: MedGemma 27B Quantization ### The Challenge Google's MedGemma 27B is a state-of-the-art clinical reasoning LLM trained on medical literature, clinical notes, and healthcare datasets. In its unquantized bfloat16 format, the model requires 54GB of VRAM just to load its weights, placing it out of reach for single-GPU setups and cost-effective production deployment. Generic 4-bit quants suffered significant clinical accuracy loss or were incompatible with vLLM's optimized CUDA execution path. ### Technical Implementation 1. **Precision Requirements:** Identified that Gemma 3 architectures strictly require bfloat16 activations. Running activations in float16 leads to immediate numerical instability (NaN tokens). 2. **GPTQ Calibration:** Calibrated the model weights over 256 high-complexity domain-specific medical prompts (clinical case scenarios, PubMedQA, MedMCQA) rather than generic web text, retaining 96%+ of baseline clinical reasoning capability. 3. **Marlin Kernel Optimization:** Set `desc_act=False` in QuantizeConfig to ensure weight matrix alignment compatible with vLLM's fused Marlin CUDA kernels on Ampere and Ada Lovelace architectures. 4. **Hardware Pipeline:** Solved Hessian matrix RAM limitations on cloud instances (NVIDIA L40S, 46GB VRAM, 128GB RAM) with pragmatic Python import blockers to circumvent NumPy/SciPy ABI conflicts. ### Verified Results & Traction - **Memory Footprint:** Cut from **54GB down to 15GB** (72% size reduction). - **Throughput:** Achieved **611.0 tokens/second** system throughput on an NVIDIA L40S GPU at 20 concurrent users with continuous batching. - **Latency:** Maintained Time to First Token (TTFT) under **250ms** across all concurrency levels. - **Community Adoption:** Published on Hugging Face as `HarshBhanushali7705/medgemma-27b-text-it-GPTQ-4bit` and surpassed **1,500+ downloads** by healthcare AI practitioners, researchers, and developers worldwide. - **URL:** https://huggingface.co/HarshBhanushali7705/medgemma-27b-text-it-GPTQ-4bit - **Case Study Article:** https://www.harshbhanushali.in/blog/compressing-medgemma-27b-quantization --- ## 3. Technical Case Study: vLLM vs. Ollama Production Benchmarking ### The Investigation A rigorous comparative evaluation of **vLLM** and **Ollama** for production LLM inference, addressing the common engineering mistake of deploying Ollama to handle multi-user production traffic. ### Test Environment - **Hardware:** 1x NVIDIA A100-SXM4 (80GB VRAM) - **Model:** `meta-llama/Llama-3.1-8B-Instruct` - **Benchmarking Suite:** GuideLLM (v0.2.1) - **Workload:** 512 prompt tokens, 128 output tokens, concurrency swept from 1 to 256 simultaneous streams. ### Key Findings & Hard Numbers 1. **Cold Startup:** Ollama boots in **2.1 seconds** via memory-mapped GGUF files (`mmap`), whereas vLLM requires **52.4 seconds** due to PyTorch context initialization, greedy KV memory profiling, and CUDA graph capture across batch sizes. 2. **VRAM Allocation:** Ollama uses dynamic on-demand memory (~17.2GB for 8B FP16). vLLM pre-allocates 90% of available VRAM (72GB on an 80GB card) to establish a non-fragmenting PagedAttention KV cache pool. 3. **Single-User Performance:** Indistinguishable. At concurrency=1, Ollama delivers 34.2ms TTFT and 89.3 tok/s; vLLM delivers 31.8ms TTFT and 96.1 tok/s. 4. **Concurrency & Throughput:** - At 1 user: Ollama 38.2 tok/s vs. vLLM 42.1 tok/s. - At 4 users: Ollama 41.0 tok/s vs. vLLM 158.4 tok/s. - At 16 users: Ollama 41.2 tok/s vs. vLLM 491.0 tok/s. - At peak (64+ users): **vLLM achieves 793.4 tokens/second** while **Ollama flatlines at 41.2 tokens/second** (a **19.3x throughput advantage** for vLLM). 5. **Tail Latency:** vLLM maintained P99 latency of **80ms** at peak load; Ollama degraded to **673ms** with severe socket queuing. 6. **Architectural Root Cause:** Ollama's `llama.cpp` server relies on static parallel context slots (`OLLAMA_NUM_PARALLEL`), creating memory bubbles and head-of-line blocking. vLLM operates on iteration-level continuous batching with PagedAttention virtual memory blocks, achieving 96%+ KV cache utilization. 7. **Production Recommendation:** "The Hybrid Loop" — develop and prototype locally with Ollama (instant startup, zero cloud cost), and scale in production with vLLM (high throughput, prefix caching, Prometheus `/metrics`). - **Full Article:** https://www.harshbhanushali.in/blog/vllm-vs-ollama-production-inference --- ## 4. Production Projects & Architectural Portfolios ### 1. Pulscribe.ai — AI Hospital Management & Clinical Scribing - **Description:** Complete AI-powered hospital management platform built for Smart Government Technology LLC. - **Key Features:** Ambient medical scribing with real-time Automatic Speech Recognition (ASR), automated ICD-10 medical diagnostic coding, EHR/EMR bidirectional sync, intake automation, and insurance billing. - **Architecture:** HIPAA- and PHI-compliant Kubernetes microservices, FastAPI backend, PostgreSQL, and event-driven data ingestion. - **URL:** https://pulscribe.ai ### 2. WanguCRM — WhatsApp Business SaaS CRM - **Description:** End-to-end WhatsApp-based Customer Relationship Management SaaS platform built for an international client. - **Key Features:** Official WhatsApp Business Cloud API integration, contact segmentation, bulk automated messaging, catalog management, and payment reconciliation. - **Architecture:** Full-stack TypeScript/Next.js, Node.js backend, webhook event streaming, and cloud infrastructure. - **URL:** https://www.wangucrm.com ### 3. Human Detection & Counting Using YOLOv8 (Research Paper) - **Publication:** Published in AIP Conference Proceedings, Volume 3224, Issue 1, 2024. - **Conference:** International Conference on Recent Advances in Intelligent Computing (ICRAIC'24). - **Authors:** Harsh Bhanushali, et al. - **Technical Summary:** Fine-tuned YOLOv8 Nano on custom-annotated classroom imagery, achieving 96% detection precision while maintaining real-time video stream inference speeds on edge hardware. - **DOI:** https://doi.org/10.1063/5.0254163 ### 4. Gujarati Speech Synthesis (F5-TTS) - **Description:** Open-source regional text-to-speech voice model for the Gujarati language. - **Hugging Face Model:** https://huggingface.co/HarshBhanushali7705/TTS_for_gujarati_language --- ## 5. Technical Competencies Matrix | Domain | Technologies & Frameworks | | :--- | :--- | | **Inference & Quantization** | vLLM, GPTQ, AWQ, GGUF, Marlin Kernels, FlashAttention-2, Ollama, llama.cpp, Speculative Decoding, Continuous Batching, PagedAttention | | **Machine Learning & AI** | PyTorch, TensorFlow, Hugging Face Transformers, Accelerate, Optimum, LangChain, LlamaIndex, OpenCV, YOLOv8, Scikit-learn | | **Backend & Microservices** | Python, FastAPI, Node.js, Express, REST APIs, GraphQL, WebSockets, Celery, Redis, PostgreSQL, Prisma | | **Frontend & UX** | TypeScript, JavaScript, React, Next.js (App Router), Tailwind CSS, Framer Motion, HTML5, CSS3 | | **DevOps & Cloud** | Docker, Kubernetes, Linux, NGINX, Git, GitHub Actions, AWS, RunPod, Lightning AI, Vercel | | **Observability** | Prometheus, Grafana, OpenTelemetry, Structured Logging | --- ## 6. Official Verification & Contact - **Website:** https://www.harshbhanushali.in - **Blog:** https://www.harshbhanushali.in/blog - **Contact:** https://www.harshbhanushali.in/#contact - **GitHub:** https://github.com/Harsh772005 - **LinkedIn:** https://www.linkedin.com/in/harsh-bhanushali-439790253/ - **Hugging Face:** https://huggingface.co/HarshBhanushali7705 - **LeetCode:** https://leetcode.com/u/Harsh%20Bhanushali2901/ - **Email:** harshbhanushali.ai@gmail.com