AI Form Coach Agent

AI Form Coach Agent is a production-ready, AI-powered fitness coaching web application. It evaluates exercise form in real time using client-side MediaPipe Pose Detection and generates structured weekly coaching progress reports using Google Agent Development Kit (ADK) and a Python MCP Server.

Supported Exercises:

1. Squats (Depth, knee valgus collapse, and spine leaning angle)
2. Push-Ups (Body alignment, chest depth, and elbow lockout)
3. Pull-Ups (Full extension dead hang, chin-over-bar vertical height, and kipping/swinging)

---

Key Features

• Strict Rep Counting: Only increments reps if range of motion and core biomechanical postures are correct.
• Real-Time Speech Coaching: Browser-based Web Speech Text-to-Speech voices read form cues and rep alerts dynamically.
• Landmark Confidence Filtering: Average visibility checks of key joints. If visibility is < 80%, the system hides corrections, pauses reps, and alerts you to adjust the camera.
• Weekly AI Reports: Aggregate consistency metrics, calculate form improvement % vs last week, list most common mistakes, and recommend corrective focus drills.
• Privacy First: Video frames are processed entirely client-side in the browser. No video is ever sent to the server. Only rep details, aggregate metrics, and joint landmarks are stored.

---

System Architecture

graph TD
    Client[React Frontend / Vite / TS / Tailwind] -->|Landmarks, Summary & Confidence| Backend[FastAPI Backend]
    Client -->|Camera Stream| MediaPipe[MediaPipe Pose client-side]
    Backend -->|Read/Write| DB[(SQLite Database)]
    Backend -->|Tool Calls| MCPServer[MCP Server python]
    MCPServer -->|Access DB| DB
    Backend -->|Orchestrate Agent| GoogleADK[Google ADK Coaching Agent]
    GoogleADK -->|Use Tools| MCPServer

Loading

---

Local Setup & Installation

Prerequisite Check

Ensure Node.js (v18+) and Python (3.10+) are installed.

1. Backend Setup

1. Navigate to the backend directory:

   cd backend

2. Initialize virtual environment:

   python -m venv venv

3. Activate virtual environment:
   • Windows PowerShell: .\venv\Scripts\Activate.ps1
   • macOS/Linux: source venv/bin/activate
4. Install dependencies:

   pip install -r requirements.txt

5. Configure .env file:
   • Open the .env file in the backend folder.
   • (Optional) Set your GEMINI_API_KEY to enable Gemini AI reports. If left blank, the backend automatically triggers local rule-based coaching reports.
6. Start the FastAPI server:

   uvicorn app.main:app --host 127.0.0.1 --port 8000 --reload

2. Frontend Setup

1. Open a new terminal and navigate to the frontend directory:

   cd frontend

2. Install package requirements:

   npm install

3. Start the Vite React development server:

   npm run dev

4. Open your browser and navigate to http://localhost:5173.

---

Running with Docker Compose

To deploy the entire stack in containers (FastAPI on port 8000, React client on port 80):

1. From the root project directory:

   docker-compose up --build

2. Open your browser and navigate to http://localhost.

---

Biomechanics Rules & Joint Angles

Squat

• Knee Angle (Hip -> Knee -> Ankle): Descends to < 100° for a valid rep.
• Lumbar Lean Angle (Shoulder -> Hip relative to vertical): Warns if lean exceeds 40° (rounded back).
• Knee Valgus Ratio (Distance(Knees) / Distance(Hips)): Warns if ratio falls below 0.85 (knees collapsing).

Push-Up

• Elbow Angle (Shoulder -> Elbow -> Wrist): Lowering to < 90° and extending to > 160° for a valid rep.
• Core Alignment Angle (Shoulder -> Hip -> Ankle): Warns of hip sagging (< 155°) or pike hips (> 200°).

Pull-Up

• Hang Extension (Shoulder -> Elbow -> Wrist): Starts from fully extended arms (> 155°).
• Chin Clearance: Chin Y-coordinate must clear wrist Y-coordinate.
• Torso Sway: Hip X displacement fluctuation warns if kipping/swinging is detected.

Deployed Link = https://ai-form-coach-orpin.vercel.app/

---

Security & Privacy Statement

The AI Form Coach Agent is engineered to protect user privacy:

1. Zero Footage Stored: Webcam video frames are read by MediaPipe in-browser and immediately discarded. No video data leaves your local machine.
2. Local Databases: Workout history is saved in a local SQLite database (fitness_coach.db).
3. Local AI Fallbacks: Runs 100% locally if no API key is set, protecting your metrics from external API requests.