AIUB•Research & Publication•vision
Real-Time Hand Detection & 3D Depth Tracking (IEEE ICAEE 2015)
Peer-reviewed computer vision research presented at IEEE ICAEE 2015 and indexed in IEEE Xplore, introducing a real-time 3D spatial hand detection and tracking pipeline using synchronized stereo webcams.
Executive Overview & Key Engineering Deliverables
- Stereo Vision Calibration: Dual-webcam optical rig calibrated via Bouguet's rectification algorithm, generating real-time horizontal epipolar alignment and disparity maps via StereoBM block matching.
- Robust Skin Color Segmentation: Fused differential YCrCb chrominance modeling (Cr - Cb) with MOG2 Gaussian Mixture background subtraction to isolate hands under dynamic lighting conditions.
- Kinematic Feature Tracking: Geometric contour analysis using image moments, polygon approximation (approxPolyDP), and convexity defect angles (<= 90°) to identify fingertip coordinates in 3D.
- Euclidean Point Cloud Generation: Real-time reprojection into 3D metric coordinates (X, Y, Z) rendered interactively via Point Cloud Library (PCL) at ~30 FPS without requiring expensive depth hardware.
C++OpenCVPoint Cloud Library (PCL)BoostStereo VisionStereoBMBouguet RectificationYCrCb Skin ModelingMOG2 Background SubtractionConvexity Defects
Publication
IEEE ICAEE 2015
Optical Baseline
12 cm Rig
Disparity Method
StereoBM
Color Model
YCrCb
LIVE SYSTEM SIMULATION & INTERACTIVE PREVIEW
STATUS: ACTIVE WEBGL/SIM3D STEREO DEPTH HAND TRACKING // 2D LIVING ROOM SIMULATION
DUAL WEBCAMS • TV ARCADE GAME • SKELETAL TRACKING (IEEE 2015)STEREO RIG: DUAL WEBCAMS ACTIVE•32 FPS SYNC // Z = (f • B) / d
Game Gesture:
TRIANGULATIONZ = (f • B) / d
STEREO RIGDual Webcams (12cm)
DISPARITY ACCURACY94.2% (IEEE Tested)
CONFERENCEIEEE ICAEE 2015
FULL-STACK SYSTEM ARCHITECTURE SPECIFICATION
CLIENT & UI TIER
- OpenCV HighGUI multi-window interactive visualizer displaying synchronized stereo video feeds (left/right cameras at 640x480 resolution).
- Real-time disparity map visualizer with CV_8U min-max normalization and COLORMAP_HOT false-color thermal depth gradient.
- Point Cloud Library (PCL) 3D Viewer (PCLVisualizer) rendering dense RGB-colored point cloud surfaces (pcl::PointXYZRGB) with real-time camera manipulation.
- Interactive parameter tuning HUD with dynamic trackbars for SAD window size, disparity search range, pre-filter caps, and speckle noise filtering.
API & REAL-TIME ENGINE
- C++ core vision pipeline executing at ~30 FPS over synchronized dual-webcam video streams (cvCaptureFromCAM).
- Adaptive skin color segmentation pipeline combining YCrCb color-space conversion (COLOR_BGR2YCrCb) with differential chrominance thresholding (Cr - Cb).
- Gaussian Mixture Model (MOG2) background subtraction isolating foreground motion and eliminating static skin-toned background clutter.
- Morphological elliptical structuring filters (MORPH_CLOSE & MORPH_OPEN) and custom contour hole-filling algorithms removing interior illumination voids.
- Spatial kinematics solver utilizing zeroth and first-order image moments for hand centroid tracking, paired with approxPolyDP and convexHull.
- Angular convexity defect classifier computing trigonometric dot products (getAngle) to identify finger webbing valleys (<= 90°) and classify active fingertips.
DATA & INFRASTRUCTURE
- XML-serialized stereo calibration parameters (FileStorage) storing intrinsic matrices (M1, M2) and radial/tangential distortion vectors (D1, D2).
- Stereo rectification matrices (R1, R2) and projection mapping matrices (P1, P2) derived from Bouguet's calibration algorithm.
- 4x4 perspective transformation matrix Q encoding camera focal length, stereo baseline, and principal point offsets for depth triangulation.
- Dual-webcam synchronized optical rig with calibrated 12 cm physical baseline mounted on rigid mounting bracket.
- Bouguet epipolar line alignment generating rectified horizontal scanlines (cvRemap) for single-row correspondence matching.
- Boost multithreaded runtime (boost::thread, boost::shared_ptr) decoupling frame acquisition from 3D PCL point cloud rendering loops.
Key Technical Achievements
- 01Published in IEEE: "Real-Time Hand Detection and Tracking with Depth Values" presented at IEEE ICAEE 2015 and indexed in IEEE Xplore.
- 02Dual-Webcam Stereo Rig Calibration: Calibrated dual cameras with Bouguet's algorithm, generating precise undistortion/rectification maps (mx1, my1, mx2, my2) and 4x4 reprojection matrix Q.
- 03Stereo Block Matching (StereoBM): Real-time disparity map computation with configurable SAD window sizes, pre-filter caps, and speckle window noise suppression.
- 04Differential YCrCb & MOG2 Skin Segmentation: Combined (Cr - Cb) chrominance subtraction with adaptive Gaussian Mixture background subtraction to cleanly segment hands in dynamic lighting.
- 05Convexity Defect Fingertip Tracking: Identified fingertip coordinates and palm centroids using polygon approximation (approxPolyDP), convex hulls, and angular defect filtering (getAngle <= 90°).
- 063D Point Cloud Reconstruction: Reprojected disparity pixels into 3D metric Euclidean space (X, Y, Z) and visualized dense colored point clouds via Point Cloud Library (PCL).
Measurable Impact & Production Verification
Successfully published at IEEE ICAEE 2015 and indexed in IEEE Xplore, proving that low-cost commodity stereo webcams with YCrCb color modeling and Block Matching can achieve accurate real-time 3D spatial hand tracking without expensive specialized depth sensors.
Key Production Deliverables & Verification
Peer-Reviewed IEEE Publication
Published at IEEE ICAEE 2015 and indexed in IEEE Xplore digital library for low-cost stereo tracking.
Calibrated Dual-Webcam Rig
12 cm physical baseline with Bouguet undistortion/rectification maps and 4x4 reprojection matrix Q.
Differential YCrCb & MOG2 Fusion
Fused (Cr - Cb) chrominance subtraction with adaptive Gaussian background subtraction for dynamic lighting.
3D Point Cloud Reconstruction
Real-time Euclidean 3D hand tracking with OpenCV StereoBM block matching and Point Cloud Library (PCL).