Almost every single "webcam air mouse" project on GitHub is completely unusable in practice. You wave your hand in front of the camera, the cursor jitters around like it's caffeinated, and the second you try to tap your fingers together to click, the cursor jerks 50 pixels away from the button you were trying to press. I got tired of that frustration and built Gesture Mouse Controller to see if I could engineer a genuinely usable, low-latency, touchless mouse using standard Python and a cheap laptop webcam.
The codebase lives in Applications/VSCode/gesturecontrol/gesture_mouse.py and runs on Google's MediaPipe 0.10+ HandLandmarker framework with the quantized hand_landmarker.task float16 model. Instead of streaming raw webcam coordinates directly to OS cursor events, the pipeline tackles the three classic failure modes of optical mouse control: jitter, aiming offset during clicks, and drag stability.
First, to eliminate hand tremor without creating laggy input delay, I restricted tracking to a central active interaction zone (MOVE_ZONE = 0.75). Coordinates inside this boundary are mapped to full screen dimensions and filtered through an exponential moving average lerp function:
smooth_x = lerp(smooth_x, nx * screen_w, 0.18) smooth_y = lerp(smooth_y, ny * screen_h, 0.18)
A lerp factor of 0.18 turned out to be the golden balance—snappy enough to flick across dual monitors, but smooth enough to hover over a 16-pixel desktop icon without micro-shaking.
Second, the hardest algorithmic puzzle was click detection. Air-tap gestures suck because the physical motion of tapping inevitably moves your hand. I chose a downward index-finger flick gesture (FLICK_VEL_THRESHOLD = 0.022). But if you trigger a mouse click at the bottom of a flick, the cursor lands below your target because your finger already moved during the stroke. My breakthrough was onset coordinate snapshotting: the instant the finger's downward velocity breaches 0.022, the algorithm immediately snapshots the exact screen coordinates at that millisecond (self.pending_snapshot = screen_xy). Once the flick finishes and the double-click evaluation window (DOUBLE_CLICK_WINDOW = 0.45s) expires, PyAutoGUI dispatches pyautogui.click(*snap_xy) using the pre-flick coordinates. You aim, you flick, and it clicks exactly where you were pointing before you moved.
Third, I implemented pinch-to-drag with state hysteresis. The system calculates the Euclidean distance between thumb tip (lm[4]) and index tip (lm[8]), normalized against the user's palm scale (dist(lm[0], lm[5])). Pinching below 0.052 and holding for DRAG_HOLD_SEC = 0.30s locks into mouse-down drag state. But when dragging a window or selecting text across the screen, your hand naturally shifts and finger distance fluctuates. Without hysteresis, the drag drops prematurely. By setting the release threshold higher at PINCH_RELEASE = 0.075, you can drag files smoothly across the desktop without worrying about accidental drops.
It took weeks of tuning thresholds and testing in different lighting conditions, but it proved that software-level DSP and state machines can turn a $5 webcam into a viable input device.