That's true -- but I do have those points to detect motion. And I can do a sort of subdivide, as in start with one big detector box, when it picks something up subdivide it, and keep going until I hit N boxes, so I only focus on the areas of motion. Then, since I'm using a bunch of data to average out the points of motion, I can try to create a field or something with the direction.
As for "mouse down", it can be holding in an area while still moving. I'll return to this project and see what I can come up with.