PS: More advanced versions of this tool may even help us include right-click and scrolling features.
It was really important for the mouse pointer to hover across the entire screen. I encountered that as soon as the hand went outside the range of the camera, the hand wasn't detected and hence the mouse jittered at that point.
After calibrating the mouse pointer with the offsets values I was able to successfully able to get rid of the jitter.
Discussion