This　paper　introduces　a　new　idea　for　interaction　between　human　and　wearable　device　which　is　using　finger-fist　posture　to　be　the　detecting　and　tracking　target.　We　built　the　detector　with　cascade　classier　using　Haar-like　features　and　the　AdaBoost　learning　algorithm.　The　detector　for　the　posture　shows　good　tolerance　for　out-of-plane　rotation　and　robustness　against　lighting　variance　and　cluster　background.　With　excellent　real-time　performance　and　high　recognition　accuracy,　the　detection　can　be　acted　as　a　tracker　to　track　the　path　of　fist　in　the　first-person　view.