There are several systems like this.
http://dedalvs.free.fr/slipa.html (by David Peterson) is AFAIK not
nearly final, but in addition to providing a new system for
transcribing sign, it also lists/reviews the major existing
So, yes, this has been done before. :-P
Also look at the material available for e.g. coding "body language".
Good book on ASL which includes this is _The Signs of Language_ and
ditto, Revisted; IIRC the first has a nice article on the origin and
description of the first system for this.