【Live2D/VTubeStudio Explanation】 How Hand Tracking Works and the Camera (Patreon)
Content
*This article is a translation of an article I published on Pixiv Fanbox on October 12, 2022.
Hello everyone、 Kanbutsu Himono.
I've got my upper body done, so I can take all the videos I want, it's great!
I got too excited while writing this article and started singing.
And when I paired it with my homemade microphone item, it looked great!
So I wrote about that in another article. ↓↓.
Let's leave it at that...
I will start explaining hand tracking and the body in this issue.
However, hand tracking in VTubeStudio (hereafter referred to as "VTS") is not well explained, and many people wonder, "How do I make it work in the first place?" I think there are many people who wonder, "How do I make it work?
First of all, I will explain the principle and the parameters that react to it.
VTS hand tracking is detected by a webcam.
Even those who normally use iOS for face tracking can combine "webcam for hands" and "iOS for face".
The accuracy of tracking looks like this.
I was contacted by a VTS official and had a chance to experience it for a bit before it was released to the public. (At that time, I used a lightly customized past model.)
My first impression was,
"A webcam can tracking with such high accuracy!!! I don't need the Leap Motion I spent over 10,000 yen to buy???." That was it
haha
I think it is comparable to Leap Motion, especially when it comes to "front facing finger tracking" and "hand angle". There is no blurring at all.
Sideways finger tracking is also a thing of beauty. It's a real mystery how they do it when the fingers in the front are hiding the fingers in the back.
However, if the hand coordinates are moved too quickly, they will be undetected.
As you become accustomed to the "hand speed at which the camera does not miss detection," you will be able to make clean movements.
Also, as a matter of course, if a hand protrudes outside the camera, the detection will not work.
I place my webcam on the ← side of the monitor, so if I move my hand too far to the → side, which creates distance from the camera, the movement will be buggy and uncomfortable.
Once I got used to this, I was able to grasp the range where the movement was not buggy.
And here are the disadvantages to write about.
One is the difficulty with the angle of the wrist.
As you can see, when made normally, the wrist moves as expected when the hand is facing forward, but when the palm is turned back, the movement is reversed
Even more! More problematic is that when the hand is turned to the side, there is no response at all at the wrist.
In short, this hand tracking function picks up the "angle from the hand", not the "angle from the monitor". So the angle of the hand when in landscape orientation is angle Z from our point of view, but from the hand's point of view it is angle Y. The hand angle Y doesn't move because it doesn't exist in the tracking item.
If the VTS management had developed this from scratch, this would not have happened (since the developers are familiar with the Live2D way of thinking), but since this is said to be an appropriation of an externally released hand tracking technology, I have a feeling that the feature was not intended for use with Live2D in the first place.
And furthermore, and furthermore. The biggest problem is this.
If I twist my wrist too far inward in front of the camera, the model's hand bugs me like crazy.
WHY!!!???
People with soft wrists need to be careful
So there is surprisingly little "clean range of motion," and it takes quite a bit of getting used to it to get your brain to grasp how much it is.
So much for how it works.
Next. Criteria for Selecting a Webcam to Consider Before Hand Modeling
You may think, "Just buy a high-performance camera with good image quality," but that's not the case.
When this feature was first released, I bought a camera with a wide angle of view to "avoid missing detection," but the wide angle of view = a large area on the screen and small hands, so the movement became proportionally smaller.
Too small haha
It's too small and frequently mis-detects the left and right hand.
More problematic is the wide angle of view, so no matter how much I move my hand to the side, I can't "get out of the tracking range"...
Then what happens...
it breaks like crazy.
If the camera has a narrow angle of view, stretching when yawning or reaching out to the side to pick up a cup will cause the hand in front of the camera to move out of the angle of view, automatically disengaging detection and placing the model's arm down.
However, if the camera's angle of view is wide enough, the hand cannot move out of the angle of view and is continually detected in a messy way, so it breaks into a mess like the video above.
So, to "move it nicely when you want to move it", an ordinary webcam used for remote meetings is sometimes the best. I have been using an old webcam that I bought in 2013. The quality is only 720p.
https://www.amazon.co.jp/gp/product/B00516G6DI/ref=ppx_yo_dt_b_asin_image_o00_s01?ie=UTF8&psc=1
(I was doing a Monster Hunter live action on the 3DS at the time and didn't have a
capture board, so I bought it to take direct pictures. Too much nostalgia)
I think the best camera height is about the same as the face (eye level).
(Everyone wants the hands brought around the face to look the highest quality, so)
I have a desk like this (sorry for the messy desk top) with three monitors filling it.
The camera is on the far left.
The bottom one is a wide-angle camera that I don't use and the top one is a normal camera that I usually use.
(The equipment below the camera I bought but don't know how to use, so it's just a camera rest;)
I think the height is more important than the horizontal position because it tracks nicely even in such an unbalanced position.
So, to summarize this article.
(1) Understanding and familiarity on the part of the person moving the model is necessary to make the model's hands move beautifully.
(2) Buy an inexpensive camera.
Next time, I would like to talk about parameters!