Home
Creators
Search
Recent
Random
Posts
DMs
Tags
Random
Importer
Import
FAQ
Account
Register
Login
Favorites
Keys
Logout
Telegram
【Live2D/VTubeStudio Explanation】 About movements and parameters detected by hand tracking (Patreon)
Content
*This article is a translation of an article I published on Pixiv Fanbox on November 8., 2022.
The other day I was able to successfully show off my hand tracking.
https://twitter.com/himono_vtuber/status/1587075600677711872?s=20&t=a5HGOnXufprkPcWYscN7EQ
The response has been tremendous, and I'm very grateful!! I'm glad I worked so hard on it!
I will continue our slow explanation of hand tracking here.
This time, we will talk about "what kind of movement is actually detected" and "what parameters should be tied to it.
First of all, it is very basic, but the hand tracking settings are made from the third menu from the left in the top-left Details , the menu with the cogwheel on the person.
“IN” is the motion parameters that can be detected.
“OUT" is the "parameters created by Live2D.
These two can be freely combined, so for example
When the hand goes up, the shoulder also goes up.
If the hand is in front of the body, turn on a parameter that brings the drawing order to the front.
It can be applied like this.
Pressing the blue "IN" button brings up a list of detectable movements.
Of these, those involved in hand tracking are those with "Hand" or "Finger" attached.
The actual hand tracking can be used for
24 movements, 12 for each hand
2 movements involving both hands
for a total of 26.
I will explain one by one from the top to the bottom (it is long, so please skim over the non-essential information as needed).
ーーーーーーーーーーーーーー
【HandLeftFound】【HandRightFound】【BothHandsFound】
This is the parameter that moves when a "hand is detected."
Like the "cheek puk" parameter, the IN value moves from Min to Max instantaneously when a hand is detected.
The IN value moves from Min to Max in an instant when a hand is detected.
You may vaguely understand that "Left" and "Right" are for one hand, and "Both" is for both hands.
However But please note that "Left" is the "←" hand as seen from the camera.
In Live2D, the "L" and "R" are named from the character's point of view, so the hands to be connected are probably reversed.
“Both" moves only when both hands are detected.
A quick way to use these parameters that comes to mind is to link them to "parameters that raise the arm" or "parameters that switch to the difference of the arm being raised" so that "the arm is instantly raised when the hand is detected. If you directly connect parameters with deformations, the switching of movements will be too abrupt, so I think it would be more natural to connect them with physics operations.
(I will explain about physics operations later.)
【Video when directly tied to deformation parameters.】
https://youtube.com/shorts/s8QPpN2ExN4?feature=share
【Using physics calculations】
https://youtube.com/shorts/zwydcqTNr0k?feature=share
By the way, I did not use these 【HandLeftFound】, 【HandRightFound】, and 【BothHandsFound】 this time.
Then how are the arms detected? I will explain later.
ーーーーーーーーーーーーーー
【HandDistance】
It is a parameter that moves with the "distance between the right hand and the left hand.
The distance of the circle between the right and left hands on the webcam screen is exactly that. By the way, it seems to become a hand sign when they are attached to each other (I just noticed that).
The values increase the farther apart the hands are from each other.
If I use it well, I could make it a "differential trigger to combine both hands" like a clap or a heart mark hand. I have not used it so far.
ーーーーーーーーーーーーーーー
【HandLeftPositionX】【HandRightPositionX】
This is a parameter that moves with the "X-axis (horizontal axis) position of the hand."
It would work well if you include shoulder and elbow movements.
It seems that if you move it slowly, it will detect even if you can't see it to some extent.
This movement can also be output by using physics calculations.
The "shoulder" and "elbow" movements are blended well and output.
ーーーーーーーーーーーーーーーー
【HandLeftPositionY】【HandRightPositionY】
This time it is a parameter that moves with the position of the hand on the Y-axis (vertical axis).
This one is a little more difficult to utilize.
Would it look good to include A or B movement?
(I have it so that it switches between A and B depending on the position of the hand on the X-axis.)
Furthermore, the "range limit" on the "Input" button and physics are well combined to do things like "only move the shoulders up and down when they are above a certain height".
ーーーーーーーーーーーーーーーー
【HandLeftPositionZ】【HandRightPositionZ】
This is a parameter that moves with the Z-axis (back and forth) position of the hand.
It is a little bit covered by the Y-axis movement B mentioned earlier, but I think it will look better if a movement to move the hand closer to the camera is included. (It is more difficult to create).
Compared to the XY-axis, it does not move so spectacularly. It adds a little bit of dynamism... or something.
(I use physics to blend the movement of the shoulder with the movement of the B of the Y-axis as described above.)
ーーーーーーーーーーーーーーーー
【HandLeftAngleX】【HandRightAngleX】
This is the movement of twisting the hand back and forth. This is the most difficult part to make.
It is designed to detect wrist movement for 360° around the wrist, but
In reality, however, only 180° of "inner rotation" can be used for clean movement.
As I mentioned in my last article, there is a mysterious bug that when the internal rotation exceeds 180°.
Once the angle at which the bug occurs is exceeded, it moves nicely from there, but the discomfort of the bug is so great that it does not look right unless it is stopped at the angle just before the bug occurs.
↑This works beautifully so far.
↑Moving the wrist to an angle greater than this will cause the wrist to rotate once.
It is possible to move it beyond the angle at which the ↑ bug occurs, but it is not practical to use because the wrist will make a full turn.
↑They also move outward a bit, but they are very buggy, so this is not realistic either (and will hurt your wrists).
This is what it looks like in numerical values. This bug doesn't seem to have anything to do with smoothing, changing tracking values, or with or without physics, so I have no way to hit it at the moment. If anyone knows a solution, please let me know.
I took the "hands only move so far" approach.
ーーーーーーーーーーーーーーーー
【HandLeftAngleZ】【HandRightAngleZ】
The angle of the wrist. This one has a few quirks as well...
As I mentioned in my last article, when the palm of the hand is facing forward, it moves nicely as normal.
However, when the back of the hand is facing forward, it moves in the opposite direction.
By the way, when the hand is pointing to the side, it does not move at all.
This is a repeat of the previous article, but this is because the values are based on the "angle from the hand". If the hand is turned over, the numerical value would be the opposite, and the ↑ movement when the hand is turned sideways is strictly "hand angle Y", so it does not move.
The hand angle Y is not yet implemented, so there is nothing we can do about it, but the movement when the back of the hand is in front can be corrected either directly or by correcting it with physics calculations. I use physics to correct for the opposite movement only when the back of the hand is in front of the hand.
(When the back of the hand is in the front, it doesn't move much to the ← side, but the wrist actually moves very little to the thumb side, so I think this is fine.)
Here's what it looks like in numerical terms.
ーーーーーーーーーーーーーーーー
【HandLeftOpen】【HandRightOpen】
This is a parameter that moves with the opening and closing of the hand. There is no distinction between fingers.
In this figure, the five circles are the movements of each finger, and the bar below them is this 【Open】 parameter.
It could be used to create a hand opening and closing, even though it would be tedious to create finger distinctions, or a differential that is only triggered when the hand is held, or many other triggers.
I didn't use it this time.
ーーーーーーーーーーーーーーーーーーーー
【HandLeftFinger_1_Thumb】
【HandLeftFinger_2_Index】
【HandLeftFinger_3_Middle】
【HandLeftFinger_4_Ring】
【HandLeftFinger_4_Pinky】
【HandRightFinger_1_Thumb】
【HandRightFinger_2_Index】
【HandRightFinger_3_Middle】
【HandRightFinger_4_Ring】
【HandRightFinger_4_Pinky】
There it is! Fingers. In the order of 1 to 5, we have "thumb", "index finger", "middle finger", "ring finger", and "little finger".
As a side note, once you get used to this, I think it makes more sense to name them in English. In Japanese, the ring finger is called the “Kusuri”="drug" finger.haha
All fingers move basically beautifully, but only the thumb moves poorly when it is "making a fist with the hand turned to the side," so if it is made too open, it will look uncomfortable.
In the beginning, we wanted to have a wider range of motion for the thumb as well, but when I held my hand, only the thumb would stick out as shown in the ↓. Now I give priority to the appearance rather than the range of motion.
That's it for the parameters! There are a lot of them, but if combined well, the large number of them can be used to make a very beautiful hand.
ーーーーーーーーーーーーーーーーーーーーーーーーーーーーー
Finally, let me tell you something important.
If you are already able to create hand tracking and have followed the settings as described in the ↑ explanation, you will notice that the movement is decidedly different from my hands.
“What? When the tracking goes off, the arm doesn't go back to where it was!”
Yes, if this is the case, "when the tracking goes off, the arm stops in that state".
At the beginning of this article.
I didn't use these [HandLeftFound], [HandRightFound], and [BothHandsFound] this time.
'Then how are the arms detected?' I will explain that later.
I think I wrote, "Strictly speaking
"How do you lower the arm when the arm is not detected?"
The question is more correct.
Differences" and "hand gestures" play an active role here.
Differences and hand gestures are performed using the rightmost button on the top-left detail menu.
When it comes to differencing, you may have a strong image of "sparkling eyes," "embarrassed face," "bye-bye motion," and other such on/off differencing and motions for use with key presses, but VTS can be used in a variety of other ways.
In this case, we will create a difference with the condition of "fixing the arm in a specific position and shape when the hand is not detected.
Press "Expression Editor" at the top of the tabs on the right side.
Press "Create New Expression".
You can see the parameters set in Live2D on the right side of the screen.
I named it "← Hand Detection" as appropriate.
Then, check the "← Parameter to be fixed when hand is not detected" checkbox and slide the bar to fine-tune the position so that the hand appears to be "lowered without discomfort.
*Pudding is just putting it on to avoid a ban, sorry hahaha.
I move everything from my shoulders to my fingertips in tandem, so you need to check all the boxes. If you only move from your elbows to the tips of your fingers, you only need to check the points to be moved.
If all goes well, press "Save" and create a diff file (exp3.json).
The process of creating the diff file up to this point can actually be done without using VTS, but I think it is easier to use VTS.
Next, press the "+" button in the lower right corner of the screen to create the difference.
The name was appropriately changed to "←Hand OFF".
“Hotkey Action" is "Set/Unset Expression”.
Put the difference file created earlier in "Expression".
Normally, I would enter some key (Shift, W, etc.) in the "Key Combination" field, but this time I did not enter anything.
And the most important "Gesture Trigger".
When the ""Gesture"" button is pressed, various hand signs will appear.
If we make the hand sign set here toward the camera, we can turn on the differential just like when we "press a key on the keyboard".
In this case, select the one with an X on the hand symbol in the lower right corner. This means "ON when no hand is detected. This is exactly the condition for this time.
Click "L" under the hand sign since this is a ←hand setting.
The time to detection is your choice, but I set it to just under 0.3. If the time is too short, it will easily lead to false positives!
Then check the "Deactive Expression when gesture not detected anymore" checkbox. This will automatically turn off the differential when a hand is detected.
(It is a bit complicated, but "when the gesture of "hands not detected" is not detected" means "when hands are detected"!)
When finished setting the gesture, press OK.
Last! The "Fade few sec" in the Hotkey setting, can be changed to change the "speed at which the hand is lowered/raised" by changing the value here.
For 0.1
For 0.3
For 0.5
At the time of the unveiling distribution, I had set it to 0.1 because I thought it would be better if it moved quickly, but when I looked at it as a whole body, I thought it would be a little uncomfortable when I took it down, so I just changed it to 0.3. I think people have different tastes in these numbers, so go with your preference!
This completes the fixation of the ← hand. →Do the same for the ← hand.
That's all for this article! From the next article, I will finally explain how to create with Live2D. (Thank you for your patience...!)
See you then!
Files
