GPU real time Skinning – two variants

Real time “Skinning”

task of the rendering engine, done in GPU

+ =

skinned model (asset in GPU RAM)

skeletal animation

(asset in GPU RAM)

animated model

GPU real time Skinning – two variants

Transform:

𝑇0 𝑇1 𝑇2 𝑇3 Bone: Weight:

bone a 𝑤0

bone b 𝑤1

bone c 𝑤2

bone d 𝑤3

blend

How is this blend done?

 linear interpolation of matrices:

“Linear Blend Skinning”

(old school, still very much used)

 quaternion based interpolation:

“Dual Quaternion Skinning”

(different -better?- artifacts)

(it is a choice of the rendering

engine)

Orthogonality of animation / models

RIG

(skeleton) Animation

Walk Animation

Jump Animation Die

Model A

Model B

Model C

Intro: skinning

Intro: skinning

Orthogonality of

animation / models: notes



With skeletal animations,

 each skinned modelcan be moved with any animation

 each animationcan be run on any skinned model



…as long as they share a common rig

 and, even if rig is slightly different (e.g. different bone lengths): “retargeting” of anim still easy / possible



100 animations, 100 characters?

 100+100 assets: no problem

 not 100x100 assets (unthinkable to author / store)



Unprecedented in the history of animations!

 does not apply to: 2D sprite sequences, blend shapes, traditional hand-drawn animations…

(recap)

Skeletal animations:

3 Assets (data strcutures)

 Rig

(or skeleton)

 Tree di bones (ossa)

 Ɐbone=> reference frame (in rest pose)

 (reference frame root bone = objects space)

 Skinned

3D Model

 Mesh with links: vertices=> bones

 Ɐ vertex: attributes: [ bone index , weights ]x Nmax

 Skeletal animations

 Sequence of keyframeposes

 Ɐ pose, Ɐbone = a local transform

example of file formats (for all three):

 .SMD (Valve), .FBX (Autodesk), .BVH (Biovision)

Animation GPU Object

UPLOAD

Life of Animation Assets in a Game Engine

DISK CENTRAL RAM GPU RAM

Animation Object

Skeleton

Final Transforms Local

Transforms

IMPORT

Animation File

IMPORT

Skeleton

with rest pose

Mesh GPU Object

Mesh SkeletonFile

File

Mesh GPU Object

(Once again,

Memory Management)

Mesh Object

Mesh GPU Object Mesh

Object

GPU Object Mesh

Object Mesh ObjectMesh

ObjectMesh ObjectMesh

Object Animation

Object Animation

AnimationFile AnimationFile

AnimationFile AnimationFile

AnimationFile AnimationFile AnimationFile

AnimationFile AnimationFile

AnimationFile AnimationFile

AnimationFile AnimationFile

AnimationFile AnimationFile

AnimationFile AnimationFile

AnimationFile AnimationFile

File

Skeleton File

Animation Object Animation

Object

Mesh SkeletonFile

SkeletonFile Object

DISK CENTRAL RAM GPU RAM

Animation GPU Object

Memory Management for skeletal animations: notes



Animations in GPU use more space than in CPU

 typically, no key-frame interpolation:

used frames are explicitly stored

 final transforms always need rotation + translation (local transform can store just rotation -- why?)

 tr. for secondary bones (if any): all explicitly stored (in CPU, they can be left to procedural computation)



The usual trade-offs are possible:

 convert all frames once, on animation import

 convert all frames on animation load to GPU

 convert a frame just before it is used, etc

Content creation Tasks:

Rigging & Skinning

Rigging – authoring of a rig defining the skeleton

(in movie industry: also of the controls to define new poses for it)

Skinning – authoring of the skinning

“paint” of (weighted) links between vertices and bones

Content creation Tasks:

Rigging & Skinning

 Rigging (of a mesh)

:

 define a rig

(a skel and a rest pose)

 bones are inside the mesh, (or a set of meshes: a shared rig)

 Skinning(of a mesh)

:

 paint vertex-to-bones links

 Animation ^{(of a rig)}

 authoring of (skeletal) animations

 (More about this later)

skinner

animator Digital modellertask! (helped/ replaced by automaticalgorithms)

Digital animator task

literal rigs

rigger

Rigging (authoring the rig):

choices

 The Rig can include, or not, minor details

 e.g. in a human model: fingers, or finger bones

 necessities vary

 costs is high:

remember: one bone = one transform

in each key-frame, of each pose, of each animation!

 Custom bones of all kinds possible

 including for cloth parts or equipment

 (e.g. flaps of a backpack, a cloak, etc)

 Often, object held in (each) hand = 1 extra bone

 so the animation dictates how the object is held during that action

 a rigid object mesh can be “mounted” on it, dynamically

 i.e. assigned to a uniform skinning values:

bone-index : right-hand object. Bone weight : 1)

Skinning

(authoring the skin)

example, in Maya (Autodesk)

Authoring

Skeletal Animations



Manual editing

 manual key-frame posing

 e.g. hierarchically:

1. pose a few main frames firsts 2. interpolate

3. interpolation looks wrong?

make it a new key-frame, retouch it 4. goto 2

 potentially, with help from:

 IK (using animation interfaces) e.g. move the foot right there

 physical simulation

e.g. for “secondary” animations (to be baked?)

 roto-scoping from captured footage



Or…

Authoring

Skeletal Animations



Motion capture

“Mo-Cap”

Interpolation of key-frames (in skeletal animation)



any two poses can be interpolated!

 how: just interpolate the local per-bone transforms

 note: that requires re-computation of final transf

 much more powerful than blend shapes

 better results, sparser keyframes possible

pose A pose B

0.5 ∙ pose A + 0.5 ∙ pose B

--- --- ---

---

Pose = keyframe



Compress animations

keyframe A 0.75 A + 0.25 B 0.50 A + 0.50 B 0.25 A + 0.75 B keyframe B

0.50 B + 0.50 C keyframe C

animation

“run”

t = 0 t = 1 t = 2 t = 3 t = 4 t = 5 t = 6

stored pose

interpolated pose (on the fly)

Interpolation of poses (at runtime):

transition between animations

 Eg: from stance to run

keyframe A 0.75 A + 0.25 B 0.50 A + 0.50 B 0.25 A + 0.75 B keyframe B

0.67 B + 0.33 C animation X

“stance”

t = 0 t = 1 t = 2 t = 3 t = 4 t = 5

0.33 B + 0.67 C keyframe C t = 6

t = 7

keyframe D 0.50 D + 0.50 E

0.75 E + 0.25 F 0.50 E + 0.50 F

t = 0+k t = 1+k t = 2+k t = 3+k t = 4+k t = 5+k 0.25 E + 0.75 F keyframe F t = 6+k

t = 7+k delay k = 3 animation Y

“run”

keyframe E

0.80 X + 0.20 Y 0.60 X + 0.40 Y 0.40 X + 0.60 Y 0.20 X + 0.80 Y 0.25 A + 0.75 B

keyframe B 0.67 B + 0.33 C 0.33 B + 0.67 C keyframe C

keyframe D 0.50 D + 0.50 E

0.75 E + 0.25 F 0.50 E + 0.50 F keyframe E

Interpolation of poses (at runtime):

transition between animations

 Eg: from stance to run

keyframe A 0.75 A + 0.25 B 0.50 A + 0.50 B animation X

“stance”

t = 0 t = 1 t = 2 t = 3 t = 4 t = 5 t = 6

t = 7 0.50 E + 0.50 F

t = 0+k t = 1+k t = 2+k t = 3+k t = 4+k t = 5+k 0.25 E + 0.75 F keyframe F t = 6+k

t = 7+k delay k = 3 animation Y

“run”

0.25 A + 0.75 B keyframe B

0.67 B + 0.33 C 0.33 B + 0.67 C keyframe C

keyframe D 0.50 D + 0.50 E

0.75 E + 0.25 F keyframe E

Interpolation of key-frames (in skeletal animation)



Pose interpolation is so powerful that

 a decent run-cycle (or walk cycle) can be made from as few as TWO POSES

 (and their mirrored)

 a decent sword-swing animation can be made from as few as TWO POSES

 charge pose, swing-delivered pose

 don’t try that with any other kind of animation!

 clearly, more key-frames

= more control

= better results

Storing poses (in GPU or CPU) Dual Quaternion Skinning



A variant:

per-bone transforms are stored as

instead of 4x3 matrices

 isometries only

 less RAM (8 floats per transform)

 (arguably) better quality

 (better interpolation)

 > GPU cost

 (necessary ops: ~ +50%)



LBS (Linear Blend Skinning) or DQS?

 a call of the game engine

Forward kinematic VS inverse kinematic



(in robotics)

Forward kinematics:

«from angles (and dists) to (x,y)»

VS

Inverse kinematics:

«from (x,y) (and dists) to angles»

Inverse Kinematic (IK):

an useful tool

 Forwardkinematics:

 “given local transfer P1, P2… PN, (as angles)

where does the foot go?”

 (one solution, trivial)

 Inversekinematics

 “if I need the foot to be in pos p, how to set

P1, P2… PN, ?”

 (under constraints, e.g. DoF at joints)

 (> 1 solutions, or none, and not trivial)

pelvis (root)

spine 1

right shoulder

shoulderleft P²

P4 P6

right left leg

leg

right calf spine 2

P3 P7

right neck foot

P5 P8

P3

P1

Inverse Kinematic (IK):

an useful tool



Two uses:

 in preprocessing (helping the task of the animator)

 in real time (performed by the game engine)



Examples of real-time uses:

 Exact positioning of feet on ground

 Exact positioning of hand on object to be grabbed

 Guarantee that hands are joined

(e.g. during 2-handed weapon wielding)

 even if this holds at keyframes, can be lost at interpolations

 can be invalidated at keyframes, due to retargeting (different bone lengths in the skeleton of this character)

 etc.

Third person view

view space

V

T¹

= ( T1 x T2 )^-1

= ( T² )^-1 x ( T¹ )^-1

camera T2

world space

First person view

world space T1

torso 1

shoulder R

shoulder L

P3

P8 P10

legR legL

calfR torso 2

P4

P7

footR neck

P9

P11

P2

P1

head

(root)hip

P12

V

camera

T2 T2 = «mid-eye» positioning

(in «head» space)

busto 1

spalla dx spalla sx

P3

P8 P10

gamba gamba dx

sx

polpaccio dx busto 2

P4

P7

piede Dx collo

P9

P11 P2

P1

testa bacino (root)

P12

Compositing poses

(

and animations)

+ =

lower joints

upper joints

busto 1

spalla dx spalla sx

P3

P8 P10

gamba gamba dx

sx polpaccio

dx busto 2

P4

P7

piede Dx collo

P9

P11 P2 P1

testa bacino

(root)

P12

busto 1

spalla dx spalla sx

P3

P8 P10

gamba gamba dx

sx polpaccio

dx busto 2

P4

P7

piede Dx collo

P9

P11 P2 P1

testa bacino (root)

P12

Pose A Pose B New Pose

busto 1

spalla dx spalla sx

P3

P8 P10

gamba gamba dx

sx polpaccio

dx busto 2

P4

P7

piede Dx collo

P9

P11 P2

P1

testa bacino (root)

P12

Compositing poses

(

and animations)

+ =

lower joints

upper joints

busto 1

spalla dx spalla sx

P3

P8 P10

gamba gamba dx

sx

polpaccio dx busto 2

P4

P7

piede Dx collo

P9

P11 P2 P1

testa bacino

(root)

P12

busto 1

spalla dx spalla sx

P3

P8 P10

gamba gamba dx

sx polpaccio

dx busto 2

P4

P7

piede Dx collo

P9

P11 P2 P1

testa bacino (root)

P12

P¹ = 0.45 ∙ + 0.55 ∙ ^{P1 P1}

also, interpolating, e.g.:

Pose A Pose B

Compositing poses

(

and animations)



Useful in different contexts:

 different character parts following different ani

 e.g. Lower body: run.

Upper body: aims/shoots/reload)



Note:

requires updating the final transformations

 (after changing the local ones)



Implementation note (Unity):

 Unity does this with “Layers” in Animation Controller

A few (pre)processing

task on skeletal animations



Compression

 input:ani with N keyframes

 output: ani with M<N keyframes



Retargeting

 input:Rig1 + (Skel animat for Rig1) + Rig2

 output: (Skel animat for Rig 2)



Building from a blend-shape animation

 input:Blend-shape

 output: Rig + Skinned Mesh + Anim

 (note: the opposite is a trivial «baking» procedure)

Compression of skeletal animations



Objective: remove keyframes

 the “redundant” ones

 preprocessing task (e.g. as a game tool)



Basic algorithm concept:

 for each keyframe Px

 tentatively remove Px

 compute interpolated version Pi from remaining keyframes

 (the prev and next ones)

 if distance( Pi ,Px)> MAX_ERR then reinsert keyframe Px

Skeletal animations:

open challenges



They are cheap + very convenient, but…



Vertices never exactly go where the artists (the skinner) would want them to…

 artifacts:

 “candy wrap” effect (especially with LBS)

 unwanted bulge-out effect (especially with DQS)

 limited expressiveness: difficult/impossible to get:

 muscle bulging,

 volume preservation (of tissues)

 skin area preservation (dynamic wrinkles)

 contact surfaces (e.g. skin pressed against skin)…

Blend-shapes Animations VS Skeletal-Animations

 Blend-shape anim.

 can interpolate keyframes (but linear trajectories)

 artist(modeler) has full, direct control over shape

 heavy in RAM

 replications of positions (+ normals)

 light to render (computation-wise)

 Skeletal animations

 can interpolate keyframes…

better (curved trajectories)

 artist(skinner) just tries to get desired shape (for all poses)

 light in RAM

 big plus: animations / models orthogonality !

 some overhead required

 transform interpolation (x vert!)

 update of final transforms before usage

note: it can be baked into a

Animations in games

(of 3D Solid Objects)

Procedural Non-Procedural

Rigid

Articulated

Free form

Skeletal Animations

Blend-Shapes

Rigid body dynamics Kinematic

animation

(ASSETS) (PHYSIC ENGINE)

Ragdolling Inverse kinematics

(generic) deformable object

simulation usually too expensive

Cloth/

garments

Ropes

Non-procedural Animations:

which one to pick? Example.

say our game

needs a model

capable of

doing this:

Non-procedural Animations:

which one to pick? Example.

Kinematic animation solution 1:

“wing” mesh

x2 instances

“hull”

mesh

“wing” mesh

x2 instances

“windscreen”

mesh

Non-procedural Animations:

which one to pick? Example.

Kinematic animation

wing B wing

A

ship space

Te

Ta

Tb Tc Td

Tship

solution 1:

rest of scene scene graph:

w.s. mesh wing

D wing

C

hull mesh

wind- screen

wing mesh animate these!

instance

Non-procedural Animations:

which one to pick? Example.

Skeletal animation

wing D

bone

wind- screen

bone

wing B

bone

wing C

bone

wing A

bone

hull

bone

Tship

solution 2:

x-wing rig (6 bones)

skeletal animation A

x-wing skinned mesh

skeletal animation B

Non-procedural Animations:

which one to pick? Example.

Blend- shape solution 3:

“x-wing” blend-shape

base shape morph 1 morph 2

Non-procedural Animations:

which one to pick? Example.

 Kinematic Animation

 assets: 3 (rigid) meshes for subparts, composed in a scene-graph subtree

 how: 5 instances, animate scene-graph transforms

 bonus: can reuse geometry for all wings: most compact on RAM

 simpler rendering

 5 draw calls (if very low poly: draw calls wasted for a few triangles)

 Skeletal animation:

 assets: one rig + one skinned mesh + few skeletal animations (1 pose each)

 mesh skinning: 1-bone-per-vertex enough in this case

 more taxing rendering (a bit)

 real-time GPU skinning

 single draw call!

 Blend shapes:

 assets: blend shape with one base shape + 2 morphs

 minimal GPU cost

 worst quality interpolation: linear

 vertices on straight paths (unless, more shapes added)

 heaviest on RAM

 (a waste of DoF!)

 not important, if very low res

 single draw call!

 but to different buffers each time / or to a larger buffer

straight (non curved) paths

Animations in games

(of 3D Solid Objects)

Procedural Non-Procedural

Rigid

Articulated

Free form

Skeletal Animations

Blend-Shapes

Rigid body dynamics Pre-made

transforms

(ASSETS) (PHYSIC ENGINE)

Ragdolling Inverse kinematics

(generic) deformable object

simulation usually too expensive

Cloth/

garments

Ropes

“G e om . C a ch es ”

Geometry Caches

(for lack of a better name)



Baked, optimized animations

 of a mixture of types e.g.

 blend shapes

 kinematic animatios

 (approximated)

 skinned animations

 (often, no hierarchy: just global transf)

 optimized

 compressed, streamed…

 Baking of a variety of simulations results

.abc most used file format:

by

Geometry Caches

(for lack of a better name)

 Baked, optimized animations

 of the appropriate types including mixtures

Geometry Caches

(a subset of Alembic) by

as Pre-made Tansforms:

Meshes: 170 Data rate: 0.13 MB/s Draw calls: 170 (same ones each frame) Input:

170 Meshes 88400 Verts

as a Blend Shape:

Meshes: 1, with N shapes Data rate: 4.3 MB/s Draw calls: 1

(different one each frame)

as a Skeletal Animation Meshes: 1, w skinning (*) Data rate: 0.13 MB/s Draw calls: 1 (same one each frame) (*)just 1 bone per vertex

Articulated Animations in Unity (+Mecanim) (notes)

 Assets (models, animation, skeletons) imported as formats:

 fbx, collada

 Animation compression

 available during import / builds

 auto reduction of: num of links per vertex, num of keyframes … :

 «Animator Controller» module  deals with:

 blending between animations: «transitions»

 compositing animations: «layers»

 e.g.: a layer overwrites upper body bones only

 and is nicely WYSIWYG (graph visualization)

 Inverse Kinematic: with scripts ( Avatar.SetIKPoistion)

 Skeletons can be:

 custom (imported as assets from any source)

 built-in standard humanoid skeleton provided

 (with 20-something bones)

 simplifies: rigging (predefined constrains), layers (predef. labelling)

Blend-shapes can be skinned

 Blend-shapes

and skeletal animations can be (and are) easily used together

 blend-shapes = multiple (rest-pose) geometries

 skinning = per-vertex bone-links

 no incompatibility: a mesh can have both per-vertex attributes!

 Many usages, e.g. for characters:

 slim/fat version for same animated character / outfit

 blend shapes = breathing (chest expands) skel-animations = actions

 faces: blend shapes = cheeks, eyebrows movements skel-animations = mandible bone

 see: “blend-shape correctives” (currently, not for games)