Sysmocap

WHAT I WANT FOR (or nearly) requires real 3d models, written in javascript

cannot output video?

A cross-platform real-time video-driven motion capture and 3D virtual character rendering system for VTuber/Live/AR/VR.

Does not require a discrete graphics card and runs smoothly even on eight-year-old computers

search for “vtuber” along with “motion capture” you will get many head-only trackers and renderers for windows but not linux, also some “broadcast templates/frameworks”. many support one single image (anime head + remove background) as input instead of 2d/3d models

face tracking only, showing face, mouth and eyes, head directions, bind to live2d models

虚拟数字人 metahuman

NextHuman Beta0.9上线公测，5分钟高品质讲解，带你进入数字人“零门槛”创作新时代，体验直通车 -> https://nexthuman.cn 免费版是Windows上面跑的需要高端1070显卡

anime character segmentation

to remove false positives, make sure we have anime face in view, otherwise mark it as a false positive.

you can use anime character recognition like moeflow or opencv anime face detector along with some phash or perceptual hash library to group similar characters, compare perceptual image similarity and line them up in a series.

aniseg, able to segment anime character and head, using mask-rcnn

yet another anime character segmentation model using solov2 and condinst

waifu segmentation

high accuracy anime character segmentation

自动画漫画画几笔就成某个人像动漫头像

https://menyifang.github.io/projects/DCTNet/DCTNet.html

自动捏脸 gan给人脸戴口罩

https://github.com/futscdav/Chunkmogrify

selfie to anime, picture to anime photos

selfie2anime with trained models

##原神mmd下载模型

模之屋（需要注册）：

https://www.aplaybox.com/u/680828836

夕蓝资源网（可直接下载）也有其他的3d模型可以下载：

https://www.seoliye.com/tags/53.html

use voice to power up static images

voice powered animated cartoon figure

jeeliz (some web deep learning runtime, like tensorflow.js) powered

weboji, highly similar to animoji, with three.js and cute fox avatar

face filter, alter the face like putting glass, minor changes to avoid privacy/copyright concerns?

openface

facial features extraction

facerig

facerig location: /Software/Program Files (x86)/FaceRig

i’ve seen python code inside facerig.

facerig does not offer head-only rendering, but that could be changed i suppose?

avatarify python

infinite avatars by using style gan, first order motion model

create static portrait avatar (svg?)

animoji from apple

facial landmark detection in python, animoji-animate

animoji apple private framework 实际上这个就是之前看到的会动的狗屎的视频来源

2d模型皮套可动虚拟Vtuber talking head

https://github.com/yuyuyzl/EasyVtuber

https://github.com/pkhungurn/talking-head-anime-3-demo

https://github.com/GunwooHan/EasyVtuber

b站官方

直播姬现在支持2d面部捕捉 3d模型动作捕捉

直播姬版本有windows macos(m1) Android版本

2d模型是live2d的模型

有待研究

the still image to singing face bot, lip-sync video generation

sadtalker

wombo.ai, likely to be talking head or yanderifier

https://github.com/mchong6/GANsNRoses/

https://github.com/williamyang1991/VToonify

生成高质量的艺术人像视频是计算机图形学和视觉中一项重要且理想的任务。虽然已经提出了一系列基于强大的 StyleGAN 成功的人像图像卡通化模型，但这些面向图像的方法在应用于视频时存在明显的局限性，在这项工作中，我们通过引入一种新颖的 VToonify 框架来研究具有挑战性的可控高分辨率肖像视频风格迁移。具体来说，VToonify 利用StyleGAN 的中高分辨率层基于编码器提取的多尺度内容特征来渲染高质量的艺术肖像，以更好地保留帧细节。作为输入，有助于输出具有自然运动的完整面部区域。 amework 与现有的基于 StyleGAN 的图像卡通化模型兼容，以将其扩展到视频卡通化，并继承了这些模型的吸引人的特性，可灵活地控制颜色和强度。这项工作展示了基于 Toonify 和 DualStyleGAN 的 VToonify 的两个实例，用于基于集合广泛的实验结果证明了我们提出的 VToonify 框架在生成具有灵活风格控制的高质量和时间连贯的艺术肖像视频方面优于现有方法的有效性

all in one colab text to talking face generation, also consider paddlespeech example:

https://github.com/ChintanTrivedi/ask-fake-ai-karen

avaliable from paddlegan as an example used in paddlespeech, the artificial host.

lip-sync accurate wav2lip:

https://github.com/Rudrabha/Wav2Lip

lipgan generate realistic lip-sync talking head animation(fully_pythonic branch or google colab notebook):

https://github.com/Rudrabha/LipGAN

google’s lipsync implementation, using tensorflow facemesh:

https://github.com/google/lipsync

https://lipsync.withyoutube.com/

https://github.com/tensorflow/tfjs-models/tree/master/facemesh

network reverse engineering for wombo.ai: