2022-07-10

Video Effects Transitions

when removing video watermarks, remember to ease in/out. that is said, do not stop blurring immediately after the end mark. instead, extend the blur time and decrease blur level incrementally. also, the blur ease-in is needed for the start mark, blur ahead of the start mark and ease in incrementally.

descriptive information generation from video/image:

https://github.com/BAAI-WuDao/CogView

https://github.com/BAAI-WuDao/BriVL

https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/install.md

video understanding/captioning:

https://github.com/rohit-gupta/Video2Language

https://github.com/byeongjokim/Automatic-Baseball-Commentary-Generation-Using-DeepLearning

https://github.com/shhdSU/Image_Captioning_DeepLearning

https://github.com/jayleicn/recurrent-transformer

https://github.com/terry-r123/Awesome-Captioning

https://github.com/vijayvee/video-captioning

https://github.com/scopeInfinity/Video2Description

https://github.com/xiadingZ/video-caption.pytorch

https://github.com/YehLi/xmodaler

https://github.com/sujiongming/awesome-video-understanding

action recognition:

https://github.com/mit-han-lab/temporal-shift-module

https://github.com/yjxiong/temporal-segment-networks

https://github.com/yjxiong/tsn-pytorch

https://github.com/open-mmlab/mmaction

https://github.com/jinwchoi/awesome-action-recognition

The data remaining only have texts, danmaku, likes, titles, intros, comments, tags, image/video analysis results(short description). You can only generate video from generated metadata or given rules. Find similar words, similar danmaku, similar features, comments or the inverse, according to the selected topic and main idea.

Analyze video when downloaded, mark its highlights, analyze texts and danmaku. Get video segments and audio segments.

Collect pictures/videos with given rules, namely finding the head of somebody, with how many likes, keywords.

Split audio and grab the main speaker. clone the voice and perhaps changes the gender.

Split video and do human/image segmentation if human/target is found. put it onto another human/target’s background masking the original human, with similar areas and movements.

Analyze video with off-topic(offline) and of-topic(online) sources.

Remove watermark according to username.

Generate danmaku and generate video accordingly. Generate texts and generate video accordingly. Doing faceswap, talking head and human/image segmentation accordingly.

2022-04-21

Mmdetection And Mmd Dancing

3d 虚拟形象动作生成视频生成虚拟偶像 Vtuber:

https://github.com/xianfei/SysMocap

human pose detection:

https://github.com/facebookresearch/VideoPose3D

opengl recording:

https://lencerf.github.io/post/2019-09-21-save-the-opengl-rendering-to-image-file/

http://www.songho.ca/opengl/gl_pbo.html#pack

https://stackoverflow.com/questions/7634966/save-opengl-rendering-to-video

https://www.codeproject.com/articles/15941/recording-directx-and-opengl-rendered-animations

https://www.glfw.org/documentation.html

download expose models:

https://expose.is.tue.mpg.de/downloads

smpl-x model download:

https://smpl-x.is.tue.mpg.de/download.php

model zoo:

https://github.com/Zhongdao/Towards-Realtime-MOT/blob/master/DATASET_ZOO.md

mmd auto tracking:

https://github.com/errno-mmd/mmdmatic/blob/master/setup.bat

https://github.com/miu200521358/expose_mmd

https://github.com/miu200521358/AlphaPose-MMD

smplx expose alternative body tracker:

https://github.com/vchoutas/smplx

face tracking:

https://github.com/Aditya-Khadilkar/Face-tracking-with-Anime-characters

anime face detector:

https://github.com/nagadomi/lbpcascade_animeface

https://github.com/qhgz2013/anime-face-detector

anime facial features:

https://github.com/pranau97/anime-detection

repair anime images:

https://github.com/youyuge34/Anime-InPainting

paint manga from sketch (with color blocks):

https://github.com/youyuge34/PI-REC

if we can re-trace the action/expression done by vtubers, we can monetize those “highlight cuts”.

you can firstly find points in datasets and then generate mmd videos, and then create trainset. you can also generate pose from raw video and then create dataset.

found occasionally when browsing MMD, but found this with so many stars, which is an instance detection/segmentation library.

https://github.com/open-mmlab/mmdetection

while rendering mmd can be done with mmd viewer like https://github.com/benikabocha/saba or could use renderer like blender or unity. we must bake physics before dancing.

found other dedicated renderer for mmd, with bullet physics:

https://github.com/jinfagang/mmc

found interesting repo of poetry composing:

https://github.com/jinfagang/tensorflow_poems

mediapipe/paddlevideo alike:

https://pypi.org/project/alfred-py/

three.js has multiple loaders:

https://github.com/mrdoob/three.js/tree/dev/examples/js/loaders

https://github.com/hanakla/three-mmd-loader

render MMD using saba lib:

https://github.com/WLiangJun/MMD-Desktop-mascot

https://github.com/miu200521358/expose_mmd/fork