2022-05-07

pyjom producer

video and audio needs to be analysised separately.

audio can be processed by chunks, splited tracks, while video can be itered frame by frame.

Video Database For Video Generation

A fastai/PyTorch package for unpaired image-to-image translation.

https://github.com/tmabraham/UPIT?auto_subscribed=false&email_source=explore

视听分割视频注意力机制

only segment video objects that make sounds, video/audio combined segmentation:

https://github.com/OpenNLPLab/AVSBench

video object tracking and segmentation unified framework:

https://github.com/MasterBin-IIAU/Unicorn

video object segmentation handle long video with ease:

https://github.com/hkchengrex/XMem

when removing video watermarks, remember to ease in/out. that is said, do not stop blurring immediately after the end mark. instead, extend the blur time and decrease blur level incrementally. also, the blur ease-in is needed for the start mark, blur ahead of the start mark and ease in incrementally.

descriptive information generation from video/image:

https://github.com/BAAI-WuDao/CogView

https://github.com/BAAI-WuDao/BriVL

https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/install.md

video understanding/captioning:

https://github.com/rohit-gupta/Video2Language

https://github.com/byeongjokim/Automatic-Baseball-Commentary-Generation-Using-DeepLearning

https://github.com/shhdSU/Image_Captioning_DeepLearning

https://github.com/jayleicn/recurrent-transformer

https://github.com/terry-r123/Awesome-Captioning

https://github.com/vijayvee/video-captioning

https://github.com/scopeInfinity/Video2Description

https://github.com/xiadingZ/video-caption.pytorch

https://github.com/YehLi/xmodaler

https://github.com/sujiongming/awesome-video-understanding

action recognition:

https://github.com/mit-han-lab/temporal-shift-module

https://github.com/yjxiong/temporal-segment-networks

https://github.com/yjxiong/tsn-pytorch

https://github.com/open-mmlab/mmaction

https://github.com/jinwchoi/awesome-action-recognition

The data remaining only have texts, danmaku, likes, titles, intros, comments, tags, image/video analysis results(short description). You can only generate video from generated metadata or given rules. Find similar words, similar danmaku, similar features, comments or the inverse, according to the selected topic and main idea.

Analyze video when downloaded, mark its highlights, analyze texts and danmaku. Get video segments and audio segments.

Collect pictures/videos with given rules, namely finding the head of somebody, with how many likes, keywords.

Split audio and grab the main speaker. clone the voice and perhaps changes the gender.

Split video and do human/image segmentation if human/target is found. put it onto another human/target’s background masking the original human, with similar areas and movements.

Analyze video with off-topic(offline) and of-topic(online) sources.

Remove watermark according to username.

Generate danmaku and generate video accordingly. Generate texts and generate video accordingly. Doing faceswap, talking head and human/image segmentation accordingly.

2022-05-03

Gpt-2 以及文本生成

免费gpt文本生成：彩云小梦以及小梦海外版

小梦的中文文本有涉及政治的检测器不能把敏感内容塞进小梦

对话生成

https://huggingface.co/thu-coai/CDial-GPT_LCCC-large/tree/main

twitter generator inspired by influencers:

https://github.com/gdemos01/TwitterInfluencerAI

chinese LM

清华130b大模型

测试地址：https://huggingface.co/spaces/THUDM/GLM-130B

模型仓库：https://github.com/THUDM/GLM-130B

https://github.com/Morizeyao/GPT2-Chinese

https://zhuanlan.zhihu.com/p/352028922

https://github.com/TsinghuaAI/CPM-1-Finetune

https://github.com/TsinghuaAI/CPM-1-Generate

https://github.com/TsinghuaAI/CPM-2-Pretrain

gpt2/cpm tutorial:

https://www.cnblogs.com/wwj99/p/12503545.html

https://github.com/OpenBMB/BMInf-demos

https://bmtrain.readthedocs.io/en/latest/index.html

2022-04-25

Content Usage

Use the original transcript for paraphrasing, while using danmaku for joke generation.

2022-04-21

Mmdetection And Mmd Dancing

3d 虚拟形象动作生成视频生成虚拟偶像 Vtuber:

https://github.com/xianfei/SysMocap

human pose detection:

https://github.com/facebookresearch/VideoPose3D

opengl recording:

https://lencerf.github.io/post/2019-09-21-save-the-opengl-rendering-to-image-file/

http://www.songho.ca/opengl/gl_pbo.html#pack

https://stackoverflow.com/questions/7634966/save-opengl-rendering-to-video

https://www.codeproject.com/articles/15941/recording-directx-and-opengl-rendered-animations

https://www.glfw.org/documentation.html

download expose models:

https://expose.is.tue.mpg.de/downloads

smpl-x model download:

https://smpl-x.is.tue.mpg.de/download.php

model zoo:

https://github.com/Zhongdao/Towards-Realtime-MOT/blob/master/DATASET_ZOO.md

mmd auto tracking:

https://github.com/errno-mmd/mmdmatic/blob/master/setup.bat

https://github.com/miu200521358/expose_mmd

https://github.com/miu200521358/AlphaPose-MMD

smplx expose alternative body tracker:

https://github.com/vchoutas/smplx

face tracking:

https://github.com/Aditya-Khadilkar/Face-tracking-with-Anime-characters

anime face detector:

https://github.com/nagadomi/lbpcascade_animeface

https://github.com/qhgz2013/anime-face-detector

anime facial features:

https://github.com/pranau97/anime-detection

repair anime images:

https://github.com/youyuge34/Anime-InPainting

paint manga from sketch (with color blocks):

https://github.com/youyuge34/PI-REC

if we can re-trace the action/expression done by vtubers, we can monetize those “highlight cuts”.

you can firstly find points in datasets and then generate mmd videos, and then create trainset. you can also generate pose from raw video and then create dataset.

found occasionally when browsing MMD, but found this with so many stars, which is an instance detection/segmentation library.

https://github.com/open-mmlab/mmdetection

while rendering mmd can be done with mmd viewer like https://github.com/benikabocha/saba or could use renderer like blender or unity. we must bake physics before dancing.

found other dedicated renderer for mmd, with bullet physics:

https://github.com/jinfagang/mmc

found interesting repo of poetry composing:

https://github.com/jinfagang/tensorflow_poems

mediapipe/paddlevideo alike:

https://pypi.org/project/alfred-py/

three.js has multiple loaders:

https://github.com/mrdoob/three.js/tree/dev/examples/js/loaders

https://github.com/hanakla/three-mmd-loader

render MMD using saba lib:

https://github.com/WLiangJun/MMD-Desktop-mascot

https://github.com/miu200521358/expose_mmd/fork

music based dance:

https://github.com/DeepVTuber/DanceNet3D

https://github.com/ColbyZhuang/music2dance_DanceNet

https://github.com/caijianfei/Music2Dance

characters:

https://www.mixamo.com/#/?page=1&type=Character

2022-04-21

Articulated Animation

This one is dead simple. Use real human to talk as you go.

https://github.com/snap-research/articulated-animation

2022-03-31

Install Vscode Extensions By Id

@id:medialang.vscode-theme-medialang-seti

＠id:medialang.medialang-highlighter

2022-03-31

Converting Partial Zhihu Viral Content

just add a trailing sentence.

请听下回分解。

2022-02-10

Python Media Automation

we first see the world, get the observation and respond in the form of content. it is a feedback loop.

to search components in videos, first take screenshots then do image search, then use the keywords to get the source video.

breakdown approach:

granualize every step, showing all possibilities to get content created and then optimoze it using standards.

filter approach:

establish some topics, create topic specific approaches to arrange the content, choose the best among all topics.

are they compatible? are you sure it is modular, scalable and extensible?

for novices, they have few unpolished ideas and waiting to realize it using code. but it lacks the feedback loop and thus you are unable to change yourself according to the reaction. breakdown approach must be used to automate the optimization, and topic based approach is simple at first hand.

to avoid copyright issues search for google.

topic based approach assues the public always have something in common and thus you only search specific things at first hand. they are easy to control, static and consistent. breakdown approach is where the evolution begins.

let’s assume our topic is about pets on weibo. pets have different kinds and the content creaters are different from each other. all we do is to download and upload. we get descriptions from our viewers, video play counts and various feedback. we improve the source by our feedback, searching for more untouched contents and more mixes like video/audio crossing.

breakdown approach is demostrated first-hand with our actor-critic model. we first view all possible posts from all sources, find what’s interesting and repost it to our target platform. this is likely to be cheating. we again choose our sources, our approach of modification based on feedback. topics are generated from the very first step.

the model of interests, which generates the topic, is the key breakdown approach. we have to eventually construct a breakdown approach to boost our searches in every aspect. feedback is one of those key features. we eventually have to view the content with the machine. suggest using the breakdown approach now.

anatomy of the post:

first thing it would be postable, according to our mandatory order. it would not be taken down or banned for a long time. banning detection is required and usually simple to test against.

second it is most profitable. we only prefer those tasks which give the most output. occasionly we choose something fresh despite lower expectations.

third it would be resourceful. consistently pinning audience in a series of videos is undoubtably competitent. this can be reached by utilizing our creativity engine based on comments and imagination, realize the unrealized.

have not yet found anything systematic on giving the full detail of such automated content creation system. we only pick up those pieces. it is important to make the entire design flexible and create miniature tests to fabricate the system. like any other famous writer/director, you could only name it but not reproduce it.

hands on the approach, no matter it is inspired by anyone or anything, it is time to begin, to complete the feedback loop.

not a pipe, but a loop.

we demonstrate the loop using fake data, then the real ones. maybe the initial topic is also meant to be fake data. the real world data is too stochastic for us to imagine. better construct something specific.

Cats video with lyrics (Algorithm)

Finally, the cats.

To harvest video by tags, authors, categorize them, count the views divided by time, model the predicted views and select only the best, original videos to upload.

The views on other platforms does not matter and views on other channels does not matter, in the long term. We might try to embrace them because of this number seen on others but we need real feedback on our channels.

Filters can be generated by regular expressions, common patterns found in text, and indexed. Filters generated from data. We index filters, record them according to views.

I am selling myself on cheap. Maybe not.

Using the same strategy of searching the web.

api of weibo (this is news!):

http://sinanews.sina.cn/interface/type_of_search.d.html?callback=initFeed&keyword=%E6%98%8E%E6%98%9F&page=1&type=siftWb&size=20&newpage=0&chwm=&imei=&token=&did=&from=&oldchwm=

response:

we need videos. you download it from baidu ai cloud.

we need feedback. we need this since this is how we try things.

anything can be hackish, especially for this damn “we media”. anything can be popular, from basic bash script to regular expressions, based on how you looking at it. we shall not feed our audience full of something than another unless we know for sure it is our target.

it is the power of the mirror! the feedback!

create a fake, dummy project full of skeletons and then realize every step till we are done.

pyjom

2022-05-07

pyjom producer

2022-05-05

Video Database

Video Database For Video Generation

2022-05-03

Gpt-2 以及文本生成

2022-04-25

Content Usage

2022-04-21

Mmdetection And Mmd Dancing

2022-04-21

Articulated Animation

2022-03-31

Install Vscode Extensions By Id

2022-03-31

Converting Partial Zhihu Viral Content

2022-02-10

Python Media Automation

2021-12-19

Cats Video With Lyrics_2

Cats video with lyrics (Algorithm)

Links

pyjom

2022-05-07 pyjom producer

2022-05-05 Video Database

Video Database For Video Generation

2022-05-03 Gpt-2 以及文本生成

2022-04-25 Content Usage

2022-04-21 Mmdetection And Mmd Dancing

2022-04-21 Articulated Animation

2022-03-31 Install Vscode Extensions By Id

2022-03-31 Converting Partial Zhihu Viral Content

2022-02-10 Python Media Automation

2021-12-19 Cats Video With Lyrics_2

Cats video with lyrics (Algorithm)

Links

2022-05-07

pyjom producer

2022-05-05

Video Database

2022-05-03

Gpt-2 以及文本生成

2022-04-25

Content Usage

2022-04-21

Mmdetection And Mmd Dancing

2022-04-21

Articulated Animation

2022-03-31

Install Vscode Extensions By Id

2022-03-31

Converting Partial Zhihu Viral Content

2022-02-10

Python Media Automation

2021-12-19

Cats Video With Lyrics_2