video QA: https://github.com/OpenGVLab/InternVideo

https://github.com/jingvsai/YOLOv8-GUI with bytetrack, deepsort

identify the object first, then do tracking.

if detection overlaps with object, then do not launch new tracking process.

OpenCV Meanshift Algorithm for Object Tracking

OpenCV CAMshift Algorithm for Object Tracking

OpenCV Optical Flow Algorithm for Object Tracking

OpenCV Object Detection and Tracking

motion verctor estimation, motion vector export, ffmpeg advanced usage

use cases

to detect hard-coded subtitles, crop the region and detect sudden changes

can also use pyscenedetect to do the job

pyav

docs

1
2

pip3 install av

remove/detect silence

… silencedetect A->A Detect silence.

… silenceremove A->A Remove silence.

frame interpolate

1
2
3
4

ffmpeg -y -i "/root/Desktop/works/pyjom/tests/random_giphy_gifs/samoyed.gif" \
-vf "minterpolate,scale=w=iw*2:h=ih*2:flags=lanczos,hqdn3d" \
-r 60 ffmpeg_samoyed.mp4

motion estimation

to get mosaic motion vectors and visualize:

1
2
3
4

ffmpeg -i "/root/Desktop/works/pyjom/tests/random_giphy_gifs/samoyed.gif" \
-vf "mestimate=epzs:mb_size=16:search_param=7, codecview=mv=pf+bf+bb"  \
mestimate_output.mp4 -y

get help

on specific filter:

1
2

ffmpeg -h filter=showspectrumpic

on all filters:

1
2

ffmpeg -filters

crop detection, picture in picture (PIP) detection

1
2
3
4

ffmpeg -i "/root/Desktop/works/pyjom/samples/video/LiEIfnsvn.mp4" \
-vf "mestimate,cropdetect=mode=mvedges,metadata=mode=print" \
-f null -

scene change detection

1
2
3
4
5
6

ffmpeg -hide_banner -i "$file" -an \
-filter:v \
"select='gt(scene,0.2)',showinfo" \
-f null \
- 2>&1

extract motion vectors

ffmpeg can produce motion vector estimation but it is not exportable, only for internal use.

mp4 format provides motion vector information thus maybe we need not to use GPU to get those ‘optical flow’ data.

extract by using ffmpeg apis

mv-extractor Extract frames and motion vectors from H.264 and MPEG-4 encoded video.

extract from mp4 file

mpegflow for easy extraction of motion vectors stored in video files

mv-tractus: A simple tool to extract motion vectors from h264 encoded videos.

take screenshot at time:

1
2

ffmpeg -ss 01:10:35 -i invideo.mp4 -vframes 1 -q:v 3 screenshot.jpg

video denoise filters:

dctdnoiz fftdnoiz hqdn3d nlmeans owdenoise removegrain vaguedenoiser nlmeans_opencl yaepblur

super-resolution, resampling:

deeplearning model, tensorflow

1
2
3
4
5
6

env LD_LIBRARY_PATH=/root/anaconda3/pkgs/cudatoolkit-10.0.130-0/lib/:/root/anaconda3/pkgs/cudnn-7.6.5-cuda10.0_0/lib/:$LD_LIBRARY_PATH \
ffmpeg -i "/root/Desktop/works/pyjom/samples/video/LiEIfnsvn.mp4" \
-y -vf \
"sr=dnn_backend=tensorflow:model=./sr/espcn.pb,yaepblur" \
supertest.mp4

use standard scale method:

1
2
3
4

ffmpeg -y -i "/root/Desktop/works/pyjom/tests/random_giphy_gifs/samoyed.gif"\
-vf "minterpolate,scale=w=iw*2:h=ih*2:flags=lanczos,hqdn3d" \
-r 60 ffmpeg_samoyed.mp4

options:

‘fast_bilinear’

Select fast bilinear scaling algorithm.

‘bilinear’

Select bilinear scaling algorithm.

‘bicubic’

Select bicubic scaling algorithm.

‘experimental’

Select experimental scaling algorithm.

‘neighbor’

Select nearest neighbor rescaling algorithm.

‘area’

Select averaging area rescaling algorithm.

‘bicublin’

Select bicubic scaling algorithm for the luma component, bilinear for chroma components.

‘gauss’

Select Gaussian rescaling algorithm.

‘sinc’

Select sinc rescaling algorithm.

‘lanczos’

Select Lanczos rescaling algorithm. The default width (alpha) is 3 and can be changed by setting param0.

‘spline’

Select natural bicubic spline rescaling algorithm.

‘print_info’

Enable printing/debug logging.

‘accurate_rnd’

Enable accurate rounding.

‘full_chroma_int’

Enable full chroma interpolation.

‘full_chroma_inp’

Select full chroma input.

‘bitexact’

Enable bitexact output.

nazobase NAZOrip basement, with cython dll docs

DBmbk a debanding toolkit, for easier bezier curve generation

ffmpeg super resolution filter could get faster if run on gpu with libtensorflow

VESPCN: real-time super resolution

mpv is a media player with VapourSynth built-in, and that’s probably how vapoursynth gets in my mac via brew dependency manager

view.py is Python module for vapoursynth scripts that previews clips

to use opencv functions with vapoursynth

svp is free on linux, offering plugin for vlc while vlc cannot be run as root

you might harvest some prebuilt binaries of vapoursynth plugins for linux

补帧算法可适用于我们的动态水印追踪系统 但是可能需要优化 才能做到比较快速的补帧 因为水印所在位置的区间实际上只是白色的 不需要过于复杂的网络 同时这种补出来的水印需要逐帧处理 或者两帧一处理 生成的区间数量会非常的多

it is much easier to do this on windows since we need quick evaluation. might run this on virtualbox?

build scripts on how to build plugins for macos, including how to configure the installation prefix.

brew compatible, macos compatible vapoursynth plugin build script provider: homebrew-vsplugins does not provide build scripts for all plugins avaliable for windows, and it requires additional linking

tutorial on how to configure it: (is it intel only?)

Alternative VapourSynth Install Method (Brew):

IMPORTANT: Brew users will need to create and set the autoload folder prior to installing VapourSynth! Simply run the following commands:

Code:

1
2
3
4
5
6

mkdir -p /usr/local/lib/vapoursynth
mkdir -p "$HOME/Library/Application Support/VapourSynth"
touch "$HOME/Library/Application Support/VapourSynth/vapoursynth.conf"
echo UserPluginDir=/usr/local/lib/vapoursynth >> "$HOME/Library/Application Support/VapourSynth/vapoursynth.conf"
echo SystemPluginDir=/usr/local/lib/vapoursynth >> "$HOME/Library/Application Support/VapourSynth/vapoursynth.conf"

(Optional) Create desktop shortcuts for the plugins and scripts folders. Run the following commands in terminal:

Code:

1
2
3
4

mkdir $HOME/Desktop/VapourSynth
ln -s /usr/local/lib/vapoursynth $HOME/Desktop/VapourSynth/Plugins
ln -s /usr/local/lib/python3.9/site-packages $HOME/Desktop/VapourSynth/Scripts

Use brew command:

Code:

1
2

brew install vapoursynth

bm3d denoising using cuda

fft3d denoising

python opencv 光流算法详解 分为sparse和dense两种 某种程度上都可以计算场景的变换激烈程度

frame interpolation using deep optical flow

openmmlab mmflow

google research: FILM (frame interpolation for large motion)

vapoursynth get started (official doc)

vapoursynth plugin database only provide prebuilt binaries for windows while the plugin source code might work with linux and macos (if it has the source code)

VSRepo plugin manager installing vapoursynth plugin via commandline tool and vsrepo is only supported on windows, for other platforms we need to compile plugins manually.

nazorip vapoursynth blogs

nazorip bezier curve

nazorip gamma curve and convolution

flowpy: tool for visualizing and processing image with optical flow

Image Restoration Upscaling Inpainting 图像修复 超分辨率

sota image inpainting: lama-cleaner still needs manual labeling on inpainting area

https://github.com/DmitryUlyanov/deep-image-prior

nas image prior

https://arxiv.org/abs/2008.11713

mmediting: OpenMMLab Image and Video Restoration, Editing and Generation Toolbox

Neuraldiff:

https://github.com/dichotomies/NeuralDiff?ref=pythonawesome.com

https://pythonawesome.com/official-pytorch-implementation-of-neuraldiff-segmenting-3d-objects-that-move-in-egocentric-videos/

AI上色 ffmpeg去特定颜色 调色

可能和GAN有关

use paddlegan for coloring

可以去掉血腥色情 暴力可能不行 需要剪辑

FFmpeg remove color:

https://video.stackexchange.com/questions/33588/using-ffmpeg-can-i-remove-the-color-from-an-area-of-the-video

http://johnriselvato.com/ffmpeg-how-to-remove-all-colors-except-one-from-a-video/

face coloring:

https://github.com/Xu-Justin/Grayscale-Face-Coloring

nvidia coloring:

https://developer.nvidia.com/blog/easily-colorize-black-and-white-photos-with-ai/

github topic on image colorization:

https://github.com/topics/image-colorization

github repos on image colorization:

https://github.com/rrupeshh/Auto-Colorization-Of-GrayScale-Image

https://github.com/aDouladiris/Grayscale-Image-Colorization

https://github.com/aakaashjois/Colorizing-Grayscale-Images

https://github.com/emilwallner/Coloring-greyscale-images

curated list on image colorization:

https://github.com/oskar-j/awesome-image-coloring

视频分析处理 视频摘要 剧本生成

自动抠像 最新 2022 较小的性能消耗：

https://github.com/hkchengrex/XMem

我fork的项目：https://github.com/ProphetHJK/XMem

我fork后添加了一些小工具，包括绿幕生成，蒙版视频生成，中文教程等

simple video captioning:

https://pythonawesome.com/a-simple-implementation-of-video-captioning/

https://github.com/232525/videocaptioning.pytorch?ref=pythonawesome.com

https://github.com/xiadingZ/video-caption.pytorch

3d cnn for video classification:

https://github.com/kcct-fujimotolab/3DCNN

end-to-end video image classification by facebook:

https://github.com/facebookresearch/ClassyVision

video understanding models and datasets:

https://github.com/sujiongming/awesome-video-understanding

video classification dataset:

​video_type_dict​ ​=​ {​’360VR’​: ​’VR’​, ​’4k’​: ​’4K’​, ​’Technology’​: ​’科技’​, ​’Sport’​: ​’运动’​, ​’Timelapse’​: ​’延时’​,

​’Aerial’​: ​’航拍’​, ​’Animals’​: ​’动物’​, ​’Sea’​: ​’大海’​, ​’Beach’​: ​’海滩’​, ​’space’​: ​’太空’​,

​’stars’​: ​’星空’​, ​’City’​: ​’城市’​, ​’Business’​: ​’商业’​, ​’Underwater’​: ​’水下摄影’​,

​’Wedding’​: ​’婚礼’​, ​’Archival’​: ​’档案’​, ​’Backgrounds’​: ​’背景’​, ​’Alpha Channel’​: ​’透明通道’​,

​’Intro’​: ​’开场’​, ​’Celebration’​: ​’庆典’​, ​’Clouds’​: ​’云彩’​, ​’Corporate’​: ​’企业’​,

​’Explosion’​: ​’爆炸’​, ​’Film’​: ​’电影镜头’​, ​’Green Screen’​: ​’绿幕’​, ​’Military’​: ​’军事’​,

​’Nature’​: ​’自然’​, ​’News’​: ​’新闻’​, ​’R3d’​: ​’R3d’​, ​’Romantic’​: ​’浪漫’​, ​’Abstract’​: ​’抽象’​}

https://github.com/yuanxiaosc/Multimodal-short-video-dataset-and-baseline-classification-model

rnn for human action recognization:

https://github.com/stuarteiffert/RNN-for-Human-Activity-Recognition-using-2D-Pose-Input

video script introduction and generation:

https://sharetxt.live/blog/how-to-generate-a-youtube-video-script-with-ai#:~:text=%20How%20to%20use%20Chibi.ai%20to%20create%20a,scan%20through%20your%20text%20and%20generate...%20More%20

fight detection using pose estimation and rnn:

https://github.com/imsoo/fight_detection

video summarizer to summarized video based on video feature:

https://github.com/Lalit-ai/Video-Summary-Generator

awesome action recognition:

https://github.com/jinwchoi/awesome-action-recognition

temporal model for video understanding:

https://github.com/mit-han-lab/temporal-shift-module

https://github.com/mit-han-lab/temporal-shift-module

https://github.com/yjxiong/tsn-pytorch

time space attention for video understanding(timesformer):

https://github.com/facebookresearch/TimeSformer

video understanding by alibaba:

https://github.com/alibaba-mmai-research/pytorch-video-understanding

video object segmentation:

https://github.com/yoxu515/aot-benchmark?ref=pythonawesome.com

video scene segmentation:

https://github.com/kakaobrain/bassl?ref=pythonawesome.com

mmaction detect actions in video:

https://pythonawesome.com/an-open-source-toolbox-for-video-understanding-based-on-pytorch/

https://github.com/open-mmlab/mmaction2

dense video captioning:

https://www.opensourceagenda.com/projects/dense-video-captioning-pytorch

https://www.opensourceagenda.com/projects/dense-video-captioning-pytorch

seq2seq video captioning:

https://blog.csdn.net/u013010889/article/details/80087601

2d cnn with LSTM video classification:

https://blog.csdn.net/qq_43493208/article/details/104387182

spp-net for image shape unification:

https://github.com/peace195/sppnet

https://github.com/yueruchen/sppnet-pytorch

running pretrained pytorchvideo video classification model from zoo:

https://pytorchvideo.org/docs/tutorial_torchhub_inference

pytorchvideo model zoo:

https://pytorchvideo.readthedocs.io/en/latest/model_zoo.html

(arxiv) end to end generative pretraining multimodal video captioning mv-gpt:

https://arxiv.org/abs/2201.08264v1

video captioning using encoder-decoder:

https://github.com/Shreyz-max/Video-Captioning

video captioning video2text keras implementation:

https://github.com/alvinbhou/Video2Text

video summarization:

https://github.com/shruti-jadon/Video-Summarization-using-Keyframe-Extraction-and-Video-Skimming

pytorch_video video classification:

https://pytorchvideo.org/docs/tutorial_classification

video feature extractor:

https://github.com/hobincar/pytorch-video-feature-extractor

剪的时候不要超过4分钟 可以用spleeter切出语音 加入自己的背景音乐

这个属于anti nsfw anti censorship 反内容审查 反视频审查 对抗机制 可以在github上面搜索

二创某种意义也是反审查

审查的 nsfw 微信小程序 可以解包 然后调用别人的接口 可能不稳定

https://github.com/superdashu/frida_with_wechat_applet

https://github.com/superdashu/pc_wxapkg_decrypt_python