pyjom dev schedules

整活

• 应 急 诈 骗 食 品 (派蒙加Rick Ashley 如何混合？）

recommendation

• use txtai to do NLU and recommend things to people

topic discovery/acquiring

trending topics

• baidu search trending

• sogou trending

• bilibili trending

• wechat trending

• toutiao trending

• tencent trending

• netease trending

• youtube trending

• reddit trending

• twitch trending

popular topics

• baijiahao popular topics

• bilibili popular topics

• douyin popular topics

personal/customized topics

• tencent qq customized (can associate with mail)

• wechat customized

• bilibili per user customized

dog/cat video generation

make render engine runnable

issues:

• video length too long (10 mins)

it was the speed calculation error.

• bgm somehow not in sync (too broad bpm/clip ranges?)

• to analyze the peaks (abrupt changes) in bgm and grab louder peaks using pyloudnorm (getting audio volume)

1
2

pip3 install pyloudnorm

1
2
3
4
5
6
7
8

import soundfile as sf
import pyloudnorm as pyln
data, rate = sf.read("0055014.wav") # load audio (with shape (samples, channels))
print(data.shape)
meter = pyln.Meter(rate) # create BS.1770 meter
loudness = meter.integrated_loudness(data) # measure loudness
print(loudness)

• place video on loudest points, abrupt changes detected by talib or just take direvative and gaussian average

• video too repetitive (small corpus?)

• do not remove subtitle and crop active region (reviewer’s resource not used? but i rather advise you to do it directly since it requires less computational power)

• do not have minimum motion threshold (reviewer’s fault? also recommend you to do this in producer)

• remove all watermarks, subtitles and crop video boundaries accordingly

• source video and audio (infinite, basic test is to find 500 sources at once without duplicate, second test is to find 500 second is to find 500 without duplicate twice), improve highlight algorithm

• find 500 songs without duplicate at once

• find 500 songs no duplicate twice

• find 500 animal videos without duplicate

• find 500 animal videos no duplicate twice

• generate appropriate title, cover, info and tags

• collect feedback after the post

• find some shocking fonts for cover and subtitle, english and chinese

• make that karaoke effect

• make ass with karaoke effect with lrc files

• make lyrics sync logic fluent, according to what have learned from karaoke effects

• make selected video clips fluent, no abrupt cuts, maybe we need pyscenedetect?

text to video, template based video generator (this is perhaps the most complex video generator ever. do it with caution, it might also includes the flipcard, narrator and slideshow based generators)

generator models subarchitecture (subcategories of template based generators)

flipcard

slideshow (video and audio, might also include the dog&cat video!)

narrator

summarized video

policy evasion, NSFW filters

• remove all hints from image, video, audio and script that may lead to copyright issues

analyze the media content and metadata, relationships

• analyze danmaku

• paraphrase the script

• cut the crap and understand each clip’s meaning

process the video clips, like changing the human figure, changing face, stylish the video, adding 2d to 3d effects

process the audio clips, like changing voice, adding sound effects, separating audio/music tracks, ducking

index, retrieve and align video and audio content according to our collected database

retrieve and align video and audio according to our smart search agent (keyword extractor, related words) and do live compilation

qq managing

• mitm chats in friends

• mitm chats in groups

• source and send pictures to qzone

• source and send pictures to chat

• reduce posting frequency by group size and feedback

• post relative video link relative to group topic

personal info collecting and email/sms bulk sending

• avoid mail being trashed or turned into junk

• collect and make mail templates for mail posting

voice changer

• vst based voice changer

• train or find a decent voice generator 御姐音语料库 小受音语料库

请在b站或者qq群里面寻找 或者什么其他的有关的地方寻找 谢谢

直播 live streaming

• source the video

如果是同一个站的 尽量放一个月以前的视频 半个月以前的音频

• prepare some space for storing live streaming data

• source the audio

• automatic interactions

• handle the vtuber model’s actions

find things in colab, kaggle, aistudio, bilibili, youtube.

see in huggingface tasks to find task-specific models, also huggingface spaces for demo on models

modelscope by alibaba supports tensorflow and pytorch

mindspore model zoo

mindspore hub

intel model zoo

run models from intel model zoo in docker container, like recommendation

openvino model zoo

百度总结的 比较全面的深度学习应用 deeplearning applications

jina hub

阿里巴巴模型库 具有许多适用于商业 自媒体的模型供选择

huggingface

苹果官方CoreML模型库

CoreML第三方模型库

paddle模型库 paddlehub

pytorch模型库

TensorFlow模型库

graphcore model zoo for IPU

文章多平台发布 浏览器插件 一键同步文章到多个内容平台，支持今日头条、WordPress、知乎、简书、掘金、CSDN、typecho各大平台，一次发布，多平台同步发布。解放个人生产力

3d pose tracker

rendered on unity. needs GPU.

Sysmocap

WHAT I WANT FOR (or nearly) requires real 3d models, written in javascript

cannot output video?

A cross-platform real-time video-driven motion capture and 3D virtual character rendering system for VTuber/Live/AR/VR.

Does not require a discrete graphics card and runs smoothly even on eight-year-old computers

Vtuber python unity

search for “vtuber” along with “motion capture” you will get many head-only trackers and renderers for windows but not linux, also some “broadcast templates/frameworks”. many support one single image (anime head + remove background) as input instead of 2d/3d models

face tracking only, showing face, mouth and eyes, head directions, bind to live2d models

虚拟数字人 metahuman

NextHuman Beta0.9上线公测，5分钟高品质讲解，带你进入数字人“零门槛”创作新时代，体验直通车 -> https://nexthuman.cn 免费版是Windows上面跑的 需要高端1070显卡

anime character segmentation

to remove false positives, make sure we have anime face in view, otherwise mark it as a false positive.

you can use anime character recognition like moeflow or opencv anime face detector along with some phash or perceptual hash library to group similar characters, compare perceptual image similarity and line them up in a series.

aniseg, able to segment anime character and head, using mask-rcnn

yet another anime character segmentation model using solov2 and condinst

waifu segmentation

high accuracy anime character segmentation

自动画漫画 画几笔就成某个人像 动漫头像

https://menyifang.github.io/projects/DCTNet/DCTNet.html

自动捏脸 gan给人脸戴口罩

https://github.com/futscdav/Chunkmogrify

selfie to anime, picture to anime photos

selfie2anime with trained models

##原神mmd下载模型

模之屋（需要注册）：

https://www.aplaybox.com/u/680828836

夕蓝资源网（可直接下载） 也有其他的3d模型可以下载：

https://www.seoliye.com/tags/53.html

use voice to power up static images

voice powered animated cartoon figure

jeeliz (some web deep learning runtime, like tensorflow.js) powered

weboji, highly similar to animoji, with three.js and cute fox avatar

face filter, alter the face like putting glass, minor changes to avoid privacy/copyright concerns?

openface

facial features extraction

facerig

facerig location: /Software/Program Files (x86)/FaceRig

i’ve seen python code inside facerig.

facerig does not offer head-only rendering, but that could be changed i suppose?

avatarify python

infinite avatars by using style gan, first order motion model

create static portrait avatar (svg?)

animoji from apple

facial landmark detection in python, animoji-animate

animoji apple private framework 实际上这个就是之前看到的会动的狗屎的视频来源

2d模型 皮套 可动 虚拟Vtuber talking head

https://github.com/yuyuyzl/EasyVtuber

https://github.com/pkhungurn/talking-head-anime-3-demo

https://github.com/GunwooHan/EasyVtuber

b站官方

直播姬现在支持2d面部捕捉 3d模型动作捕捉

直播姬版本有windows macos(m1) Android版本

2d模型是live2d的模型

有待研究

qq seems to release mac qq with electron, lot more easier for reverse engineering

how to reverse go binary, golang reverse

opqbot官方已经说了 登陆过程中会用到远程的服务器 这个服务器究竟在干什么不得而知 可能和登陆有关也可能没有关系 但是服务器维护期间是没法扫码登录的 如果有可以正常使用的secdata是可以直接启动服务的 不需要服务器 所以估计这个服务器很可能就是拿来解析cookie的

login error:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

2022/08/14 00:01:24.808 [I]  Scan Status 48 Uin 0
2022/08/14 00:01:25.880 [I]  Scan Status 48 Uin 0
2022/08/14 00:01:26.937 [I]  Scan Status 53 Uin 0
2022/08/14 00:01:27.998 [I]  Scan Status 53 Uin 0
2022/08/14 00:01:29.054 [N]  User <userId> 登录中..请勿连续操作,登录成功后或释放连接后在继续操作 登陆成功后请勿频繁扫码再次登陆(除非冻结导致的掉线) 发不出去群消息请挂机几天 TX日常风控
=========本框架 🎈 免费 🎈 使用 谨防 ⚠️ 诈骗 ⚠️ 收费 切勿用于 🈲️ 非 🈲️ 法用途
=========交流群:757360354 TG群组:https://t.me/IOTQQ
=========开源社区 👍 https://github.com/opq-osc 👍
=========项目主页 😄 https://github.com/OPQBOT/OPQ/wiki 😄
=========项目Wiki 📒 https://github.com/OPQBOT/OPQ/wiki 📒
2022/08/14 00:02:30.234 [W]  recvPump session 0D48F5949075DA13D3A9F83838903920
2022/08/14 00:02:30.234 [A]  Default Closed:0D48F5949075DA13D3A9F83838903920
2022/08/14 00:02:30.235 [D]  Unregister In Conn -> 0D48F5949075DA13D3A9F83838903920

关于自动加群 可以考虑使用安卓手机自启动功能（需要下载startup manager 或者boot manager（有root权限和xposed框架）） 用termux-appium 自动操作手机在联网的情况下自启动加群

现在有两个标准onebot nonebot

这两个协议都不支持主动加好友 加群 还有收红包方法 至少mac qq协议支持这些方法 但是其他的协议比如手表 ipad协议支不支持就不清楚了

onebot有大量的qq适配器 而nonebot有大量的插件和除了qq以外的连接器

nonebot可以连接onebot

在onebot的qq适配器中 oicq可以查看qq历史聊天记录（有待验证） 可能对qq的数据爬取有帮助 视频爬取 oicq这个适配器有在群里面加好友的方法addFriend(gid, uid)可以参考,提供了一些用于逆向qq协议的程序：

txhook 该软件适合在安卓8.0以上系统运行，理论支持安卓7.0以上，但是很多问题。群号：901422091 702991373

• 获取ShareKey\PublicKey\D2\A2…

• 主动拦截固定Ecdh密钥及版本

• 对Jce\Protobuf的自动分析

• 过滤抓包，支持高级过滤（长按抓包页面的搜索栏展示/隐藏图标）

protobuf online decode

protobuf unpack-tools

也有一些可以进行二次开发的qq web api 搜索QQ号和群号 且有个性签名等更多信息 或许可以搜索关键词？

这些适配器中有的提供了qq频道的支持：

oicq-guild

也可以考虑用frida ghidra radare2 cutter来逆向opqbot的go编译好了的程序 或者逆向分析opqbot的网络请求数据 甚至直接动态调用opqbot里面的方法 直接用其他机器人登陆之后获得的cookie进行操作

to get the token, login first, then visit here or click “sign in” here

据说扫码登录只支持同一个ip下面的登陆 不知道为什么这个opqbot登陆失败 但是其他机器人都提供了账号密码登陆的渠道 将opqbot的协议逆向出来 或许可以提高登陆成功率 实现相同的功能

默认(可修改)在 ./data/your-account/ 下会自动生成device.json设备文件，登录完成后此设备文件长期有效

设备文件的生成并非随机，而是使用固定算法，一个账号会永远生成同一份设备文件

如果需要在异地服务器上登录，建议先在常用地通过设备验证并登录挂机一段时间

由于会生成相同设备文件，只要不手动修改，只需验证一次，在任何地区都可直接登录

it seems the login issue of opqbot is related to the account itself, not gitter token, software version or proxy

by the way we could always use go-cqhttp, without the ability to collect red packet and add group/friends.

qq add group/friends may be enabled by our windows virtual machines. without opq, it is very memory intensive.

tokens:

1
2

74eb7eb14aa36d1b9c2c663bc49335e8becd5318

1
2

2d391bd7639362032d09abfc5a9cc6368b7664d5

1
2

bdf52599d992665509ee5b0b533d5eed08452def

no consecutive clip sequences have then same file source.

no identical clips in render sequence.

Detect media file corruption, Python suggest binary file name extension

to rule out those corrupted media files, or unplayable files. maybe simply by parsing these files is not enough, we need a dedicated file corruption detector.

to truncate these files and see errors produced by media readers. use text file with media file extension to test them.

A curated list of awesome data labeling tools

Images

• labelImg - LabelImg is a graphical image annotation tool and label object bounding boxes in images

• CVAT - Powerful and efficient Computer Vision Annotion Tool

• labelme - Image Polygonal Annotation with Python

• VoTT - An open source annotation and labeling tool for image and video assets

• imglab - A web based tool to label images for objects that can be used to train dlib or other object detectors

• Yolo_mark - GUI for marking bounded boxes of objects in images for training neural network Yolo v3 and v2

• PixelAnnotationTool - Software that allows you to manually and quickly annotate images in directories

• OpenLabeling - Label images and video for Computer Vision applications

• imagetagger - An open source online platform for collaborative image labeling

• Alturos.ImageAnnotation - A collaborative tool for labeling image data

• deeplabel - A cross-platform image annotation tool for machine learning

• MedTagger - A collaborative framework for annotating medical datasets using crowdsourcing.

• Labelbox - Labelbox is the fastest way to annotate data to build and ship computer vision applications

• turktool - A modern React app for scalable bounding box annotation of images

• Pixie - Pixie is a GUI annotation tool which provides the bounding box, polygon, free drawing and semantic segmentation object labelling

• OpenLabeler - OpenLabeler is an open source desktop application for annotating objects for AI appplications

• Anno-Mage - A Semi Automatic Image Annotation Tool which helps you in annotating images by suggesting you annotations for 80 object classes using a pre-trained model

• CATMAID - Collaborative Annotation Toolkit for Massive Amounts of Image Data

• make-sense - makesense.ai is a free to use online tool for labelling photos

• LOST - Design your own smart Image Annotation process in a web-based environment

• Annotorious - A JavaScript library for image annotation.

• Sloth - Tool for labeling image and video data for computer vision research.

Text

• YEDDA - A Lightweight Collaborative Text Span Annotation Tool (Chunking, NER, etc.). ACL best demo nomination.

• ML-Annotate - Label text data for machine learning purposes. ML-Annotate supports binary, multi-label and multi-class labeling.

• TagEditor - Annotation tool for spaCy

• SMART - Smarter Manual Annotation for Resource-constrained collection of Training data

• PIAF - A Question-Answering annotation tool

Audio

• EchoML - Play, visualize, and annotate your audio files

• audio-annotator - A JavaScript interface for annotating and labeling audio files.

• audio-labeler - An in-browser app for labeling audio clips at random, using Docker and Flask.

• wavesurfer.js - Simple annotations tool, check the example.

• peak.js - Browser-based audio waveform visualisation and UI component for interacting with audio waveforms, developed by BBC UK.

• Praat - Doing Phonetics By Computer

• Aubio - Tool designed for the extraction of annotations from audio signals.

Video

• UltimateLabeling - A multi-purpose Video Labeling GUI in Python with integrated SOTA detector and tracker

• VATIC - VATIC is an online video annotation tool for computer vision research that crowdsources work to Amazon’s Mechanical Turk.

Time Series

• Curve - Curve is an open-source tool to help label anomalies on time-series data

• TagAnomaly - Anomaly detection analysis and labeling tool, specifically for multiple time series (one time series per category)

• time-series-annotator - The CrowdCurio Time Series Annotation Library implements classification tasks for time series.

• WDK - The Wearables Development Toolkit (WDK) is a set of tools to facilitate the development of activity recognition applications with wearable devices.

3D

• webKnossos - webKnossos is an open-source web-based tool for visualizing, annotating, and sharing large 3D image datasets. It features fast 3D data browsing, skeleton (line-segment) annotations, segmentation and proof-reading tools, mesh visualization, and collaboration features. The public instance webknossos.org hosts a collection of published datasets and can be used without a local setup.

• KNOSSOS - KNOSSOS is a software tool for the visualization and annotation of 3D image data and was developed for the rapid reconstruction of neural morphology and connectivity.

Lidar

• semantic-segmentation-editor - Web labelling tool for camera and LIDAR data

MultiDomain

• Label Studio - Label Studio is a configurable data annotation tool that works with different data types

• Dataturks - Dataturks support E2E tagging of data items like video, images (classification, segmentation and labelling) and text (full length document annotations for PDF, Doc, Text etc) for ML projects.

这个人的空间链接目前可以访问@2022 september 4

可以在被拉黑了之后快速点击右上角的分享链接 分享到其他人 其他群里面 或者点击生成链接 即可在浏览器里面查看这个人的动态 但是不知道这个链接有没有时效性 现在看起来就是一堆乱码 app里面的分享也不知道有没有时效性

不知道能不能搜索或者遍历 如果不能的话只能黑进去了 不过那样的话出来的数据肯定更多

要知道被拉黑，本地肯定有用户的ID， 有了ID就可以拿过去到其他新注册的Soul账号上面使用 通过底层api访问

可以考虑用Frida或者网上的一些脚本来分析破解SoulAPP 单独使用Frida估计不能利用Python遍历 还是需要破解协议证书才可以自由访问

frida usage, code examples for windows

radare2 tutorial with code

speechbrain has features of Speech Recognition, Speaker Recognition, Speech Enhancement, Speech Processing, Multi Microphone Processing, Text-to-Speech, and also supports Spoken Language Understanding, Language Modeling, Diarization, Speech Translation, Language Identification, Voice Activity Detection, Sound classification, Grapheme-to-Phoneme, and many others.

概述

视频里面的语言分为图片上面打出来的字幕以及人说的话

涉及到的问题分别为： 图片文字的语言分类 以及音频语言分类

音频识别

online speech recognition

pip install SpeechRecognition

offline, need to provide language id:

https://pypi.org/project/automatic-speech-recognition/

use paddlespeech if possible, for chinese and english

图片语言识别

use google cloud to detect language type in image:

https://github.com/deduced/ml-ocr-lang-detection

Detects and Recognizes text and font language in an image

https://github.com/JAIJANYANI/Language-Detection-in-Image

图片语言文字分类 可以用easyocr实现 加载多个模型 比如 中文加英文加日语 b站其他语言的可能也不怎么受欢迎 最多再加韩语

可以从视频简介 标题 链接里面提取出句子 每个句子进行语言分类 确定要使用的OCR模型 也有可能出现描述语言和视频图片文字语言不一致的情况

wolfram language提供了一个图片分类器 分类出来的结果可能很有意思 可以结合苹果的图片关注区域生成器来结合使用

ImageIdentify[pictureObj]

这个方法还支持subcategory分类 支持多输出 具体看文档

https://www.imageidentify.com/about/how-it-works

wolfram支持cloud deploy 到wolfram cloud不过那样可能不行

文本语言识别分类

lingua performs good in short text, can be used in java or kotlin

supporting detecting different languages:

cld2 containing useful vectors containing text spans python binding

1
2
3
4
5
6
7
8

>>> import pycld2 as cld2
>>> text_content = """ A accès aux chiens et aux frontaux qui lui ont été il peut consulter et modifier ses collections et exporter Cet article concerne le pays européen aujourd’hui appelé République française.
Pour d’autres usages du nom France, Pour une aide rapide et effective, veuiller trouver votre aide dans le menu ci-dessus.
Welcome, to this world of Data Scientist. Today is a lovely day."""
>>> _, _, _, detected_language = cld2.detect(text_content,  returnVectors=True)
>>> print(detected_language)
((0, 323, 'FRENCH', 'fr'), (323, 64, 'ENGLISH', 'en'))

original cld3 is designed for chromium and it relies on chromium code to run

official cld3 python bindings

additional Python language related library from geeksforgeeks:

textblob is a natural language processing toolkit

1
2
3
4
5
6

from textblob import TextBlob
text = "это компьютерный портал для гиков. It was a beautiful day ."
lang = TextBlob(text)
print(lang.detect_language())
# ru

langid performs good in short text

textcat (r package)

google language detection library in python: langdetect

javascript:

https://github.com/wooorm/franc

python version of franc:

pyfranc

wlatlang.org provides whatlang-rs as rust package, also whatlang-py as python bindings