在编写抖音文案的时候 用豆包来洗稿 提示词：改写下面的文章 查重率不超过30%

媒体的意义和AI类似 别人知道的就不要发了 有可能出错 别人不知道的就发 有可能有用 在兴趣圈周围探索 拓宽视野

网页转文章

readbility.js

pagescraper in php

elinks -dump <url>

可以把一个文字或者其他类型的内容当成模板 其他文字 视频 图片当作素材 根据模板收集素材 形成内容 注意素材不能是模板本身 素材不能单一 不然被认定为抄袭

文章洗稿 基于标题和context生成段落：

https://github.com/yangjianxin1/CPM

bert mask:

https://huggingface.co/fnlp/bart-base-chinese

https://huggingface.co/hfl/chinese-macbert-base

chinese paraphrase:

https://github.com/ZhuiyiTechnology/roformer

https://github.com/ZhuiyiTechnology/simbert

可能不是paraphrase模型

https://huggingface.co/lijingxin/mt5-for-zh-paraphrase/tree/main

https://huggingface.co/facebook/m2m100_418M

https://github.com/jiangnanboy/chinese_sentence_paraphrase

chinese summarize generator:

一般抽取式的提取 都需要有gpt生成器在中间插入一些句子

hanlp自带抽取文本方案

抽取式文本摘要

bart t5 pegasus中文文本摘要 有训练数据集 训练教程

一直在想怎么能正确高效的处理seo中，采集的文章怎么去伪原创和洗稿。如果是人工操作的话，那就太麻烦了。采集下来的文章不进行伪原创又害怕被飓风算法命中。

1，tr算法提取摘要再人工重组新的文章。

正好今天发现了python中的textrank4zh库，依赖于jieba、numpy和networkx库，可以通过tr算法进行文章的摘要提取。然后根据摘要再人工洗稿，整合成一篇全新的文章。

测试一篇蚂蜂窝上面的问答，蚂蜂窝问答下面是有很多个答主的内容，通过python爬取所有内容，然后再利用tr算法提取摘要，根据摘要进行重组出一篇新的文章。这样基本上可以成功躲避飓风算法。

先安装依赖库，然后再利用tr4进行摘要提取。

1
2
3
4
5
6
7
8
9
10
11

from textrank4zh import TextRank4Keyword, TextRank4Sentence
content = "" # 这里是python采集下来的content html内容
text = re.sub('<.*?>','',content)
text = re.sub(r'\s','',text)
zy = ''
tr4s = TextRank4Sentence()
tr4s.analyze(text=text, lower=True, source = 'all_filters')
# 可修改num值，设置摘要长度。
for item in tr4s.get_key_sentences(num=10):
zy = zy + item.sentence

2，利用google翻译双向翻译洗稿

之前有接触一个所谓人工智能洗稿的网站小发猫，说的是利用NLP算法进行洗稿，本来我以为洗稿只有同义词替换这个办法。

后来研究了一下小发猫，我首先觉得这个绝对不是利用什么所谓的NLP算法来洗稿，研究了一下发现可能是利用google翻译进行双向翻译，就是先中文翻译英文，然后再拿翻译出来的英文再翻译成中文。

自己也开发了一个这样的伪原创工具，发现其实并不好用。如果不仔细读，这样双向翻译出来的文章还能读，但是仔细读的话。其实语法习惯还有用词根本不准确，甚至有些情况还改变了这句话原有的语义。

Translators/Paraphraser for casual usage

baidu translator (api) provided by paddlehub

baidu language detector (api)

text style transfer:

https://blog.csdn.net/qq_27590277/article/details/106991084

python google translate api:

pip install googletrans

google translate in php:

https://github.com/Stichoza/google-translate-php

paraphrase via rephrasing and reordering

pegasus paraphrase:

increase the num_beams and temperature

https://analyticsindiamag.com/how-to-paraphrase-text-using-pegasus-transformer/

https://www.thepythoncode.com/article/paraphrase-text-using-transformers-in-python

example paraphrase project using LSTM as decoder and encoder:

https://github.com/vsuthichai/paraphraser

paraphrase with t5:

https://github.com/Vamsi995/Paraphrase-Generator

paraphrase dataset:

https://github.com/Wys997/Chinese-Paraphrase-from-Quora

文本纠错

https://github.com/James4Ever0/pycorrector

数据增强 变换句子形式

https://yongzhuo.blog.csdn.net/article/details/89166307

https://github.com/zhanlaoban/eda_nlp_for_Chinese

calculate perplexity:

https://github.com/DUTANGx/Chinese-BERT-as-language-model

https://github.com/James4Ever0/nlp-fluency

https://zhuanlan.zhihu.com/p/265677864

https://github.com/mattzheng/py-kenlm-model

multi-purpose tool for chinese: 偏旁部首 情感分析

https://github.com/SeanLee97/xmnlp

敏感词过滤 语言检测 训练语料库

https://github.com/fighting41love/funNLP

paraphraser.io

multilingual paraphrase database:

paraphrase.org

simbert

https://www.zhihu.com/question/317540171

BERT：原始版本bertRoberta：哈工大开源的中文wwm roberta模型BERT-SQ：本人在百度知道相似句数据集(Sim-Query)上微调后的bert模型Roberta-SQ：同上BERT-Whitening： @苏剑林 最新博客中提出的白化模型。Roberta-Whitening：同上

https://yongzhuo.blog.csdn.net/article/details/89166307

language fluency test:

https://github.com/baojunshan/nlp-fluency

many paraphraser models for english are on huggingface, but few for chinese.

https://huggingface.co/lijingxin/mt5-for-zh-paraphrase

https://pypi.org/project/genienlp/

https://github.com/salesforce/decaNLP

parrot paraphraser with nlu engines for english:

https://github.com/PrithivirajDamodaran/Parrot_Paraphraser

sentence level paraphraser:

https://github.com/vsuthichai/paraphraser

document level paraphraser, with sentence rewriting and reordering(shuffle):

https://github.com/L-Zhe/CoRPG

https://pypi.org/project/lexsub/

https://github.com/hit-joseph/lexical-paraphrase-extraction

synonyms (python library)

you can also train a contextual search tool using fine-tuned repurposed paraphrase model.

https://pypi.org/project/nlp-text-search/

文言文

https://github.com/raynardj/yuan

粤语

https://huggingface.co/x-tech

huggingface有英语翻译到其他语言的模型 没有翻译成中文的模型

在线

https://github.com/nidhaloff/deep-translator

https://github.com/UlionTse/translators

translatepy

离线

https://huggingface.co/tasks/translation

https://huggingface.co/Helsinki-NLP/opus-mt-zh-en

https://github.com/argosopentech/argos-translate

libretranslate

https://github.com/Teuze/translate

https://github.com/xhlulu/dl-translate/

facebook/mbart-large-50-many-to-many-mmt

mbart50

m2m100

view under https://huggingface.co/tasks to see great models fitting exact needs.

Use the original transcript for paraphrasing, while using danmaku for joke generation.

I found several evation methods, like paraphraser, random character swapper for text, and video blur, mirror to post the same video again. I guess it is essential not to let any part of the content look like the original one.