Web Scraping

2024-03-31

Mediacrawler

easyspider

Mediacrawler utilizes browser side javascript execution to circumvent encryption algorithms and issues valid requests.

可以爬取的内容: 小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频｜评论爬虫、微博帖子｜评论爬虫

知乎爬虫可以按照类似的逻辑进行编写

2023-01-18

Spider, Web Scraping, Captcha Bypass

there are multiple spider collections on github, most of them are rudimentary and project specific, may not always suit your needs.

learnspider in which you can find simple captcha js code and many antimeasure of scrapers for practice. for solutions: learning_spider

spider collections covers zhihu.com

2022-12-13

Web Scraping Logic

select targets for scraping. it could be your browsing history, package indexs, social media (dynamic contents, with different accessing methods than web scraping)

if not accessible, access it with proxies, cookies.

finally store the content into compat and usable formats, categorized and linked

2022-09-12

Elinks/Lynx With Python: How To Speed Up Headless Website Browsing/Parsing/Scraping With Cookies

newscrawl 狠心开源企业级舆情新闻爬虫项目：支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除；爬虫一键部署；爬虫监控可视化; 配置集群爬虫分配策略；👉 现成的docker一键部署文档已为大家踩坑

general news extractor for extracting main content of news, articles

1 2	pip3 install gne

first of all, set it up with a normal user agent

even better, we can chain it with some customized headless puppeteer/phantomjs (do not load video data), dump the dom when ready, and use elinks/lynx to analyze the dom tree.

to test if the recommendation bar shows up:

https://v.qq.com/x/page/m0847y71q98.html

to make web page more readable:

https://github.com/luin/readability

load webpage headlessly:

https://github.com/jsdom/jsdom

https://github.com/ryanpetrello/python-zombie