common web applications could be big, like search engines. this is a list of open-sourced self hosted services:
self-hosted
2022-07-11
2022-06-08
Search Engines DIY
my custom search engine built upon thesaurus/synonyms/antenyms, fzf and grep
RETRO retrieval based attention net, though using faiss, unclear if it is search related. on page 8 of the paper there are different retrieval based models for selections. LDA (topic modeling) can assist search by discovering similar topics.
download nltk data here. when downloading manually, beware of the url path and id, so you would put things in order.
you would patch nltk in order to download via proxy. these data files are hosted on github assets.
check keyword urlopen and filedownloder.py under /data/data/com.termux/files/usr/lib/python3.10/site-packages/nltk
maybe you can explore further with online search engines? select your keyword then search again.
thesaurus will slow down things. make it into a preprocessor.
related shits can be found here
search engine optimization
markuplm markup language model used for feature rich information extraction, webqa, arxiv paper: reading wikipedia to answer open domain questions
zinc search, go implementation of elastic search alternative
I bet there are many many alternatives. even for a relational database or graph database it can be a search engine by its nature.
how the heck can i search my own notes? slice it into little segments? standard excerpt included.
search for search engine in github.
search engines are related to spiders/crawlers.
how to utilize these search engines is a problem/challenge. use url filters, generic extractors, readbility.js, summarizers like sumy.
many specialized search engines that can search image, video and audio. one example is Jina
semantic search tool, multimedia search tool, neural search tool
https://github.com/searxng/searxng
parse popular search engine results like baidu, bing:
https://github.com/bisohns/search-engine-parser
search and scrape news
https://github.com/01joy/news-search-engine
image search engine
https://github.com/matsui528/sis
search engines used by hackers, social engineering, onion sites:
https://github.com/edoardottt/awesome-hacker-search-engines
search engine with customized recommendation:
https://github.com/mtianyan/FunpySpiderSearchEngine
seo tools 百度下拉词获取 推荐词相关词
https://github.com/marcobiedermann/search-engine-optimization
a self-hosted search engine that can be deployed on heroku, google alike:
https://github.com/benbusby/whoogle-search
txtai:
semantic search tool
pip3 install txtai
using sentence-transformer models from huggingface sentence embedding
https://github.com/neuml/txtai
yacy:
distributed search engine circumvent censorship
provide rss feeds
searx:
meta search engine self-hosted
has third-party hosted searx websites avaliable:
https://searx.space/ total 83 online(currently)
mwmbl:
distributed crawler central search engine, can be self-hosted
written in python
video search engine:
generate summary from frames
https://github.com/AkshatSh/VideoSearchEngine
yuno:
context based search engine for anime, anime search engine with transformer and deep learning. text based search. more like a semantic search tool, or neural search tool.
Yuno is a context based search engine that indexes over 0.5 million anime reviews and other anime informations. To help you find anime with specific properties. This search engine will help people of r/AnimeSuggest who are looking for specific type of anime to watch.
This search engine was created to solve the problem of finding an object with specific properties and the object in this case is anime. But this search engine can be easily extended to any domain like books,movies,etc. Without the need of any kind of handcrafted dataset.
TypeSense:
dedicated client for every popular programminhg language
consume much fewer ram than meilisearch
need to write custom web interface via nodejs
upload data via client api
MeiliSearch:
good for small dataset
consume whoopy 900mb for my 9mb json dataset.
has intuitive web interface.
upload document via web post.
2022-05-29
tutorial: build QA pipeline with no dependencies with haystack
visit jina hub to get multiple embedding models and workflows
finetuner: text to image search via clip
datawhale provides tutorials on machine learning, also provide book materials, topics are: numpy, matplotlib, pandas,
vced: holy gift from datawhale able to edit video by text, video auto editor, cutter
VCED 可以通过你的文字描述来自动识别视频中相符合的片段进行视频剪辑。该项目基于跨模态搜索与向量检索技术搭建，通过前后端分离的模式，帮助你快速的接触新一代搜索技术。
jina:
https://github.com/jina-ai/jina/
documentation:
quick demos:
dress Fashion image search: jina hello fashion
robot QA chatbot: pip install “jina[demo]” && jina hello chatbot
newspaper Multimodal search: pip install “jina[demo]” && jina hello multimodal
fork_and_knife Fork the source of a demo to your folder: jina hello fork fashion ../my-proj/
Create a new Jina project: jina new hello-jina
ai video metadata generation: