on non-macos platforms, use zeal instead.

the core problem is that after uninstallation of xcode, one cannot launch the Apple Doc Helper or some binary afterwards inside the Apple docset under dash documentation folder. the docset can be copied to this folder automatically by dash but without xcode it cannot be opened.

original youtube-dl

the fork with faster (maybe?) download speed: yt-dlp

alternatives

miniconda

miniforge, better apple m1 support

mamba, multithreaded

jax

docs

autograd and xla (Accelerated Linear Algebra)

pyro

probabilistic programming

getting started

examples

sample code

numpyro

getting started

pyro implementation in numpy, alpha stage

scikit-learn

machine learning in python

libsvm

install official python bindings:

1
2

pip install -U libsvm-official

third-party python libsvm package installed by:

1
2

pip install libsvm

opennlp

hands-on docs

model zoo

opennlp uses onnx runtime(maybe?), may support m1 inference.

opennlp is written in java. after installing openjdk on macos with homebrew, run this to ensure openjdk is detected:

1
2

sudo ln -sfn $(brew --prefix)/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

opennlp has a language detector for 103 languages, including chinese. opennlp has a sentence detector (separator) which could be trained on chinese (maybe?)

in order to use opennlp with less code written, here’s how to invoke java from kotlin

dl4j

found on mannings article about better search engine suggestions. in this example it is used with lucene, which has image retrieval (LIRE) capability. lucene is also avaliable as lucene.net in dotnet/c#.

to install lucene.net:

1
2

dotnet add package Lucene.Net --prerelease

deep learning library for java

xgboost

gradient boost is used to train decision trees and classification models.

lightgbm

Light Gradient Boosting Machine

have official commandline tools. installation on macos:

1
2

brew install lightgbm

install python package on macos:

1
2
3

brew install cmake
pip3 install lightgbm

pymc

examples

if want to enable jax sampling, install numpyro or blackjax via pip

difference between pymc3 (old) and pymc (pymc4):

pymc is optimized and faster than pymc3

pymc3 use theano as backend while pymc use aesara (forked theano)

docs with live demo of pymc

PyMC is a probabilistic programming library for Python that allows users to build Bayesian models with a simple Python API and fit them using Markov chain Monte Carlo (MCMC) methods.

fastai

a high level torch wrapper including “out of the box” support for vision, text, tabular, and collab (collaborative filtering) models.

docs

courses

on the twitter list related to opennlp shown up on its official website, fastai has been spotted.

fastai does not support macos. or is it? fastai is on top of pytorch. initial support starts with 2.7.8 and now it is currently 2.7.9

searching ‘samoyed’ like this in github we get a dataset for pets classification called imagewoof from fastai 2020 tutorial series. more image classes like subcategories of cats may be found in imagenet.

the fix

to enable the service:

1
2

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.locate.plist

to update locate db:

1
2

sudo /usr/libexec/locate.updatedb

or, more conveniently:

1
2
3

sudo ln -s /usr/libexec/locate.updatedb /usr/local/sbin/updatedb
sudo updatedb

alternative

use mdfind

Termux:Boot Autostart Program Fixes

according to this, termux:boot on android 10 and above will not work. instead, change all executables with relative paths in init scripts to their absolute paths. if any referred executable is a script file containing other executable with non-absolute paths(except for those built-in programs like am), change that too.

mostly we hold wakelock, start sshd, crond or nginx and other non-blocking, non-interactive apps at start.

it is funny that macOS still supports AMD GPUs, means any intel Mac (not M-series!) can now utilize internal/external AMD GPUs as long as frameworks like jax and pytorch support MPS/Metal.

calling python code from swift using pythonkit:

1
2
3
4
5
6
7
8

func downloadVideo(link: String){
let sys = Python.import("sys")
sys.path.append(dirPath)
let example = Python.import("sample")
let response = example.downloadVideo(link, dirPath)
videoPath = String(response)
}

run macos in docker with kvm

neural engine

it is used for coreml inference, not training

run coreml on hackintosh

first, download macos montery using the mac.

then, install it on hackintosh, with associated nvidia drivers.

next test gpu avalibility via system info panel.

then install xcode commandline tools and check coreml avalibility

run coreml with swift on linux

darling is at its very premature stage, just like the wine. now it is testing something called “darlingserver” which is a full userspace implementation and is prone to tons of problems. swift repl is not working and installing xcode commandline tools 14 will hang this thing. i suggest you to do light model training on macbook air and convert it to onnx if want to use it everywhere.

before reinstallation of darling, make sure you have removed all darling related files by checking updatedb; locate darling | grep -v <compile directory>

visit here to install darling from source (maybe that’s the only way)

if want to install darling on kali, you must outsource all deeplearning models to other disks, and collect all other big files to somewhere else or trash them. use systemwide user broadcast method to warn me if any of the disk is missing. use automatic symlink change method to adapt the external disk mountpoint changes.

darling can install xcode commandline tools with macos sdk, so maybe it can run coreml models with swift using cpu. gpu support is currently not known. maybe that requires metal support.

thermal and battery life concerns, and more

consider using external gpus (eGPUs) with thunderbolt 3 and AMD GPUs to avoid overheating. currently that can only be done with intel Macs.

battery life is currently bad for intel/amd notebooks of x86/64 architecture.

heavy lifting jobs are likely to be run on Mac Studio with M1 Ultra and 128GB RAM. Macbook Air M1 with 8GB RAM is simply not feasible.

aside of Apple platforms, these APIs are virtually useless.

to run these on other non-apple machines, you need to tweak and install macOS on x86-64 platforms with macOS supported GPUs(may have low performance), which will definitely not taking any advantage of huge shared RAM with CPU, and may run poorly on CoreML/CreateML, may not support deepspeed stage 2/3 or BMI(big model inference)

tensorflow with m1 support

using tensorflow metal plugin, which sets up miniforge and install tensorflow-metal within.

install without miniforge(works!)

1
2

pip3 install tensorflow-macos tensorflow-metal

validation:

1
2

python3 -c "import tensorflow as tf; physical_devices = tf.config.list_physical_devices('GPU'); print('Num GPUs:', len(physical_devices)); print(physical_devices)"

pytorch with m1 support, using MPS (Metal performance shader)

install from nightly release channel, with minimum system version requirements 12.3 (which this machine had been qualified after system update, now 12.5)

1
2
3

# MPS acceleration is available on MacOS 12.3+
pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

validation

1
2

python3 -c "import torch; print('MPS avaliable:',torch.backends.mps.is_available()); print('Built with MPS:',torch.backends.mps.is_built())"

run python inside swift

use pythonkit

automatic machine learning using CreateML

1
2

import CreateML

CreateML is similar to any other AutoML tools, like AutoKeras, AutoTrain by Huggingface (works by training against a selected set of user-provided models)

using CoreML

curated, largest coreml models collection

CoreML models can be created by CreateML and some customization can be done via protocol MLCustomLayer.

onnxruntime can run onnx models on CoreML, via c#, since that library is maintained by microsoft.

to install c# on macos:

1
2

brew install dotnet-sdk

to install and launch dotnet repl:

1
2
3

dotnet tool install -g dotnet-repl
dotnet repl

paddlepaddle support

convert into onnx first, then run on onnxruntime.

paddlepaddle itself currently only supports running on M1 CPU only via rosetta 2.

Links

Swift Core ML 3 implementations of GPT-2, DistilGPT-2, BERT, and DistilBERT for Question answering.

train image classifier and text classifier in which CreateMLUI is deprecated (gone)

train source code classifier with flightschool which is a free swift tutorial books provider

classifying sounds with coreml return sound type along with timestamp

detect human pose using coreml

apple speech recognization api request

pytorch mps backend

text classification using createml

onnx model zoo

Getting CoreML Models

CoreML Model Zoo

Apple Machine Learning Related APIs (may need user permission within or without xcode by means of Info.plist or something)

many methods have been tried. NetBIOS not working. DHCP server not found. nmap script engine(NSE) uses lua to automate sniffing and attacks.

if you install p2p server nodes on primary server (with hard-to-crack password and proper configs (no brute-forcing)?) you might want to add that (n2n) server node at home.

p2p network

nps also supports p2p

(deprecated! does not pass the connectivity test) opengnb p2p network, faster than n2n v3, can run without public ip

gost as an frp alternative

turned out n2n is necessary, since the speed comparasion strongly disencourage the usage of frp directly.

n2n test commands, using compatible v3 protocol to communicate:

supernode v3: n2n.laiyx.win:10090

warning: it is useless to add multiple supernodes.

1
2
3

-l nton.eu.org:10090 -l n2n.lu8.win:10090 -l n2n.haoren.eu.org:10090 -l
supernode.ntop.org:7777 -l 47.102.102.77:10090 -l n2n.myan.cc:10090 -l n2n.sfcs.eu.org:10090 -l n2n.eriol.cn:10090 -l n2n.x0x.cn:10090 -l n2n.vvcd.win:10090

kali:

1
2

sudo edge -c <name> -k <password> -a 192.168.100.1 -f -l n2n.laiyx.win:10090 -Er -A3 -e auto

macos, since we use sudo you might consider doing it with system service:

1
2

sudo edge -c <name> -k <password> -a 192.168.100.2 -f -l n2n.laiyx.win:10090 -Er -A3 -e auto

public shared n2n supernodes

you could test the speed and decide to use it or not.

in kali discovery service, when local connection is not avaliable, usually the p2p network is preferred than direct frp tunneling.

brew has tinc as a package!

tinc conf

tinc setup with core server

remote access with vps using tinc

install and config tinc on linux

tinc is somehow complex and it may requires some tinkering on tinc-up or using docker.

install n2n without macports

use n2n to send udp packages among clients, try to create direct link between devices which will speed up ssh connection speed. supernode creation could be used along with frpc

somehow brew does not have n2n as a package. macports has it, which requires xcode (huge!) to be installed.

peervpn tutorial

daemonize (launch at startup)

on macos, when crontab is created, cron will be automatically launched by launchd.

cronjobs may need to launch with the $(which env) prefix.

the problem of internet disconnetion will most not likely to interfere with the server since frpc has auto reconnection and the update hook is the filesystem watchdog, which will not run when no changes made (including the offline period)

the watchdog may be replaced by some mirror fuse system, which will report every access request to our dedicated server.

we have seen this behavior (filesystem mirroring) in our gitfuse code. but does that support symlink? should we really take care of that? or should we forget that and just use inotify instead?

maybe it will affect the client when mounting the remote filesystem using sshfs or rclone, but that has to be verified.

serve and mount remote filesystem

before serving, make sure the path /media/root/help/pyjom exists by running our mount script

create htpasswd file:

1
2

htpasswd -bc webdav_htpasswd <username> <password>

use rclone:

1
2

rclone serve webdav /media/root/help/pyjom --addr 0.0.0.0:8468 --key /root/.local/share/code-server/localhost.key --cert /root/.local/share/code-server/localhost.crt --htpasswd /root/Desktop/works/sync_git_repos/remote_deploys/webdav_htpasswd -L

before mounting, use rclone config to setup remote associated with a name. make sure the hostname is localhost instead of ip address to avoid certificate issues. do not install rclone from brew since it does not support fuse. instead, install from here

1
2

rclone mount webdav_local_nginx:/ /Volume/CaseSensitive/pyjom_remote_mountpoint --ca-cert /Users/jamesbrown/Desktop/works/host_discovery_ssh_local_connect/certificates/localhost.crt

after mounting, seems zsh on macos is not working very well with macfuse. bash works. does bash/fish works with sshfs as well? maybe that will save efforts.

encryption and invalid HTTPS certificates

use nginx to redirect remote server as localhost, since the host name on the certificate is localhost we cannot let chrome to trust anything other than that

1
2
3
4
5
6
7
8
9
10

worker_processes auto;
error_log error.log;
events { }
stream {
server {
listen 127.0.0.1:7576;
proxy_pass REMOTE_HOST:7576;
}
}

code-server(browser) color fixes

1
2
3
4
5
6
7
8
9
10
11

.cursor{
background: white;
}
body.web{
caret-color: white;
}
.monaco-editor .view-line span.inline-selected-text{
background: blue;
color: white;
}

connectors other than frp

code-server recommends some other methods like cloudflared and ngrok. 花生壳可能也有用 但是可能不好用

methods

try out code-server by coder, might work?

also we use builtin vscode connectors, using ssh.

currently we only have one, which uses direct ip address instead of a hijacked domain. maybe it is time to consider some faster server providers.

use a universal ssh as workspace extension called SSH FS

drawbacks of SSH FS extension

some drawbacks of this SSH FS plugin is that it cannot use the plugins from remote machine, also having issue whe jumping to remote files from terminal output. to run code-insider instead of code-oss, maybe we could spin up the official ssh connector, which can only be automated by publickey authentication.

syncing, updating and viewing using watchdog and sshfs(deprecated since it shares connection with vscode remote and maybe slower than rclone serve webdav?)

to mount the filesystem via sshfs:

1
2

sshfs root@192.168.10.4:/media/root/help/pyjom /Volumes/CaseSensitive/pyjom_remote_mountpoint -o follow_symlinks

to make sure the changes are updated regularly, we need a filesystem watchdog on kali, which will trigger the action of syncing, utilizing inotify. shall that be adopted on macos? maybe. but my extra editors can be vim or nvim, so it is not so hard to predict. but if it can monitor the file read events, we don’t need those legacy editor program hooks.

at least we need to see the output, so we need to mount the remote filesystem as sshfs, then use ffplay to view it.

solution

for now, two viable ways:

one using code-server, the other using code-server-insider provided by code-insider. when using builtin code-server-insider, remember it will not share the plugins installed by code-insider. the remote executable location is at /root/.vscode-server-insiders/bin/12b08be500f8a307f30e92cbc3ee39ba115eab69/bin/code-server-insider or something. must set the local setting remote.SSH.useLocalServer to false.

when using code-server, one can connect to the workspace using browser, instead of vscode builtin remote connector.

sometimes we install a single package which brings about tons of dependencies in python, pip-autoremove comes in handy.

install it by pip3 install pip-autoremove

though be careful these dependencies might not be used in other existing packages, they are sometimes still being used in your code!