Project structure of: BlinkDL/RWKV-LM

• RWKV-LM RWKV-based Text Generation and Chatbots
  • README.md Trains enhanced RWKV language model with features and optimizations
  • RWKV-v1 Neural network training, Transformer models, text generation utilities.
    • src Neural network training, Transformer models, RWKV-v1 source code, text generation utilities.
      • model.py Neural network weights, time curves, Transformer models with rotary pos embeddings.
      • trainer.py Train models, handle datasets, adjust learning rate.
      • utils.py Tight top-k, p sampling with temperature control for text gen
    • train.py Trains RWKV language model efficiently.
  • RWKV-v2-RNN Trains and saves RWKV-v2 RNN models.
    • cuda CUDA implementation of RNN forward/backward passes
      • timex_cuda.cu Optimized CUDA code for RNN forward pass with dot products
      • timex_op.cpp Timex forward/backward CUDA kernel.
    • run.py Trains RWKV text generation model, evaluates on dataset.
    • src RWKV-v2 RNN Language Generation
      • model.py RWKV v2-RNN model, performance optimized, CUDA, time-decay parameters.
      • model_run.py Language generation model with iterative layers and attention
      • trainer.py Trains RWKV-v2 model with configurable parameters.
      • utils.py RWKV v2-RNN Dataset: Tokenization, Conversion, Sampling.
    • train.py Trains RWKV-v2 RNN, adjusts params, saves models at intervals.
  • RWKV-v3 Train RWKV-v3 models for verification
    • cuda CUDA RNN, dot product, shared memory, timex op, PyTorch kernel.
      • timex_cuda.cu CUDA code for RNN dot product computation with shared memory.
      • timex_op.cpp CUDA kernel: timex op, PyTorch forward-backward.
    • run.py RWKV model training script
    • src Training datasets and tokenizer for RWKV-v3
      • model.py RWKV Language Model: PyTorch, CUDA, time-based mixing
      • model_run.py RWKV-v3 Transformer Model Initialization
      • trainer.py Trainer class: CUDA optimized, data loading, updates params, logs progress.
      • utils.py Custom RWKV dataset and tokenizer, top-p sampling, conversion functions.
    • train.py Trains RWKV-LM and GPT, optimizes GPU, saves partial models.
    • verify.py RWKV-v3/verify.py: Verifies RWKV model using GPT, creates RWKV-GPT and RWKV-RNN
  • RWKV-v4 RWKV-V4: Transformer RNN Language Model Framework
    • cuda Efficient CUDA RNN ops, overflow avoidance for WKV model passes.
      • wkv_cuda.cu RWKV-v4 CUDA kernel: efficient RNN operations, overflow avoidance.
      • wkv_op.cpp WKV model forward/backward passes with CUDA, PyTorch
    • run.py RWKV language model text generation script
    • src Transformer model and trainer with utilities.
      • binidx.py Indexed binary dataset class with memmap support.
      • model.py RWKV model with CUDA, attention, and DeepSpeed optimizer.
      • model_run.py RWKV-v4 Transformer Model CUDA Efficient
      • trainer.py Trains RWKV language model, multi-GPU support.
      • utils.py Python utils: Dataset, vocab, tokenizer, data handling.
    • train.py Trains RWKV-v4 model with DeepSpeed and FP16.
    • verify.py RWKV model initialization and comparison script
  • RWKV-v4neo Trains customizable RWKV chatbot models.
    • chat.py Multilingual chatbot with RWKV language model.
    • cuda CUDA-accelerated RWKV ops for BFloat16
      • wkv5_cuda.cu Optimized CUDA neural network forward pass with shared memory.
      • wkv5_op.cpp CUDA-optimized PyTorch ops for BFloat16 data in wkv5 module
      • wkv_cuda.cu Matrix multiplication, gradients, CUDA RWKV optimization.
      • wkv_cuda_bf16.cu Performs GPU-accelerated neural network forward pass with BFloat16.
      • wkv_op.cpp CUDA functions wrapped for PyTorch with PyBind11
      • wkv_op_bf16.cpp CUDA BF16 Tensor Op Functions for CUDA, Python Bindings
    • img_demoAE.py Image processing code with ToBinary, R_ENCODER, and neural network model.
    • math_demo
      • run.py RWKV NLP model, tokenization classes, RWKV_RNN functions.
    • run.py RWKV Language Model: Text Generation & Refinement
    • src Trains RWKV, AWS S3 storage, prime number utilities.
      • binidx.py Memory-mapped indexed dataset handling in Python
      • dataset.py Custom PyTorch dataset for data loading and resizing.
      • model.py Neural network model setup with CUDA, layer normalization & RWKV_TimeMix.
      • model_img.py RWKV-v4neo: DeepSpeed, VGG16, RWKV-LM model with encoder-decoder
      • model_run.py RWKV Language Model initialization and optimization.
      • trainer.py Trains PyTorch Lightning model, saves on AWS S3
      • utils.py Prime number verification utility code
    • train.py Trains RWKV language model with customizable parameters, PyTorch Lightning.
    • verify.py Trains RWKV language model, prints outputs.
  • RWKV-v5 Large-scale RWKV text model utilities.
    • cuda CUDA-optimized neural network operations and code for shared memory, matrix ops, and parallelism.
      • wkv5_cuda.cu CUDA code optimizes neural network with shared memory, matrix ops, and parallelism.
      • wkv5_op.cpp CUDA-optimized PyTorch neural network operations with BFloat16.
    • demo.jsonl Restaurant Attack, AI Advancements, Legal Drama, and Growth
    • make_data.py Binary file manipulation, RWKV indexing, prime number verification.
    • src RWKV Transformer Training with DeepSpeed and Utilities
      • binidx.py Memory-mapped, indexed dataset handler
      • dataset.py Trains RWKV Language Model dataset.
      • model.py RWKV transformer layer with DeepSpeed CUDA optimization.
      • trainer.py Trains PyTorch Lightning model with AWS S3 saving, LR scheduling, and progress logging.
      • utils.py Prime number calculation and verification.
    • tokenizer
      • rwkv_tokenizer.py Tokenizer using trie for encoding and decoding bytes.
    • train.py Trains RWKV Language Model with Pytorch Lightning and Deepspeed