rasbt--deeplearning-models
rasbt
358 行
24 KiB
Markdown
358 行
24 KiB
Markdown
|
|
# Deep Learning Models
|
|
|
|
A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.
|
|
|
|
|
|
|
|
## Traditional Machine Learning
|
|
|
|
- Perceptron [](pytorch_ipynb/basic-ml/perceptron.ipynb) [](tensorflow1_ipynb/basic-ml/perceptron.ipynb)
|
|
|
|
- Logistic Regression [](pytorch_ipynb/basic-ml/logistic-regression.ipynb) [](tensorflow1_ipynb/basic-ml/logistic-regression.ipynb)
|
|
|
|
- Softmax Regression (Multinomial Logistic Regression) [](pytorch_ipynb/basic-ml/softmax-regression.ipynb) [](tensorflow1_ipynb/basic-ml/softmax-regression.ipynb)
|
|
|
|
- Softmax Regression with MLxtend's plot_decision_regions on Iris [](pytorch_ipynb/basic-ml/softmax-regression-mlxtend-1.ipynb)
|
|
|
|
|
|
## Multilayer Perceptrons
|
|
|
|
|
|
|
|
- Multilayer Perceptron [](pytorch-lightning_ipynb/mlp/mlp-basic.ipynb) [](pytorch_ipynb/mlp/mlp-basic.ipynb) [](tensorflow1_ipynb/mlp/mlp-basic.ipynb)
|
|
|
|
- Multilayer Perceptron with Dropout [](pytorch-lightning_ipynb/mlp/mlp-dropout.ipynb) [](pytorch_ipynb/mlp/mlp-dropout.ipynb) [](tensorflow1_ipynb/mlp/mlp-dropout.ipynb)
|
|
|
|
- Multilayer Perceptron with Batch Normalization [](pytorch-lightning_ipynb/mlp/mlp-batchnorm.ipynb) [](pytorch_ipynb/mlp/mlp-batchnorm.ipynb) [](tensorflow1_ipynb/mlp/mlp-batchtnorm.ipynb)
|
|
|
|
- Multilayer Perceptron with Backpropagation from Scratch [](pytorch_ipynb/mlp/mlp-fromscratch__sigmoid-mse.ipynb) [](tensorflow1_ipynb/mlp/mlp-fromscratch__sigmoid-mse.ipynb)
|
|
|
|
|
|
## Convolutional Neural Networks
|
|
|
|
|
|
#### Basic
|
|
|
|
- Convolutional Neural Network [](pytorch-lightning_ipynb/cnn/cnn-basic.ipynb) [](pytorch_ipynb/cnn/cnn-basic.ipynb) [](tensorflow1_ipynb/cnn/cnn-basic.ipynb)
|
|
|
|
- Convolutional Neural Network with He Initialization [](pytorch-lightning_ipynb/cnn/cnn-he-init.ipynb) [](pytorch_ipynb/cnn/cnn-he-init.ipynb)
|
|
|
|
#### Concepts
|
|
|
|
- Replacing Fully-Connnected by Equivalent Convolutional Layers [](pytorch_ipynb/cnn/fc-to-conv.ipynb)
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
#### AlexNet
|
|
|
|
- AlexNet on CIFAR-10 [](pytorch-lightning_ipynb/cnn/cnn-alexnet-cifar10.ipynb) [](pytorch_ipynb/cnn/cnn-alexnet-cifar10.ipynb)
|
|
|
|
|
|
#### DenseNet
|
|
|
|
- DenseNet-121 Digit Classifier Trained on MNIST [](pytorch-lightning_ipynb/cnn/cnn-densenet121-mnist.ipynb) [](pytorch_ipynb/cnn/cnn-densenet121-mnist.ipynb)
|
|
|
|
- DenseNet-121 Image Classifier Trained on CIFAR-10 [](pytorch-lightning_ipynb/cnn/cnn-densenet121-cifar10.ipynb) [](pytorch_ipynb/cnn/cnn-densenet121-cifar10.ipynb)
|
|
|
|
|
|
#### Fully Convolutional
|
|
|
|
- "All Convolutionl Net" -- A Fully Convolutional Neural Network [](pytorch-lightning_ipynb/cnn/cnn-allconv.ipynb) [](pytorch_ipynb/cnn/cnn-allconv.ipynb)
|
|
|
|
#### LeNet
|
|
|
|
- LeNet-5 on MNIST [](pytorch-lightning_ipynb/cnn/cnn-lenet5-mnist.ipynb) [](pytorch_ipynb/cnn/cnn-lenet5-mnist.ipynb)
|
|
|
|
- LeNet-5 on CIFAR-10 [](pytorch-lightning_ipynb/cnn/cnn-lenet5-cifar10.ipynb) [](pytorch_ipynb/cnn/cnn-lenet5-cifar10.ipynb)
|
|
|
|
- LeNet-5 on QuickDraw [](pytorch-lightning_ipynb/cnn/cnn-lenet5-quickdraw.ipynb) [](pytorch_ipynb/cnn/cnn-lenet5-quickdraw.ipynb)
|
|
|
|
|
|
#### MobileNet
|
|
|
|
- MobileNet-v2 on Cifar-10 [](pytorch-lightning_ipynb/cnn/cnn-mobilenet-v2-cifar10.ipynb) [](pytorch_ipynb/cnn/cnn-mobilenet-v2-cifar10.ipynb)
|
|
|
|
- MobileNet-v3 small on Cifar-10 [](pytorch-lightning_ipynb/cnn/cnn-mobilenet-v3-small-cifar10.ipynb) [](pytorch_ipynb/cnn/cnn-mobilenet-v3-small-cifar10.ipynb)
|
|
|
|
- MobileNet-v3 large on Cifar-10 [](pytorch-lightning_ipynb/cnn/cnn-mobilenet-v3-large-cifar10.ipynb) [](pytorch_ipynb/cnn/cnn-mobilenet-v3-large-cifar10.ipynb)
|
|
|
|
|
|
#### Network in Network
|
|
|
|
- Network in Network Trained on CIFAR-10 [](pytorch-lightning_ipynb/cnn/cnn-nin-cifar10.ipynb) [](pytorch_ipynb/cnn/nin-cifar10.ipynb)
|
|
|
|
#### VGG
|
|
|
|
- Convolutional Neural Network VGG-16 Trained on CIFAR-10 [](pytorch-lightning_ipynb/cnn/cnn-vgg16.ipynb) [](pytorch_ipynb/cnn/cnn-vgg16.ipynb) [](tensorflow1_ipynb/cnn/cnn-vgg16.ipynb)
|
|
|
|
- VGG-16 Smile Classifier Trained on CelebA [](pytorch-lightning_ipynb/cnn/cnn-vgg16-celeba.ipynb) [](pytorch_ipynb/cnn/cnn-vgg16-celeba.ipynb)
|
|
|
|
- VGG-16 Dogs vs Cats Classifier [](pytorch_ipynb/cnn/cnn-vgg16-cats-dogs.ipynb)
|
|
|
|
- Convolutional Neural Network VGG-19 [](pytorch-lightning_ipynb/cnn/cnn-vgg19.ipynb) [](pytorch_ipynb/cnn/cnn-vgg19.ipynb)
|
|
|
|
|
|
#### ResNet
|
|
|
|
- ResNet and Residual Blocks [](pytorch_ipynb/cnn/resnet-ex-1.ipynb)
|
|
|
|
- ResNet-18 Digit Classifier Trained on MNIST [](pytorch_ipynb/cnn/cnn-resnet18-mnist.ipynb)
|
|
|
|
|
|
- ResNet-18 Gender Classifier Trained on CelebA [](pytorch_ipynb/cnn/cnn-resnet18-celeba-dataparallel.ipynb)
|
|
|
|
- ResNet-34 Digit Classifier Trained on MNIST [](pytorch_ipynb/cnn/cnn-resnet34-mnist.ipynb)
|
|
|
|
- ResNet-34 Object Classifier Trained on QuickDraw [](pytorch_ipynb/cnn/cnn-resnet34-quickdraw.ipynb)
|
|
|
|
- ResNet-34 Gender Classifier Trained on CelebA [](pytorch_ipynb/cnn/cnn-resnet34-celeba-dataparallel.ipynb)
|
|
|
|
- ResNet-50 Digit Classifier Trained on MNIST [](pytorch_ipynb/cnn/cnn-resnet50-mnist.ipynb)
|
|
|
|
- ResNet-50 Gender Classifier Trained on CelebA [](pytorch_ipynb/cnn/cnn-resnet50-celeba-dataparallel.ipynb)
|
|
|
|
- ResNet-101 Gender Classifier Trained on CelebA [](pytorch_ipynb/cnn/cnn-resnet101-celeba.ipynb)
|
|
|
|
- ResNet-101 Trained on CIFAR-10 [](pytorch_ipynb/cnn/cnn-resnet101-cifar10.ipynb)
|
|
|
|
- ResNet-152 Gender Classifier Trained on CelebA [](pytorch_ipynb/cnn/cnn-resnet152-celeba.ipynb)
|
|
|
|
---
|
|
|
|
|
|
## Normalization Layers
|
|
|
|
- BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier [](pytorch_ipynb/cnn/nin-cifar10_batchnorm.ipynb)
|
|
|
|
- Filter Response Normalization for Network-in-Network CIFAR-10 Classifier [](pytorch_ipynb/cnn/nin-cifar10_filter-response-norm.ipynb)
|
|
|
|
|
|
## Metric Learning
|
|
|
|
- Siamese Network with Multilayer Perceptrons [](tensorflow1_ipynb/metric/siamese-1.ipynb)
|
|
|
|
## Autoencoders
|
|
|
|
#### Fully-connected Autoencoders
|
|
|
|
- Autoencoder (MNIST) [](pytorch_ipynb/autoencoder/ae-basic.ipynb) [](tensorflow1_ipynb/autoencoder/ae-basic.ipynb)
|
|
|
|
|
|
- Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier [](pytorch_ipynb/autoencoder/ae-basic-with-rf.ipynb) [](tensorflow1_ipynb/autoencoder/ae-basic-with-rf.ipynb)
|
|
|
|
|
|
|
|
#### Convolutional Autoencoders
|
|
|
|
- Convolutional Autoencoder with Deconvolutions / Transposed Convolutions [](pytorch_ipynb/autoencoder/ae-deconv.ipynb) [](tensorflow1_ipynb/autoencoder/ae-deconv.ipynb)
|
|
|
|
|
|
- Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance [](pytorch_ipynb/autoencoder/ae-deconv-jaccard.ipynb)
|
|
|
|
- Convolutional Autoencoder with Deconvolutions (without pooling operations) [](pytorch_ipynb/autoencoder/ae-deconv-nopool.ipynb)
|
|
|
|
- Convolutional Autoencoder with Nearest-neighbor Interpolation [](pytorch_ipynb/autoencoder/ae-conv-nneighbor.ipynb) [](tensorflow1_ipynb/autoencoder/ae-conv-nneighbor.ipynb)
|
|
|
|
- Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA [](pytorch_ipynb/autoencoder/ae-conv-nneighbor-celeba.ipynb)
|
|
|
|
- Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw [](pytorch_ipynb/autoencoder/ae-conv-nneighbor-quickdraw-1.ipynb)
|
|
|
|
#### Variational Autoencoders
|
|
|
|
- Variational Autoencoder [](pytorch_ipynb/autoencoder/ae-var.ipynb)
|
|
|
|
- Convolutional Variational Autoencoder [](pytorch_ipynb/autoencoder/ae-conv-var.ipynb)
|
|
|
|
|
|
#### Conditional Variational Autoencoders
|
|
|
|
- Conditional Variational Autoencoder (with labels in reconstruction loss) [](pytorch_ipynb/autoencoder/ae-cvae.ipynb)
|
|
|
|
|
|
- Conditional Variational Autoencoder (without labels in reconstruction loss) [](pytorch_ipynb/autoencoder/ae-cvae_no-out-concat.ipynb)
|
|
|
|
|
|
- Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss) [](pytorch_ipynb/autoencoder/ae-cnn-cvae.ipynb)
|
|
|
|
|
|
- Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss) [](pytorch_ipynb/autoencoder/ae-cnn-cvae_no-out-concat.ipynb)
|
|
|
|
|
|
## Generative Adversarial Networks (GANs)
|
|
|
|
- Fully Connected GAN on MNIST [](pytorch_ipynb/gan/gan.ipynb) [](tensorflow1_ipynb/gan/gan.ipynb)
|
|
|
|
|
|
- Fully Connected Wasserstein GAN on MNIST [](pytorch_ipynb/gan/wgan-1.ipynb)
|
|
|
|
- Convolutional GAN on MNIST [](pytorch_ipynb/gan/gan-conv.ipynb) [](tensorflow1_ipynb/gan/gan-conv.ipynb)
|
|
|
|
|
|
- Convolutional GAN on MNIST with Label Smoothing [](pytorch_ipynb/gan/gan-conv-smoothing.ipynb) [](tensorflow1_ipynb/gan/gan-conv-smoothing.ipynb)
|
|
|
|
- Convolutional Wasserstein GAN on MNIST [](pytorch_ipynb/gan/dc-wgan-1.ipynb)
|
|
|
|
|
|
- "Deep Convolutional GAN" (DCGAN) on Cats and Dogs Images [](pytorch_ipynb/gan/dcgan-cats-and-dogs.ipynb)
|
|
|
|
|
|
- "Deep Convolutional GAN" (DCGAN) on CelebA Face Images [](pytorch_ipynb/gan/dcgan-celeba.ipynb)
|
|
|
|
|
|
## Graph Neural Networks (GNNs)
|
|
|
|
- Most Basic Graph Neural Network with Gaussian Filter on MNIST [](pytorch_ipynb/gnn/gnn-basic-1.ipynb)
|
|
|
|
- Basic Graph Neural Network with Edge Prediction on MNIST [](pytorch_ipynb/gnn/gnn-basic-edge-1.ipynb)
|
|
|
|
- Basic Graph Neural Network with Spectral Graph Convolution on MNIST [](pytorch_ipynb/gnn/gnn-basic-graph-spectral-1.ipynb)
|
|
|
|
## Recurrent Neural Networks (RNNs)
|
|
|
|
|
|
#### Many-to-one: Sentiment Analysis / Classification
|
|
|
|
- A simple single-layer RNN (IMDB) [](pytorch_ipynb/rnn/rnn_simple_imdb.ipynb)
|
|
|
|
- A simple single-layer RNN with packed sequences to ignore padding characters (IMDB) [](pytorch_ipynb/rnn/rnn_simple_packed_imdb.ipynb)
|
|
|
|
- RNN with LSTM cells (IMDB) [](pytorch_ipynb/rnn/rnn_lstm_packed_imdb.ipynb)
|
|
|
|
- RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors [](pytorch_ipynb/rnn/rnn_lstm_packed_imdb-glove.ipynb)
|
|
|
|
- RNN with LSTM cells and Own Dataset in CSV Format (IMDB) [](pytorch_ipynb/rnn/rnn_lstm_packed_own_csv_imdb.ipynb)
|
|
|
|
- RNN with GRU cells (IMDB) [](pytorch_ipynb/rnn/rnn_gru_packed_imdb.ipynb)
|
|
|
|
- Multilayer bi-directional RNN (IMDB) [](pytorch_ipynb/rnn/rnn_lstm_bi_imdb.ipynb)
|
|
|
|
- Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News) [](pytorch_ipynb/rnn/rnn_bi_multilayer_lstm_own_csv_agnews.ipynb)
|
|
|
|
|
|
#### Many-to-Many / Sequence-to-Sequence
|
|
|
|
- A simple character RNN to generate new text (Charles Dickens) [](pytorch_ipynb/rnn/char_rnn-charlesdickens.ipynb)
|
|
|
|
## Ordinal Regression
|
|
|
|
- Ordinal Regression CNN -- CORAL w. ResNet34 on AFAD-Lite [](pytorch_ipynb/ordinal/ordinal-cnn-coral-afadlite.ipynb)
|
|
|
|
- Ordinal Regression CNN -- Niu et al. 2016 w. ResNet34 on AFAD-Lite [](pytorch_ipynb/ordinal/ordinal-cnn-niu-afadlite.ipynb)
|
|
|
|
- Ordinal Regression CNN -- Beckham and Pal 2016 w. ResNet34 on AFAD-Lite [](pytorch_ipynb/ordinal/ordinal-cnn-beckham2016-afadlite.ipynb)
|
|
|
|
|
|
## Tips and Tricks
|
|
|
|
- Cyclical Learning Rate [](pytorch_ipynb/tricks/cyclical-learning-rate.ipynb)
|
|
|
|
- Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet) [](pytorch_ipynb/tricks/cnn-alexnet-cifar10-batchincrease.ipynb)
|
|
|
|
- Gradient Clipping (w. MLP on MNIST) [](pytorch_ipynb/tricks/gradclipping_mlp.ipynb)
|
|
|
|
|
|
## Transfer Learning
|
|
|
|
- Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10) [](pytorch_ipynb/transfer/transferlearning-vgg16-cifar10-1.ipynb)
|
|
|
|
## Visualization and Interpretation
|
|
|
|
- Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) [](pytorch_ipynb/viz/cnns/cats-and-dogs/cnn-viz-grad__vgg16-cats-dogs.ipynb)
|
|
|
|
- Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) [](pytorch_ipynb/viz/cnns/cats-and-dogs/cnn-viz-guided-backprop__vgg16-cats-dogs.ipynb)
|
|
|
|
|
|
## PyTorch Workflows and Mechanics
|
|
|
|
|
|
#### PyTorch Lightning Examples
|
|
|
|
- MLP in Lightning with TensorBoard -- continue training the last model [](pytorch_ipynb/lightning/lightning-mlp.ipynb)
|
|
|
|
- MLP in Lightning with TensorBoard -- checkpointing best model [](pytorch_ipynb/lightning/lightning-mlp-best-model)
|
|
|
|
|
|
|
|
#### Custom Datasets
|
|
|
|
- Custom Data Loader Example for PNG Files [](pytorch_ipynb/mechanics/custom-dataloader-png/custom-dataloader-example.ipynb)
|
|
|
|
|
|
- Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5 [](pytorch_ipynb/mechanics/custom-data-loader-csv.ipynb)
|
|
|
|
- Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA [](pytorch_ipynb/mechanics/custom-data-loader-celeba.ipynb)
|
|
|
|
- Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw [](pytorch_ipynb/mechanics/custom-data-loader-quickdraw.ipynb)
|
|
|
|
- Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset [](pytorch_ipynb/mechanics/custom-data-loader-svhn.ipynb)
|
|
|
|
- Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD) [](pytorch_ipynb/mechanics/custom-data-loader-afad.ipynb)
|
|
|
|
- Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images [](pytorch_ipynb/mechanics/custom-data-loader_dating-historical-color-images.ipynb)
|
|
|
|
- Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST [](pytorch_ipynb/mechanics/custom-data-loader-quickdraw.ipynb)
|
|
|
|
#### Training and Preprocessing
|
|
|
|
- Generating Validation Set Splits [](pytorch_ipynb/mechanics/validation-splits.ipynb)
|
|
|
|
- Dataloading with Pinned Memory [](pytorch_ipynb/cnn/cnn-resnet34-cifar10-pinmem.ipynb)
|
|
|
|
- Standardizing Images [](pytorch_ipynb/cnn/cnn-standardized.ipynb)
|
|
|
|
- Image Transformation Examples [](pytorch_ipynb/mechanics/torchvision-transform-examples.ipynb)
|
|
|
|
- Char-RNN with Own Text File [](pytorch_ipynb/rnn/char_rnn-charlesdickens.ipynb)
|
|
|
|
- Sentiment Classification RNN with Own CSV File [](pytorch_ipynb/rnn/rnn_lstm_packed_own_csv_imdb.ipynb)
|
|
|
|
|
|
#### Improving Memory Efficiency
|
|
|
|
- Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10) [](pytorch_ipynb/mechanics/gradient-checkpointing-nin.ipynb)
|
|
|
|
#### Parallel Computing
|
|
|
|
- Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA [](pytorch_ipynb/cnn/cnn-vgg16-celeba-data-parallel.ipynb)
|
|
|
|
- Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example) [](pytorch_ipynb/mechanics/model-pipeline-vgg16.ipynb)
|
|
|
|
#### Other
|
|
|
|
- PyTorch with and without Deterministic Behavior -- Runtime Benchmark [](pytorch_ipynb/mechanics/deterministic_benchmark.ipynb)
|
|
|
|
- Sequential API and hooks [](pytorch_ipynb/mechanics/mlp-sequential.ipynb)
|
|
|
|
- Weight Sharing Within a Layer [](pytorch_ipynb/mechanics/cnn-weight-sharing.ipynb)
|
|
|
|
- Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib [](pytorch_ipynb/mechanics/plot-jupyter-matplotlib.ipynb)
|
|
|
|
|
|
#### Autograd
|
|
|
|
- Getting Gradients of an Intermediate Variable in PyTorch [](pytorch_ipynb/mechanics/manual-gradients.ipynb)
|
|
|
|
## TensorFlow Workflows and Mechanics
|
|
|
|
#### Custom Datasets
|
|
|
|
- Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives [](tensorflow1_ipynb/mechanics/image-data-chunking-npz.ipynb)
|
|
|
|
- Storing an Image Dataset for Minibatch Training using HDF5 [](tensorflow1_ipynb/mechanics/image-data-chunking-hdf5.ipynb)
|
|
|
|
- Using Input Pipelines to Read Data from TFRecords Files [](tensorflow1_ipynb/mechanics/tfrecords.ipynb)
|
|
|
|
- Using Queue Runners to Feed Images Directly from Disk [](tensorflow1_ipynb/mechanics/file-queues.ipynb)
|
|
|
|
- Using TensorFlow's Dataset API [](tensorflow1_ipynb/mechanics/dataset-api.ipynb)
|
|
|
|
|
|
#### Training and Preprocessing
|
|
|
|
- Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives [](tensorflow1_ipynb/mechanics/saving-and-reloading-models.ipynb)
|
|
|