dmlc--dgl
Zihao Ye
9f32554296
* change the signature of node/edge filter * upd filter * Support multi-dimension node feature in SPMV * push transformer * remove some experimental settings * stable version * hotfix * upd tutorial * upd README * merge * remove redundency * remove tqdm * several changes * Refactor * Refactor * tutorial train * fixed a bug * fixed perf issue * upd * change dir * move un-related to contrib * tutuorial code * remove redundency * upd * upd * upd * upd * improve viz * universal done * halt norm * fixed a bug * add draw graph * fixed several bugs * remove dependency on core * upd format of README * trigger * trigger * upd viz * trigger * add transformer tutorial * fix tutorial * fix readme * small fix on tutorials * url fix in readme * fixed func link * upd
47 行
1.5 KiB
Markdown
47 行
1.5 KiB
Markdown
# Transformer in DGL
|
|
In this example we implement the [Transformer](https://arxiv.org/pdf/1706.03762.pdf) and [Universal Transformer](https://arxiv.org/abs/1807.03819) with ACT in DGL.
|
|
|
|
The folder contains training module and inferencing module (beam decoder) for Transformer and training module for Universal Transformer
|
|
|
|
## Requirements
|
|
|
|
- PyTorch 0.4.1+
|
|
- networkx
|
|
- tqdm
|
|
|
|
## Usage
|
|
|
|
- For training:
|
|
|
|
```
|
|
python translation_train.py [--gpus id1,id2,...] [--N #layers] [--dataset DATASET] [--batch BATCHSIZE] [--universal]
|
|
```
|
|
|
|
- For evaluating BLEU score on test set(by enabling `--print` to see translated text):
|
|
|
|
```
|
|
python translation_test.py [--gpu id] [--N #layers] [--dataset DATASET] [--batch BATCHSIZE] [--checkpoint CHECKPOINT] [--print] [--universal]
|
|
```
|
|
|
|
Available datasets: `copy`, `sort`, `wmt14`, `multi30k`(default).
|
|
|
|
## Test Results
|
|
|
|
### Transfomer
|
|
|
|
- Multi30k: we achieve BLEU score 35.41 with default setting on Multi30k dataset, without using pre-trained embeddings. (if we set the number of layers to 2, the BLEU score could reach 36.45).
|
|
- WMT14: work in progress
|
|
|
|
### Universal Transformer
|
|
|
|
- work in progress
|
|
|
|
## Notes
|
|
|
|
- Currently we do not support Multi-GPU training(this will be fixed soon), you should only specifiy only one gpu\_id when running the training script.
|
|
|
|
## Reference
|
|
|
|
- [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
|
|
- [Tensor2Tensor](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/)
|