dmlc--dgl
Zihao Ye
264d96cdf5
* upd * upd * upd * upd * upd * upd * fix pinsage also * upd * upd * upd Co-authored-by: Ubuntu <ubuntu@ip-172-31-29-3.us-east-2.compute.internal> Co-authored-by: Quan Gan <coin2028@hotmail.com> Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
44 行
1.4 KiB
Markdown
44 行
1.4 KiB
Markdown
# Transformer in DGL
|
|
|
|
**This example is out-dated, please refer to [BP-Transformer](http://github.com/yzh119/bpt) for efficient (Sparse) Transformer implementation in DGL.**
|
|
|
|
In this example we implement the [Transformer](https://arxiv.org/pdf/1706.03762.pdf) with ACT in DGL.
|
|
|
|
The folder contains training module and inferencing module (beam decoder) for Transformer.
|
|
|
|
## Dependencies
|
|
|
|
- PyTorch 0.4.1+
|
|
- networkx
|
|
- tqdm
|
|
- requests
|
|
- matplotlib
|
|
|
|
## Usage
|
|
|
|
- For training:
|
|
|
|
```
|
|
python3 translation_train.py [--gpus id1,id2,...] [--N #layers] [--dataset DATASET] [--batch BATCHSIZE] [--universal]
|
|
```
|
|
|
|
By specifying multiple gpu ids separated by comma, we will employ multi-gpu training with multiprocessing.
|
|
|
|
- For evaluating BLEU score on test set(by enabling `--print` to see translated text):
|
|
|
|
```
|
|
python3 translation_test.py [--gpu id] [--N #layers] [--dataset DATASET] [--batch BATCHSIZE] [--checkpoint CHECKPOINT] [--print] [--universal]
|
|
```
|
|
|
|
Available datasets: `copy`, `sort`, `wmt14`, `multi30k`(default).
|
|
|
|
## Test Results
|
|
|
|
- Multi30k: we achieve BLEU score 35.41 with default setting on Multi30k dataset, without using pre-trained embeddings. (if we set the number of layers to 2, the BLEU score could reach 36.45).
|
|
- WMT14: work in progress
|
|
|
|
## Reference
|
|
|
|
- [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
|
|
- [Tensor2Tensor](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/)
|