30c4625171
fix. ( #794 )
2019-08-27 09:56:38 -07:00
66971c1aa4
[Feature] Add edge sampling for link prediction ( #780 )
...
* add edge sampler.
* add test and run.
* add negative sampling.
* remap the edge subgraph vid.
* negative graph excludes edges of positive edgs.
* remove print.
* avoid sampling NodeFlow when expand_factor or num_hops is 0.
* fix a bug when excluding nodes in negative graph.
* support multigraph.
* exclude positive edges.
* fix memory leak.
* return subgraph object directly.
* fix many problems.
* add comments.
* address comments
2019-08-21 23:05:24 -07:00
02029dce9b
Sync initializers ( #772 )
2019-08-17 20:25:54 +08:00
16061925d5
[NodeFlow] Non-uniform neighbor sampling ( #711 )
...
* nonuniform sampler
* unit test
* test on out neighbors
* error checks
* lint
* fix
* clarification
* use macro switcher
* use empty array for uniform sampling
* oops
* Revert "oops"
This reverts commit a11f9ae707aaeb67fb5921c887a17d3711d5b04a.
* Revert "use empty array for uniform sampling"
This reverts commit 8526ce4cade89f2c1b09a08aca8830375ebafb31.
* re-reverting
* use a method
2019-08-07 09:42:08 +08:00
c3516f1a8e
[Network] Refactoring Communicator ( #679 )
...
* Refactoring Communicator
* fix lint
* change non-const reference
* add header file
* use MemoryBuffer
* update PR
* fix bug on csr shape
* zero-copy msg_queue
* fix lint
* fix lint
* fix lint
* add header file
* fix windows build error
* fix windows build error
* update
* fix lint
* update
* fix lint
* fix lint
* add more test
* fix windows test
* update windows test
* update windows test
* update windows test
* update
* fix lint
* fix lint
* update
* update
* update
* update
* use STATUS code
* update test
* remove mem_cpy
* fix lint
* update
* finish
* ConstructNFTensor
* add test for deallocator
* update
* fix lint
2019-08-05 10:49:25 +08:00
e9e587b6cb
[Refactor] C random number generator ( #729 )
...
* rng refactor
* fix bugs
* unit test
* remove setsize
* lint
* fix test
* use explicit instantiation instead of inlining
* stricter test
* use tvm solution
* moved python interface to dgl.random
* lint
* address comments
* make getthreadid an inline function
2019-07-31 17:54:30 +08:00
67dc11971f
[Refactor] Use object system for all CAPIs ( #716 )
...
* WIP: using object system for graph
* c++ side refactoring done; compiled
* remove stale apis
* fix bug in DGLGraphCreate; passed test_graph.py
* fix bug in python modify; passed utest for pytorch/cpu
* fix lint
* address comments
2019-07-23 12:47:05 -04:00
8059fa6f8b
print errors if users sample from mutable graph. ( #649 )
...
* give warning.
* fix.
* update error msg.
2019-06-12 17:54:02 +08:00
94ecb8eb96
[BUGFIX] copy graph index to shared memory. ( #634 )
...
* copy graph index to shared memory.
* fix.
* fix.
* fix.
* use a diff name for in-csr and out-csr.
* fix lint.
* remove print.
* add test.
* add comments.
2019-06-12 01:32:25 -07:00
16ec2a8b10
we need to sync for mxnet. ( #648 )
2019-06-11 18:22:21 -07:00
411bcd7e6c
[BUGFIX] extand DGLGraph graph_data. ( #632 )
...
* enable tutorial test in CI.
* extand DGLGraph graph_data.
* update doc.
* Revert "enable tutorial test in CI."
This reverts commit cd774067180922bb6ae979bde4aecbffc61c8147.
* accept DGLGraph in graph store.
2019-06-09 17:22:04 -07:00
2e9949d228
[BUGFIX] Fix bugs in shared mem graph store. ( #630 )
...
* fix graph store for Pytorch.
* add test.
* fix dtype error in test
* disable test on GPU.
* test avoid windows.
* fix shared-memory test.
* use script to control testing environment.
* update test.
* enable all tests.
* fix test script.
2019-06-09 13:08:47 -07:00
5da57663cc
[BUGFIX] fix sampler. ( #616 )
...
* fix sampler.
* update doc.
* fix.
2019-06-06 20:47:59 -07:00
01a4cc5b7b
[Graph] Add API to convert graph to simple graph ( #587 )
...
* to simple
* WIP: multigraph flag
* graph index refactor; pass basic testing
* graph index refactor; pass basic testing
* fix bug in to_simple; pass torch test
* fix mx utest
* fix example
* fix lint
* fix ci
* poke ci
* poke ci
* WIP
* poke ci
* poke ci
* poke ci
* change ci workspace
* poke ci
* poke ci
* poke ci
* poke ci
* delete ci
* use enum for multigraph flag
2019-06-01 23:32:46 -04:00
372203f04a
[Test] Fix tests in test_shared_mem_store. ( #588 )
...
* fix test.
* better assert.
* more asserts.
* print to stderr.
* destroy g.
* fix tests.
* add timeout in sync_barrier.
* test _sync_barrier.
* fix.
* avoid printing messages.
* fix test.
* fix test.
* fix.
2019-06-01 21:18:01 -04:00
16af6ebd3b
[Test] Add cpp test to CI ( #567 )
...
* add cpptest to ci
* try windows
* Update Jenkinsfile
* Update Jenkinsfile
* Update Jenkinsfile
* Update task_cpp_unit_test.bat
* Change localhost to 127.0.0.1
* windows socket gtest
* lint
* Update task_cpp_unit_test.bat
* fix batch script
* export all symbols
* Update Jenkinsfile
* compile once
* try mxnet compact
2019-05-30 11:01:16 -04:00
21e172d076
[Bugfix] fix numpy integer bug ( #580 )
2019-05-29 21:48:36 -04:00
b3097725f2
[BugFix] Fix error when bfs_level = 0 in Entity Classification with RGCN ( #559 )
2019-05-25 17:38:22 -07:00
8f378d90ad
[BUGFIX] fix bugs for running GCN on giant graphs. ( #561 )
...
* load mxnet csr.
* enable load large csr.
* fix
* fix.
* fix int overflow.
* fix test.
2019-05-25 11:09:33 -07:00
924efc6520
[Perf] Improve performance of graph store. ( #554 )
...
* fix.
* use inplace.
* move to shared memory graph store.
* fix.
* add more unit tests.
* fix.
* fix test.
* fix test.
* disable test.
* fix.
2019-05-23 12:23:31 -07:00
b2b8be25bb
[API] update graph store API. ( #549 )
...
* add init_ndata and init_edata in DGLGraph.
* adjust SharedMemoryGraph API.
* print warning.
* fix comment.
* update example
* fix.
* fix examples.
* add unit tests.
* add comments.
2019-05-21 03:27:05 -07:00
cdfca992e1
[BUGFix] Improve multi-processing training ( #526 )
...
* fix.
* add comment.
* remove.
* temp fix.
* initialize for shared memory.
* fix graphsage.
* fix gcn.
* add more unit tests.
* add more tests.
* avoid creating shared-memory exclusively.
* redefine remote initializer.
* improve initializer.
* fix unit test.
* fix lint.
* fix lint.
* initialize data in the graph store server properly.
* fix test.
* fix test.
* fix test.
* small fix.
* add comments.
* cleanup server.
* test graph store with a random port.
* print.
* print to stderr.
* test1
* test2
* remove comment.
* adjust the initializer signature.
2019-05-20 14:23:19 -07:00
3a1392e641
[Model] add multiprocessing training with sampling. ( #484 )
...
* reorganize sampling code.
* add multi-process training.
* speed up gcn_cv
* fix graphsage_cv.
* add new API in graph store.
* update barrier impl.
* support both local and distributed training.
* fix multiprocess train.
* fix.
* fix barrier.
* add script for loading data.
* multiprocessing sampling.
* accel training.
* replace pull with spmv for speedup.
* nodeflow copy from parent with context.
* enable GPU.
* fix a bug in graph store.
* enable multi-GPU training.
* fix lint.
* add comments.
* rename to run_store_server.py
* fix gcn_cv.
* fix a minor bug in sampler.
* handle error better in graph store.
* improve graphsage_cv for distributed mode.
* update README.
* fix.
* update.
2019-05-06 23:51:56 -07:00
fe7d5e9b8b
[Sampler] Change Distributed Sampler API ( #499 )
...
* Change Distributed Sampler API
* fix lint
* fix lint
* update demo
* update
* update
* update
* update demo
* update demo
2019-04-22 09:41:47 +08:00
b89dcce16c
[RPC] Refactoring networking APIs ( #496 )
...
* Refactoring network API
* update demo
* update
* update demo
* update demo
* add num_sender
* update
* fix lint
* fix lint
* fix lint
* update
2019-04-16 16:33:29 +08:00
bfdd1eaa44
[Feature] Create shared memory graph store. ( #468 )
...
* accelerate gcn_ns.
* add timing.
* run infer with whole graph.
* distributed gcn_ns.
* reconstruct gcn_ns.
* minor fix.
* change graphsage_cv for numa.
* fix #OMP threads.
* accelerate graphsage_cv.
* fix a weird bug.
* add profiler in graphsage_cv.
* accelerate graphsage_cv.
manually aggregate neighbors' embeddings with pull.
* load csr directly in gcn_ns_sc.
* parallel sort for graph index.
* Revert "parallel sort for graph index."
This reverts commit 86fe2c7117fe5e56b0d481b39849c258b166945b.
* run gcn_ns_sc on GPUs.
* acc gcn_cv_sc.
* change gcn_cv for numa.
* fix gcn_cv to use numa and gpu.
* improve graphsage_cv to use numa and gpu.
* improve gcn_ns.
* improve graphsage_cv.
* init shared memory graph store.
* fix.
* enable init ndata.
* improve tests.
* add bidirectional communication.
* link to rt.
* fix compilation error.
* fix shared memory init.
* use MessageQueue for inter-process communication.
* reconstruct immutable graph csr.
* fix gcn.
* load csr to shared memory.
* fix minor bugs.
* add comments.
* refactor SharedMemory.
* fix bugs in ImmutableGraph.
* create CSR graph from shared memory.
* add more test for loading a csr graph.
* terminate graph store properly.
* allow initializing ndata in the graph store server.
* use RPC for inter-process communication.
* a script for loading a graph.
* allow customizing port.
* list all ndata and edata.
* support dtype.
* reorganize SharedMemoryGraphStore.
* fix ndata shape.
* reconstruct gcn_ns.
* print info.
* set omp in gcn_ns.
* reset sampling examples.
* fix lint.
* fix lint.
* reset gcn.
* disable shared memory in windows.
* fix.
* fix.
* reset changes.
* revert nodeflow changes.
* fix cmake.
* fix test.
* fix test.
* fix test.
* fix test.
* add comments.
* fix test.
* move vector out.
* fix lint.
* fix lint.
* move SharedMemory.
* update cmake.
* update comment.
* fix comments.
* Revert "update cmake."
This reverts commit 592445e37077f70a6e3f2e5245f9a3d086b04f3b.
* update cmake.
* add comments.
* rename.
* change the comment.
* fix a bug.
* rename.
* add comments.
* add comments.
* add init_edata.
* rewrite memory alloc.
* move vector to CSR.
* fix.
* init data.
* Revert "init data."
This reverts commit 2b217b9553911b7dd84a9f1d9b68430b5aa18e23.
* init data.
* init new columns correctly.
2019-04-08 12:09:02 -07:00
8058f1c579
[Fix] Update inner API of distributed sampler ( #478 )
...
* update inner API of distributed sampler
* update
2019-04-05 10:00:36 +08:00
da3ab84c3c
Add demo for distributed sampler ( #474 )
...
* add C++ rpc infrastructure and distributed sampler
* update
* update lint
* update lint
* update lint
* update
* update
* update
* updare
* update
* update
* update
* update serialize and unittest
* update serialize
* lint
* update
* update
* update
* update
* update
* update
* update unittest
* put Finalize() to __del__
* update unittest
* update
* delete buffer in Finalize
* update unittest
* update unittest
* update unittest
* update unittest
* update
* update
* fix small bug
* windows socket impl
* update API
* fix bug in serialize
* fix bug in serialzie
* set parent graph
* update
* update
* update
* update
* update
* update
* fix lint
* fix lint
* fix
* fix windows compilation error
* fix windows error
* change API to lower-case
* update test
* fix typo
* update
* add SamplerPool
* add SamplerPool
* update
* update test
* update
* update
* update
* update
* add example
* update
* update
* add distributed sampler demo
* add index
* update demo of distributed sampler
* fix lower-case
* print subg index
* update README.md
* update
* remove --gpu args
2019-04-03 16:09:59 -07:00
5d494c6207
[RPC] add C++ RPC infrastructure and distributed sampler ( #465 )
...
* add C++ rpc infrastructure and distributed sampler
* update
* update lint
* update lint
* update lint
* update
* update
* update
* updare
* update
* update
* update
* update serialize and unittest
* update serialize
* lint
* update
* update
* update
* update
* update
* update
* update unittest
* put Finalize() to __del__
* update unittest
* update
* delete buffer in Finalize
* update unittest
* update unittest
* update unittest
* update unittest
* update
* update
* fix small bug
* windows socket impl
* update API
* fix bug in serialize
* fix bug in serialzie
* set parent graph
* update
* update
* update
* update
* update
* update
* fix lint
* fix lint
* fix
* fix windows compilation error
* fix windows error
* change API to lower-case
* update test
* fix typo
* update
* add SamplerPool
* add SamplerPool
* update
* update test
* update
* update
* update
* update
* add example
* update
* update
2019-03-29 16:48:45 +08:00
6066fee935
[Model][Feature] PinSage & Random Walk with Restart ( #453 )
...
* random walk traces generation
* remove outdated comments
* oops put in the wrong place
* explicit inline
* moving rand_r to util
* pinsage-like model on movielens
* the code runs now
* support cuda
* using readonly graph
* moving random walk to public function
* per-thread seed and openmp support
* pinsage-like model on movielens
* the code runs now
* support cuda
* using readonly graph
* using C random walk
* removing profile decorators
* param initialization
* no grad
* leaky relu fixes everything
* train and save
* WIP
* WIP
* WIP
* seems to work
* evaluation output
* swapping order of val/test and train
* debug
* hyperparam tuning
* prior/training dataset split changes
* random walk reorg
* random walk with restart
* signed comparison fix
* migrating random walk to nodeflow
* Revert "migrating random walk to nodeflow"
This reverts commit f2565347cced7c912a58a529b257c033d9f375b7.
* add README and remove dataset
* new endpoint
* lint
* lint x2
* oops forgot test
* including bpr - better for baseline
* addressing fixes
* throwing random walks out from SamplerOp class
* forgot to move RandomWalk; why did this even work?
* removing legacy garbage
* add todo
* address comments
* stupid bug fix
* call ndarrayvector converter to handle traces
2019-03-29 13:01:18 +08:00
341dbea0a1
[Bug] Fix unused prefetch switch for NeighborSampler and LayerSampler ( #457 )
...
* forgot prefetch switch
* yet another stupid bug
2019-03-19 14:57:10 +08:00
681e5217dd
[Refactor] Sampler code refactor ( #454 )
...
* refactored sampler code
* docstring
* fix tutorial
2019-03-18 10:41:35 -04:00
ca2a7e1ca1
[Refactor] Nodeflow, sampling, CAPI ( #430 )
...
* enable cython
* add helper function and data structure for void_p vector return
* move sampler from graph index to contrib.sampling
* WIP
* WIP
* refactor layer sampling
* pass tests
* fix lint
* fix graphsage
* remove comments
* pickle test
* fix comments
* update dev guide for cython build
2019-03-05 14:07:22 -05:00
d3c24cc25b
[BugFix] Fix bug in RGCN data processing and use index_select to improve speed ( #429 )
...
* use index_select instead of __getitem__
* fix bug in dataset processing
* fix edge_type shape bug
* comments
2019-03-04 10:52:21 -05:00
fb4246e564
[Feature]Uniform layer-wise sampler ( #416 )
...
* migrate to node-flow
* uniform layer sampler test cases
* more test cases
* documentations
* fix lint errors
* fix lint errors
* fix lint errors
* iota
* add asnumpy
* requested changes
* fix indptr error
* fix lint errors
* requested changes & fix lint errors
* fix lint errors
* fix LayerSampler unit test
2019-03-02 16:37:18 -08:00
a88f351174
[Doc] update doc string ( #426 )
2019-03-02 14:00:52 -05:00
bea07b41b3
[Sampler] fix the API of neighbor sampler. ( #407 )
...
* don't return aux_info.
* fix sampler test.
* fix sse.
* fix.
* add comment.
2019-02-28 13:45:21 -08:00
7e30382e4f
[Model][MXNet] neighbor sampling & skip connection & control variate & graphsage ( #322 )
...
* neighbor sampling draft
* val/test acc
* control variate draft
* control variate
* update
* fix new_history
* maintain aggregated history while updating new history
* preprocess the first layer, change push to pull
* update
* fix subg_degree
* nodeflow
* clear
* readme
* doc and unittest for self loop
* address comments
* rename
* update
* fix
* Update node_flow.py
* Update node_flow.py
2019-02-28 13:29:51 -08:00
2e1cbd5d68
[Feature] Random walk traces generation ( #392 )
...
* random walk traces generation
* remove outdated comments
* oops put in the wrong place
* explicit inline
* moving rand_r to util
* moving random walk to public function
* per-thread seed and openmp support
* type cast styles
2019-02-20 10:41:59 -05:00
f370e628cd
[Feature] add NodeFlow API ( #361 )
...
* sample layer subgraphs.
* fix.
* fix.
* add layered subgraph.
* fix lint.
* fix.
* fix tutorial.
* fix.
* remove copy_to_parent.
* add num_layers
* move sampling code to sampler.cc
* fix.
* move subgraph construction out.
* Revert "move subgraph construction out."
This reverts commit 24b3d13b0d8ed5f91847ea75a7674ee8f7d45cff.
* change to NodeFlow.
* use NodeFlow in Python.
* use NodeFlowIndex.
* add node_mapping and edge_mapping.
* remove unnecessary code in SSE tutorial.
* Revert "remove unnecessary code in SSE tutorial."
This reverts commit 093f0413d5fa2e63ca5f80c46c80a126a9fb720c.
* fix tutorial.
* move to node_flow.
* update gcn cv updater.
* import NodeFlow.
* update.
* add demo code for vanilla control variate sampler.
* update.
* update.
* add neighbor sampling.
* return flow offsets.
* update node_flow.
* add test.
* fix sampler.
* fix graph index.
* fix a bug in sampler.
* fix map_to_layer_nid and map_to_flow_eid.
* fix apply_flow.
* remove model code.
* implement flow_compute.
* fix a bug.
* reverse the csr physically.
* add mini-batch test.
* add mini batch test.
* update flow_compute.
* add prop_flows
* run on specific nodes.
* test copy
* fix a bug in creating frame in NodeFlow.
* add init gcn_cv_updater.
* fix a minor bug.
* fix gcn_cv_updater.
* fix a bug.
* fix a bug in NodeFlow.
* use new h in gcn_cv_updater.
* add layer_in_degree and layer_out_degree.
* fix gcn_cv_updater for gpu.
* temp fix in NodeFlow for diff context.
* allow enabling/disabling copy back.
* add with-updater option.
* fix a bug in computing degree.
* add with-cv option.
* rename and add comments.
* fix lint complain.
* fix lint.
* avoid assert.
* remove assert.
* fix.
* fix.
* fix.
* fix.
* fix the methods in NodeFlow.
* fix lint.
* update SSE.
* remove gcn_cv_updater.
* correct comments for the schedulers.
* update comment.
* add map_to_nodeflow_nid
* address comment.
* remove duplicated test.
* fix int.
* fix comments.
* fix lint
* fix.
* replace subgraph with NodeFlow.
* move view.
* address comments.
* fix lint.
* fix lint.
* remove static_cast.
* fix docstring.
* fix comments.
* break SampleSubgraph.
* move neighbor sampling to sampler.cc
* fix comments.
* rename.
* split neighbor_list.
* address comments.
* fix.
* remove TODO.
2019-02-19 15:49:28 -08:00
929742b588
[Feature] Reimplement Immutable graph index in DGL ( #342 )
...
* init.
* it's compiled.
* add immutable graph constructor.
* add immutable graph API.
* fix.
* impl get adjacency matrix.
* fix.
* fix graph_index from scipy matrix.
* add neighbor sampling.
* remap vertex ids.
* fix.
* move sampler test.
* fix tests.
* add comments
* remove mxnet-specific immutable graph.
* fix.
* fix lint.
* fix.
* try to fix windows compile error.
* fix.
* fix.
* add test.
* unify Graph and ImmutableGraph.
* fix bugs.
* fix compile.
* move immutable graph.
* fix.
* remove print.
* fix lint.
* fix
* fix lint.
* fix lint.
* fix test.
* fix comments.
* merge GraphIndex and ImmutableGraphIndex.
* temp fix.
* impl GetAdj.
* fix lint
* fix.
* fix.
* fix.
* fix.
* fix.
* use csr only for readonly graph.
* Revert "use csr only for readonly graph."
This reverts commit 8e24bb033af8504531b22849de5b7567b168e0d5.
* remove code.
* fix.
* fix.
* fix.
* fix.
* fix.
* fix.
* address comments.
* fix for comments.
* fix comments.
* revert.
* move test_graph_index to compute.
* fix.
* fix.
* impl GetAdj for coo.
* fix.
* fix tests.
* address comments.
* address comments.
* fix comment.
* address comments.
* use lambda.
* other comments.
* address comments.
* modify the semantics of edges.
* fix order.
* use DGLIdIter
* fix.
* remove NotImplemented.
* revert some code.
2019-01-17 09:59:53 +08:00
707334ce85
minor spelling tweaks ( #349 )
...
* minor spelling tweaks
* Update CONTRIBUTORS.md
2019-01-10 12:02:31 -05:00
30107407fd
add prefetcher for neighbor sampler ( #298 )
2018-12-15 19:14:17 -08:00
0cbe521b99
[Doc] Fix document building for multiple tensor backends ( #301 )
...
* remove loading backend in sse
* build mxnet in a second run
* a third run to fix everything
* fix docstring in sampler
* fix docstring in transformer
* README for doc build
* some reordering and printing
* use filename_pattern instead of ignore_pattern to reduce warning at the cost of redundant work...
* add env var to use mxnet in Jenkinsfile
* remove step 3, it's already fixed by previous commit
2018-12-12 11:07:46 -05:00
18ed7868fe
[Doc] update neighbor sampler doc. ( #287 )
2018-12-11 12:48:22 -05:00
cd907cddfa
[Tutorial] Add sse tutorial & MXNet Tutorial CI ( #252 )
...
* add sse tutorial
* add mxnet tutorial ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* Fix ci
* Fix ci
* Fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* Fix CI
Fix CI image
* permission fix
* fix a bug in the code.
* small fix
* fix doc
* fix ci
* shorten the iters
* fix
* remove extra file
* add load_backend api to dynamically switch to another backend
* try fix
* fix tutorial
* fix tutorial
* fix bug in tutorial
2018-12-05 16:21:44 -05:00
7c7cc7e0c2
[sampler] Adjust the sampler API for the future extension. ( #243 )
...
* return seed ids.
* fix tests.
* implement.
2018-12-05 10:05:10 -08:00
274e1082f5
[Doc] API doc string for many components ( #245 )
...
* builtin function docstring
* init docstring
* view docstring
* add init to index and fix
* node /edge feature
* Example -> Examples
* fix func hyperlink
* traversal
* fix
* from/to_networkx docstring
* minor
* fix duplicate label
* comment about tensor framework
* from scipy sparse matrix
* fix format errors
2018-12-04 17:52:00 -05:00
a5a35d1f7a
[Sampler] improve random shuffle performance in sampler ( #228 )
...
* fix.
* make it generic.
* add the API.
* fix.
* remove mxnet.
2018-12-04 09:24:32 -08:00
5680f8dc13
[DOC] Documentation strings for graph mutation, queries, and subgraphing ( #207 )
...
* docstrings for graph mutation & queries
* [Doc] contrib.sampler. (#200 )
* subgraph docs
* updates
2018-12-02 14:41:25 -05:00
Da Zheng