Add DataReader for parallel training with one DB session
authorCyprien Noel <cyprien.noel@gmail.com>
Tue, 19 May 2015 01:06:09 +0000 (18:06 -0700)
committerEvan Shelhamer <shelhamer@imaginarynumber.net>
Sun, 9 Aug 2015 22:13:11 +0000 (15:13 -0700)
commitbcc8f50a95ecad954d1887f3fb273eaa298e2274
treed42bb8d582c926532ac4051620c272d8f0ce70f1
parentddcdc9d711e81312caf127e8aa512c3298101297
Add DataReader for parallel training with one DB session

- Make sure each solver accesses a different subset of the data
- Sequential reading of DB for performance
- Prefetch a configurable amount of data to host memory
- Distribute data to solvers in round-robin way for determinism
include/caffe/data_layers.hpp
include/caffe/data_reader.hpp [new file with mode: 0644]
src/caffe/data_reader.cpp [new file with mode: 0644]
src/caffe/layers/base_data_layer.cpp
src/caffe/layers/data_layer.cpp
src/caffe/proto/caffe.proto
src/caffe/test/test_layer_factory.cpp
src/caffe/test/test_upgrade_proto.cpp
src/caffe/util/blocking_queue.cpp