merge

Author: JiayiFeng

doc/design/cpp_data_feeding.md

@@ -1,6 +1,6 @@
 # C++ Data Feeding
 
-While using Paddle V2 API for Training, data feeding completely depends on the Python code. To get rid of the Python environment and achieve the goal of "wrapping the whole training by a while loop op" in Paddle Fluid, a C++ data feeding mechanism is required. 
+While using Paddle V2 API for training, data feeding completely depends on the Python code. To get rid of the Python environment and achieve the goal of "wrapping the whole training by a while loop op" in Paddle Fluid, a C++ data feeding mechanism is required.
 
 In this document we show the fundamental design of a C++ data feeding process, which includes data reading, shuffling and batching.
 
@@ -16,35 +16,67 @@ In order to handle the above mentioned problem, a new concept called 'Reader' is
 ```cpp
 class ReaderBase {
  public:
-  explicit ReaderBase(const std::vector<DDim>& shapes) : shapes_(shapes) {
-    PADDLE_ENFORCE(!shapes_.empty());
-  }
-  // Read the next batch of data. (A 'batch' can be only one instance)
-  // If the next batch doesn't exist, '*out' will be an empty std::vector.
+  // Reads the next batch of data. (A 'batch' can be only one instance)
+  // If the next batch doesn't exist, it throws an exception
   virtual void ReadNext(std::vector<LoDTensor>* out) = 0;
 
-  // Reinitialize the reader and read the file from the beginning.
-  virtual void ReInit() = 0;
+  // Checks whether the next instance exists.
+  virtual bool HasNext() = 0;
 
-  // Get a certain read in data's shape.
-  DDim shape(size_t idx) const;
-  // Get shapes of all read in data.
-  std::vector<DDim> shapes() const { return shapes_; }
-  // Set shapes of read in data.
-  void set_shapes(const std::vector<DDim>& shapes) { shapes_ = shapes; }
+  // Reinitializes the reader and read the file from the beginning.
+  virtual void ReInit() = 0;
 
   virtual ~ReaderBase() {}
+};
+```
+
+### FileReader
+
+`FileReader` is derived from the `ReaderBase`. It is still an abstract class and will further be derived by Readers of respective specific format.
+
+```cpp
+class FileReader : public ReaderBase {
+ public:
+  explicit FileReader(const std::vector<DDim>& shapes) : shapes_(shapes) {}
+
+  void ReadNext(std::vector<LoDTensor>* out) override final {
+    ReadNextImpl(out);
+    CheckShapes(out);
+  }
+
+  virtual void ReadNextImpl(std::vector<LoDTensor>* out) = 0;
 
  protected:
+  // Checks whether the out shapes is consistent with shapes_
+  CheckShape(const std::vector<LoDTensor>* out);
+
   std::vector<DDim> shapes_;
 };
 ```
 
-### `FileReader` and `DecoratedReader`
+A file reader binds with a single file, and reads one instance of data from the file at a time. Each type of file reader shall implement its own `ReadNextImpl()`, `HasNext()` and `ReInit()`.
+
+### DecoratedReader
+
+A decorated reader takes another reader(both file reader and decorated reader are OK) as its 'underlying reader'. It gets data from its underlying reader, does some process on them(shuffling,  batching or something else), then yields processed data. The output data of a decorated reader can be a single instance or a batch. `ShuffleReader` and `BatchReader` are both decorated readers.
+
+```cpp
+class DecoratedReader : public ReaderBase {
+ public:
+  explicit DecoratedReader(ReaderBase* reader) : reader_(reader) {
+    PADDLE_ENFORCE_NOT_NULL(reader_);
+  }
+
+  void ReInit() override { reader_->ReInit(); }
+
+ protected:
+  ReaderBase* reader_;
+};
+```
 
-These two classes are derived from the `ReaderBase` and will further be derived by more specific readers. Thus, in our design, there are two kinds of readers: file readers and decorated readers. A file reader reads from a file of some specific format, and yield only one instance of data at a time. For example, RecordIO reader, jpg reader, .... A decorated reader takes another reader(both file reader and decorated reader are OK) as its 'underlying reader'. It gets data from its underlying reader, does some processing on them(shuffling, or batching), then yields processed data. The output data of a decorated reader can be a single instance or a batch. `ShuffleReader` and `BatchReader` are both decorated readers.
+All the `FileReader` and `DecoratedReader` share exactly the same interfaces as defined in `ReaderBase`. So they can be decorated for more than one time: We can **shuffle** a reader's outputs and then **batch** the shuffle outputs. The interface consistency also allows related ops use readers without knowing what they are exactly.
 
-All the readers share exactly the same interface as defined in `ReaderBase`. So they can be decorated for more than one time: We can **shuffle** a reader's outputs and then **batch** the shuffle outputs. The interface consistency also allows related ops use readers without knowing what they are exactly.
+### ThreadedReader
 
 
 ### `ReaderHolder`

paddle/fluid/framework/reader.h

@@ -43,13 +43,24 @@ class ReaderBase {
 
 class FileReader : public ReaderBase {
  public:
-  explicit FileReader(const std::vector<DDim>& shapes) : ReaderBase(shapes) {}
+  explicit FileReader(const std::vector<DDim>& shapes) : shapes_(shapes) {}
+
+  void ReadNext(std::vector<LoDTensor>* out) override final {
+    ReadNextImpl(out);
+    CheckShapes(out);
+  }
+
+  virtual void ReadNextImpl(std::vector<LoDTensor>* out) = 0;
+
+ protected:
+  CheckShape(const std::vector<LoDTensor>* out);
+
+  std::vector<DDim> shapes_;
 };
 
 class DecoratedReader : public ReaderBase {
  public:
-  explicit DecoratedReader(ReaderBase* reader)
-      : ReaderBase(reader->shapes()), reader_(reader) {
+  explicit DecoratedReader(ReaderBase* reader) : reader_(reader) {
     PADDLE_ENFORCE_NOT_NULL(reader_);
   }