o Me&@s2ddlmZddlmZdgZGdddeZdS)) data_feed_pb2) text_format DataFeedDescc@s8eZdZdZddZddZddZdd Zd d Zd S) rag :api_attr: Static Graph Datafeed descriptor, describing input training data format. This class is currently only used for AsyncExecutor (See comments for class AsyncExecutor for a brief introduction) DataFeedDesc shall be initialized from a valid protobuf message from disk. See :code:`paddle/fluid/framework/data_feed.proto` for message definition. A typical message might look like: .. code-block:: python import paddle.fluid as fluid f = open("data.proto", "w") print >> f, 'name: "MultiSlotDataFeed"' print >> f, 'batch_size: 2' print >> f, 'multi_slot_desc {' print >> f, ' slots {' print >> f, ' name: "words"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, ' slots {' print >> f, ' name: "label"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, '}' f.close() data_feed = fluid.DataFeedDesc('data.proto') However, users usually shouldn't care about the message format; instead, they are encouraged to use :code:`Data Generator` as a tool to generate a valid data description, in the process of converting their raw log files to training files acceptable to AsyncExecutor. DataFeedDesc can also be changed during runtime. Once you got familiar with what each field mean, you can modify it to better suit your need. E.g.: .. code-block:: python import paddle.fluid as fluid data_feed = fluid.DataFeedDesc('data.proto') data_feed.set_batch_size(128) data_feed.set_dense_slots('wd') # The slot named 'wd' will be dense data_feed.set_use_slots('wd') # The slot named 'wd' will be used Finally, the content can be dumped out for debugging purpose: .. code-block:: python print(data_feed.desc()) Args: proto_file(string): Disk file containing a data feed description. cCs|t|_d|j_t|d}t||jWdn1s"wY|jjdkrZsz)DataFeedDesc.__init__..) rr proto_descZ pipe_commandopenrParsereadr enumeratemulti_slot_descslots_DataFeedDesc__name_to_index)selfZ proto_filefrrr __init__Ts     zDataFeedDesc.__init__cCs ||j_dS)a Set :attr:`batch_size` in :ref:`api_fluid_DataFeedDesc` . :attr:`batch_size` can be changed during training. Example: .. code-block:: python import paddle.fluid as fluid f = open("data.proto", "w") print >> f, 'name: "MultiSlotDataFeed"' print >> f, 'batch_size: 2' print >> f, 'multi_slot_desc {' print >> f, ' slots {' print >> f, ' name: "words"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, ' slots {' print >> f, ' name: "label"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, '}' f.close() data_feed = fluid.DataFeedDesc('data.proto') data_feed.set_batch_size(128) Args: batch_size (int): The number of batch size. Returns: None. N)r batch_size)rrrrr set_batch_size_s $zDataFeedDesc.set_batch_sizecC8|jjdkr td|D] }d|jjj|j|_q dS)ay Set slots in :attr:`dense_slots_name` as dense slots. **Note: In default, all slots are sparse slots.** Features for a dense slot will be fed into a Tensor, while those for a sparse slot will be fed into a LoDTensor. Example: .. code-block:: python import paddle.fluid as fluid f = open("data.proto", "w") print >> f, 'name: "MultiSlotDataFeed"' print >> f, 'batch_size: 2' print >> f, 'multi_slot_desc {' print >> f, ' slots {' print >> f, ' name: "words"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, ' slots {' print >> f, ' name: "label"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, '}' f.close() data_feed = fluid.DataFeedDesc('data.proto') data_feed.set_dense_slots(['words']) Args: dense_slots_name (list(str)): a list of slot names which will be set dense. Returns: None. rzNOnly MultiSlotDataFeed needs set_dense_slots, please check your datafeed.protoTN)rr ValueErrorrrrZis_dense)rZdense_slots_namer rrr set_dense_slotss 'zDataFeedDesc.set_dense_slotscCr)a\ Set if a specific slot will be used for training. A dataset shall contain a lot of features, through this function one can select which ones will be used for a specific model. Example: .. code-block:: python import paddle.fluid as fluid f = open("data.proto", "w") print >> f, 'name: "MultiSlotDataFeed"' print >> f, 'batch_size: 2' print >> f, 'multi_slot_desc {' print >> f, ' slots {' print >> f, ' name: "words"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, ' slots {' print >> f, ' name: "label"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, '}' f.close() data_feed = fluid.DataFeedDesc('data.proto') data_feed.set_use_slots(['words']) Args: use_slots_name: a list of slot names which will be used in training Note: Default is not used for all slots rzLOnly MultiSlotDataFeed needs set_use_slots, please check your datafeed.protoTN)rr rrrrZis_used)rZuse_slots_namer rrr set_use_slotss %zDataFeedDesc.set_use_slotscCs t|jS)aU Returns a protobuf message for this DataFeedDesc Example: .. code-block:: python import paddle.fluid as fluid f = open("data.proto", "w") print >> f, 'name: "MultiSlotDataFeed"' print >> f, 'batch_size: 2' print >> f, 'multi_slot_desc {' print >> f, ' slots {' print >> f, ' name: "words"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, ' slots {' print >> f, ' name: "label"' print >> f, ' type: "uint64"' print >> f, ' is_dense: false' print >> f, ' is_used: true' print >> f, ' }' print >> f, '}' f.close() data_feed = fluid.DataFeedDesc('data.proto') print(data_feed.desc()) Returns: A string message )rZMessageToStringr)rrrr descs zDataFeedDesc.descN) __name__ __module__ __qualname____doc__rrrrrrrrr rs> &/ -N)Zpaddle.fluid.protorZgoogle.protobufr__all__objectrrrrr s