o Ner4@sddlmZddlmZddlZddlZddlmZmZm Z ddl m Z m Z ddgZ Gd ddeZGd d d eZGd ddeZdS) )print_function)divisionN)SamplerSequenceSampler RandomSampler)DatasetIterableDataset BatchSamplerDistributedBatchSamplerc@s4eZdZdZ     d ddZddZd d ZdS) r a A base implement of batch sampler used by `paddle.io.DataLoader` which yield mini-batch indices(a list/tuple with length as mini-batch size and holds sample indices) iterably. Batch sampler used by :code:`paddle.io.DataLoader` should be a subclass of :code:`paddle.io.BatchSampler`, BatchSampler subclasses should implement following methods: :code:`__iter__`: return mini-batch indices iterably. :code:`__len__`: get mini-batch number in an epoch. Args: dataset(Dataset): this could be a :code:`paddle.io.Dataset` implement or other python object which implemented :code:`__len__` for BatchSampler to get indices as the range of :attr:`dataset` length. Default None. sampler (Sampler): this could be a :code:`paddle.io.Dataset` instance which implemented :code:`__iter__` to yield sample indices. :attr:`sampler` and :attr:`dataset` can not be set in the same time. If :attr:`sampler` is set, :attr:`shuffle` should not be set. Default None. shuffle(bool): whether to shuffle indices order before genrating batch indices. Default False. batch_size(int): sample indice number in a mini-batch indices. drop_last(bool): whether drop the last incomplete batch dataset size is not divisible by the batch size. Default False Returns: BatchSampler: an iterable object for indices iterating Examples: .. code-block:: python from paddle.io import RandomSampler, BatchSampler, Dataset # init with dataset class RandomDataset(Dataset): def __init__(self, num_samples): self.num_samples = num_samples def __getitem__(self, idx): image = np.random.random([784]).astype('float32') label = np.random.randint(0, 9, (1, )).astype('int64') return image, label def __len__(self): return self.num_samples bs = BatchSampler(dataset=RandomDataset(100), shuffle=False, batch_size=16, drop_last=False) for batch_indices in bs: print(batch_indices) # init with sampler sampler = RandomSampler(RandomDataset(100)) bs = BatchSampler(sampler=sampler, batch_size=8, drop_last=True) for batch_indices in bs: print(batch_indices) see `paddle.io.DataLoader` NFrcCs|dur$|dus Jdt|tsJdt||r Jd||_n,t|tr-Jd|dus5Jdt|tsCJdt||rKt||_nt||_t|t rY|dks`Jd|||_ t|tsqJd t|||_ dS) Nz'either dataset or sampler should be setz1sampler should be a paddle.io.Sampler, but got {}z+shuffle should be False when sampler is setz1dataset should not be a paddle.io.IterableDatasetz'should not set both dataset and samplerz-shuffle should be a boolean value, but got {}rz3batch_size should be a positive integer, but got {}z/drop_last should be a boolean value, but got {}) isinstancerformattypesamplerr boolrrint batch_size drop_last)selfdatasetrshufflerrrUD:\Projects\ConvertPro\env\Lib\site-packages\paddle/fluid/dataloader/batch_sampler.py__init__fs:             zBatchSampler.__init__ccsXg}|jD]}||t||jkr|Vg}q|js(t|dkr*|VdSdSdS)Nr)rappendlenrr)r batch_indicesidxrrr__iter__s   zBatchSampler.__iter__cCs.t|j}|t|j |jd7}||jSNr)rrrrrr num_samplesrrr__len__s  zBatchSampler.__len__)NNFrF)__name__ __module__ __qualname____doc__rrr"rrrrr sK  c@seZdZdddZddZdS)_InfiniteIterableSamplerrcCs"t|ts Jd||_||_dS)Nz:dataset should be an instance of paddle.io.IterableDataset)r r rr)rrrrrrrs z!_InfiniteIterableSampler.__init__ccs dg|jVq)N)rrrrrrsz!_InfiniteIterableSampler.__iter__N)r)r#r$r%rrrrrrr's  r'c@s:eZdZdZ    d ddZddZdd Zd d ZdS) r aSampler that restricts data loading to a subset of the dataset. In such case, each process can pass a DistributedBatchSampler instance as a DataLoader sampler, and load a subset of the original dataset that is exclusive to it. .. note:: Dataset is assumed to be of constant size. Args: dataset(paddle.io.Dataset): this could be a `paddle.io.Dataset` implement or other python object which implemented `__len__` for BatchSampler to get sample number of data source. batch_size(int): sample indice number in a mini-batch indices. num_replicas(int, optional): porcess number in distributed training. If :attr:`num_replicas` is None, :attr:`num_replicas` will be retrieved from :code:`paddle.distributed.ParallenEnv`. Default None. rank(int, optional): the rank of the current process among :attr:`num_replicas` processes. If :attr:`rank` is None, :attr:`rank` is retrieved from :code:`paddle.distributed.ParallenEnv`. Default None. shuffle(bool): whther to shuffle indices order before genrating batch indices. Default False. drop_last(bool): whether drop the last incomplete batch dataset size is not divisible by the batch size. Default False Examples: .. code-block:: python import numpy as np from paddle.io import Dataset, DistributedBatchSampler # init with dataset class RandomDataset(Dataset): def __init__(self, num_samples): self.num_samples = num_samples def __getitem__(self, idx): image = np.random.random([784]).astype('float32') label = np.random.randint(0, 9, (1, )).astype('int64') return image, label def __len__(self): return self.num_samples dataset = RandomDataset(100) sampler = DistributedBatchSampler(dataset, batch_size=64) for data in sampler: # do something break NFcCs||_t|tr |dksJd||_t|tsJd||_t|ts(Jdddlm}|durCt|tr;|dks?Jd||_n|j|_|dur]t|trU|dksYJd||_ n|j |_ ||_ d|_ tt t|jd|j|_|j|j|_dS) Nrz'batch_size should be a positive integerz!shuffle should be a boolean valuez$drop_last should be a boolean number) ParallelEnvz)num_replicas should be a positive integerz%rank should be a non-negative integerg?)rr rrrrZpaddle.fluid.dygraph.parallelr)nranks local_rankrepochmathceilrr! total_size)rrrZ num_replicasZrankrrr)rrrrs:      z DistributedBatchSampler.__init__c#stj}t|}||djt|7}t|jks#Jjr7tjj |j d7_ fdd}j dkrF||}t|j ksOJt |}g}|D]}| |t|jkrj|Vg}qWjsyt|dkr{|VdSdSdS)Nrcsg}jjj}|jdksJ|j}tjjt||jjD]}||||jq+|t||d}||j|jd||S)Nrr)r/rr*ranger+rextend)indicesZsubsampled_indicesZlast_batch_sizeZlast_local_batch_sizeir(rr_get_indices_by_batch_size s(    zDDistributedBatchSampler.__iter__.._get_indices_by_batch_sizer)rrnpZarangetolistr/rrandomZ RandomStater,r*r!iterrrr)rr!r2r4Z _sample_iterrrrr(rrs.     z DistributedBatchSampler.__iter__cCs*|j}|t|j |jd7}||jSr)r!rrrr rrrr".s zDistributedBatchSampler.__len__cCs ||_dS)aL Sets the epoch number. When :attr:`shuffle=True`, this number is used as seeds of random numbers. By default, users may not set this, all replicas (workers) use a different random ordering for each epoch. If set same number at each epoch, this sampler will yield the same ordering at all epoches. Arguments: epoch (int): Epoch number. Examples: .. code-block:: python import numpy as np from paddle.io import Dataset, DistributedBatchSampler # init with dataset class RandomDataset(Dataset): def __init__(self, num_samples): self.num_samples = num_samples def __getitem__(self, idx): image = np.random.random([784]).astype('float32') label = np.random.randint(0, 9, (1, )).astype('int64') return image, label def __len__(self): return self.num_samples dataset = RandomDataset(100) sampler = DistributedBatchSampler(dataset, batch_size=64) for epoch in range(10): sampler.set_epoch(epoch) N)r,)rr,rrr set_epoch3s %z!DistributedBatchSampler.set_epoch)NNFF)r#r$r%r&rrr"r9rrrrr s: '+ ) __future__rrnumpyr5r-rrrrrrr __all__r objectr'r rrrrs  {