How to use the lineflow.core.IterableDataset function in lineflow

To help you get started, we’ve selected a few lineflow examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github tofunlp / lineflow / tests / test_core.py View on Github external
def setUp(self):
        self.base = range(100)
        self.data = IterableDataset(iter(self.base))
github tofunlp / lineflow / lineflow / core.py View on Github external
def filter(self, predicate: Callable[[Any], bool]) -> 'IterableDataset':
        """Filters this dataset by a predicate function.

        Args:
            predicate (Callable[[Any], bool]): A predicate function.

        Returns ('IterableDataset'):
            The dataset containing the examples for which ``predicate`` returns ``True``.
        """
        return IterableDataset(lineflow_filter(predicate, self, lazy=True))
github tofunlp / lineflow / lineflow / core.py View on Github external
def __len__(self) -> int:
        self._prepare()
        return super(IterableDataset, self).__len__()
github tofunlp / lineflow / lineflow / core.py View on Github external
def window(self, window_size: int, shift: int = None) -> 'IterableDataset':
        """Combines input examples into a dataset of windows.

        Args:
            window_size (int): the number of examples of the input dataset to combine into a window.
            shift (int, optional): The forward shift of the sliding window in each iteration.

        Returns ('IterableDataset'):
            The dataset of windows.
        """
        return IterableDataset(lineflow_window(self, window_size, shift, lazy=True))
github tofunlp / lineflow / lineflow / core.py View on Github external
def flat_map(self, map_func: Callable[[Any], Any]) -> 'IterableDataset':
        """Applies a function across the examples of this dataset and then flattens the result.

        Args:
            map_func (Callable[[Any], Any]): A function to apply.

        Returns ('IterableDataset'):
            The dataset applied the function and flattened.
        """
        return IterableDataset(lineflow_flat_map(map_func, self, lazy=True))