How to use the fsspec.utils.infer_storage_options function in fsspec

To help you get started, we’ve selected a few fsspec examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github intake / filesystem_spec / fsspec / implementations / hdfs.py View on Github external
def _get_kwargs_from_urls(path):
        ops = infer_storage_options(path)
        out = {}
        if ops.get("host", None):
            out["host"] = ops["host"]
        if ops.get("username", None):
            out["user"] = ops["username"]
        if ops.get("port", None):
            out["port"] = ops["port"]
        return out
github intake / intake / intake / source / cache.py View on Github external
def sanitize_path(path):
    """Utility for cleaning up paths."""

    storage_option = infer_storage_options(path)

    protocol = storage_option['protocol']
    if protocol in ('http', 'https'):
        # Most FSs remove the protocol but not HTTPFS. We need to strip
        # it to match properly.
        path = os.path.normpath(path.replace("{}://".format(protocol), ''))
    elif protocol == 'file':
        # Remove trailing slashes from file paths.
        path = os.path.normpath(path)
        # Remove colons
        path = path.replace(':', '')
    # Otherwise we just make sure that path is posix
    return make_path_posix(path)
github intake / filesystem_spec / fsspec / implementations / hdfs.py View on Github external
def _strip_protocol(cls, path):
        ops = infer_storage_options(path)
        return ops["path"]
github intake / filesystem_spec / fsspec / implementations / webhdfs.py View on Github external
def _strip_protocol(cls, path):
        return infer_storage_options(path)["path"]
github intake / filesystem_spec / fsspec / implementations / ftp.py View on Github external
def _get_kwargs_from_urls(urlpath):
        out = infer_storage_options(urlpath)
        out.pop("path", None)
        out.pop("protocol", None)
        return out
github intake / filesystem_spec / fsspec / implementations / ftp.py View on Github external
def _strip_protocol(cls, path):
        return "/" + infer_storage_options(path)["path"].lstrip("/").rstrip("/")
github intake / filesystem_spec / fsspec / implementations / sftp.py View on Github external
def _strip_protocol(cls, path):
        return infer_storage_options(path)["path"]