Solid-File-Client

A library for managing Solid files and folders.

current version :
previous version :

Table of Contents : Introduction | Installing | Importing, Invoking, Logging In | Error Handling | High Level Methods | Advanced Options | Low-level Methods | ACL management | Solid RDF utilities | Note on Terminology | Acknowledgements

Introduction

Solid-File-Client is a JavaScript library with high-level methods to create, read, and manage files and folders. The methods may be used in browser scripts or in nodejs shell scripts. The library supports both text and binary files and can read and write data from Solid Pods, from local file systems, and from virtual in-memory storage. It can also recursively move and copy entire folder trees between any of those storage locations. For advanced users, there are also a number of options and lower-level methods which allow fine-tuned control over linked resources and other features of Solid.

Note for users of version 0.x of Solid-File-Client

There are a number of changes which are not backward compatible. See Guide for transitioning to v.1 for details and hints for upgrading.

Using alternate storage spaces

Solid-file-client can work with web resources (https://). It can also work with resources stored in a browser's local storage (app://), or a local file system (file://). See the Upload Demo for an example of copying files from a local filesystem to a pod in the browser and Node Upload Demo for the same thing in node. See Solid-Rest for a description of using browser local storage or accessing the local file system from node scripts.

In node, the copyFile() and copyFolder() commands can upload files from a local file system to a remote pod and vice-versa:

To upload

   await fc.copyFolder( "file:///somepath/foo/", "https://somehost/somepath/foo/" )

To download

   await fc.copyFolder( "https://somehost/somepath/foo/", "file:///somepath/foo/", 

Using with front-ends

Several front-ends for Solid-File-Client have been built. In a browser you can use the Solid-IDE GUI or create your own GUI using Solid-File-Widget or any GUI framework such as React or Vue. In node or from the command line, you can use Solid-Shell, an interactive shell, batch-processor, and command-line front-end for Solid-File-Client.

Overview of writing methods

By default, all high-level methods that create, copy, or move files or folders have these behaviors :

• the source always completely overwrites the target
• if the path to the item doesn't pre-exist, it will be created
• linked files (.acl and .meta) are copied/moved along with their resources

For many purposes, these defaults will suffice. However, if you need to, you may change any of them with option flags. There are several options for merging folder trees as well as for using Solid's POST features. See the sections on Overwriting, on Creating Paths, and on Linked Files for more information.

Demo scripts

See the Upload & Copy Demos for working examples.

Installing

If you are writing scripts only for the browser, you may wish to use a CDN code repository rather than using a local version. See here for an example of using a CDN.

If you are writing scripts for node or you want a local version, install using npm

    npm install solid-file-client
    change to the solid-file-client folder
    npm install   // this pulls in dependencies
    npm run build // this creates the executables

Once installed the executables will be found within the solid-file-client folder :

    dist/node/solid-file-client.bundle.js      // for node scripts
    dist/window/solid-file-client.bundle.js   // for browser scripts

You can also clone or fork the github repository if wish.

Importing, invoking, and logging-in

Here is the general process for a script using Solid-File-Client :

• Import the solid-file-client and solid-auth-cli(ent) libraries
• Instantiate an auth object
• Instantiate a file-client object using the auth object
• Use the auth object to login and for session management
• Use the file-client object to read and write files and folders

Here is a short node script illustrating the process.

    const auth = require('solid-auth-cli')
    const FC   = require('solid-file-client')
    const fc   = new FC( auth )
    async function run(){
        let session = await auth.currentSession()
        if (!session) { session = await auth.login() }
        console.log(`Logged in as ${session.webId}.`)
        if( await fc.itemExists( someUrl ) {
            let content = await fc.readFile( someUrl )
            // ... other file methods
            // ... and/or other auth methods
        }
    }
    run()

See Using with Node for details of logging in with node and command line scripts. See Using in a Browser for a detailed example of importing, invoking, and logging in from a browser script.

For more information on auth and session functions see solid-auth-client for the browser and solid-auth-cli for node.

Error Handling

All Solid-File-Client methods should throw an error if they do not succeed. To trap and examine errors, use try/catch:

    fc.readFile(url).then((content) => {
        console.log(content)
    })
    .catch(err => console.error(`Error: ${err}`))

Or, in an async method :

    try {
        let content = await fc.readFile( someUrl )
        console.log(content)
    }
    catch(error) {
        console.log( error )         // A full error response 
        console.log( error.status )  // Just the status code of the error
        console.log( error.message ) // Just the status code and statusText
    }

See Interpreting Error Responses for further options and a description of errors for recursive methods like copyFolder().

High-level Methods

createFile( fileURL, content, contentType, options )

postFile( fileURL, content, contentType, options )

Creates a new file at the specified URL. Content is required, even if only a blank string. ContentType should be something like "text/turtle" or "image/png" and is required.

Default behavior :

• If a file already exists at that URL, it will be overwritten.
• If the file's parent path does not exist, it will be created.

See Overwriting and Creating Paths if you need to change the default behavior.

Note for all: Previous versions of of Solid Server tried to guess the content-type from the file extension but you should not depend on this behavior. Previous versions would sometimes add an extension e.g. if you created a file named "foo" with type "text/turtle", it would be renamed "foo.ttl". This is no longer the case, the file will be named "foo" and will still have the type "text/turtle".

Note for advanced users : This method uses PUT, if you prefer the behavior of POST (for example creating alternate versions of a resource rather than replacing it) use the postFile() method, which takes the same parameters as createFile().

patchFile( fileUrl, patchContent, patchContentType )

Modifies the content of fileUrl adding or removing triples.

fileUrl is parsed with N3.js ('text/turtle' files and others)

patchContentType is either 'text/n3' or 'application/sparl-update'

• 'text/n3' patchContent can use inserts, deletes and where. N3 notation is a turtle content with graph objects.

    @prefix solid: .
    @prefix : <#>.
    @prefix ex: .
    <> solid:patches ;
          solid:deletes { <> a :test. };
          solid:inserts { <#new> ex:temp :321; ex:temp1 :200, :300. }.

if triples <#new> ex:temp1 :250 exists. The solid:inserts clause can be expressed as :

          solid:where { ?a ex:temp1 :250. };
          solid:inserts { ?a ex:temp :321; ex:temp1 :200, :300. }.

• 'application/sparql-update' patchContent can only use sparql INSERT and DELETE. Turtle notation is used for prefix and triples.

    @prefix ex: .
    INSERT { <#new> ex:temp :321; ex:temp1 :200, :300 .}
    DELETE { <> a :test. }

Nota on needed acl auth to patch fileUrl :

. with insert Append or Write,

. and with delete Write.

createFolder( folderURL, options )

Creates a new folder at the specified URL.

Default behavior :

• If a folder already exists at that URL, it will be overwritten and all of its contents lost.
• If the folder's parent path does not exist, it will be created.

See Overwriting and Creating Paths to change the default behavior.

readFile( fileURL, options )

On success, the readFile() method returns the contents of the specified file. The return value will be a string for text files and a blob for binary files such as images and music.

Advanced users : If you want the content as a ReadableStream, or you need to specify an accept header, use the get() or fetch() methods - see Low-level Methods.

readFolder( folderURL, options )

On success, the readFolder() method returns a folder object in this format:

{
     type : "folder",
     name : // folder name (without path),
      url : // full URL of the resource,
 modified : // dcterms:modified date
    mtime : // stat:mtime
     size : // stat:size
   parent : // parentFolder or undef if none
    files : // an array of files in the folder
  folders : // an array of sub-folders in the folder
    links : // an array of links for the folder itself IF links=include specified
}

Each item in the arrays of files and sub-folders will be a file object which is the same as a folder object except it does not have the last two fields (files,folders). The content-type in this case is not guessed, it is read from the folder's triples, i.e. what the server sends.

By default, readFolder() does not list linked resources (.acl and .meta files). To change this behavior, see Linked files.

readHead( folderOrFileURL, options ), head( folderOrFileURL, options )

The readHead() method returns all the headers for an item as a string. It is the equivalent of curl -I.

    console.log( await fc.readHead(url) ) // prints all headers

The head() method returns a standard header response which you may inspect useing the headers.get() method.

    let response = await fc.head( url )
    let contentType = response.headers.get('content-type')

itemExists( fileOrFolderURL )

Returns true if the URL exists and false otherwise.

deleteFile( fileURL, options )

Deletes the specified file and all linked filed (.acl,.meta)

deleteFolder( folderURL )

Recursively deletes a folder and all of its contents including all linked files (.acl, .meta).

moveFile( sourceURL, targetURL, options ), copyFile( sourceURL, targetURL, options )

Copies or moves the specified source to the target.

Defaults :

• If a file already exists at that target, it will be overwritten.
• If the target URL's parent path does not exist, it will be created.
• Linked files (.acl and .meta) will be copied if they exist.

See Advanced Options to modify these default behaviors.

moveFolder(sourceURL,targetURL,options), copyFolder(sourceURL,targetURL,options)

Recursively copies or moves a folder and all of its contents.

Defaults :

• If the target exists, it is replaced by the source.
• If the top target folder's parent path does not exist, it will be created.
• Linked files (.acl and .meta) will be copied/moved if they exist.

These default behaviors may all be modified. For example, you can choose from several ways to merge the source and target folders. See Advanced Options for more details.

createZipArchive(sourceURL, archiveURL, options), extractZipArchive(archiveURL, targetFolderURL, options)

Zip folder/file and Unzip file.

Defaults :

• zip/unzip : Linked files (.acl and .meta) will be ziped/unziped if they exist.

• unzip

  * TargetFolderURL is where the archive file is unzipped
  * if unzipped root file or root folder exist they shall be overwritten
  * .acl are checked for validity (notably for acl:Control by at least one agent type (agent, agentClass, agentGroup))
  

These default behaviors may all be modified. For example, you can choose from several ways to unzip the archiveURL, or check for .acl validity. See Advanced Options for more details.

Advanced Options

Overwriting

By default, methods which create, copy, or move files or folders will overwrite an item of the same name in the target space, replacing it with the source item. This behavior may be modified in several ways

With any of the create/copy/move methods, you can use the itemExists() method to prevent overwriting items.

    if( !(await fc.itemExists(x)) ) {
        await fc.createFolder(x) // only create if it doesn't already exist
    }

With the copyFolder() and moveFolder() methods, you can elect to merge the source and target with preference for the source or preference for the target:

• default - target is replaced by source
• merge=keep_source - target becomes source plus items found only in target
• merge=keep_target - target becomes target plus items found only in source

For example :

    await copyFolder( source, target, {merge:"keep_source"} )

To avoid typos, you may also import the constants for these options:

    const { MERGE } = SolidFileClient
    await copyFolder( source, target, {merge: MERGE.KEEP_SOURCE } )

Creating Paths

When you create a file or folder and the path to that item doesn't already exist, Solid-File-Client will create the path for you if it can. For example, if you ask to create /foo/bar/baz.txt but there is no /foo/ folder and there is no /bar/ folder, Solid-File-Client will create /foo/ and then create /bar/ inside it and then create baz.txt inside that.

If you would rather the program fails if the path you asked for doesn't exist, you may set the "createPath" flag to false.

• createFile(),createFolder(),copyFile(),copyFolder(),moveFile(),moveFolder()

  • default - create intermediary paths if they are missing
  • createPath=false - fail if intermediary paths are missing

• note for copyFolder() and moveFolder(), the createPath option applies only to the top-level target folder, not to folders within the target which are handled by the merge option

For example:

      await createFile( url, {createPath:false} )

Linked Files

One of Solid's unique features is the use of linked files. Currently the two main types of linked files are .acl files which control access to the main file they are linked from and .meta files which describe additional features of the file they are linked from. Solid-file-client, by default treats these linked files as tightly bound to the resource they are referencing. In other words when you delete, move, or copy a file that has linked files, the linked files will also be deleted, moved, or copied. These defaults should be sufficient for most basic usage.

Advanced users can modify how linked files are handled with the withAcl, withMeta, agent, and links option flags shown below.

• copyFile(),copyFolder(),moveFile(),moveFolder()

  • default - .acl and .meta items are copied/moved, acl is modified
  • withAcl=false - .acl items are not copied/moved
  • withMeta=false - .meta items are not copied/moved

Solid servers provide the possible location of linked resources in the headers of all resources. Solid-file-client supports the links=include_possible option to include these possible locations without checking to see if the linked file actually exists. The possible locations tell you where to create the linked file if they don't already exist.

• readFolder()

  • default - linked items are not listed
  • links=include - linked items are listed, when they exist
  • links=include_possible - possible locations of links are listed

For example:

      await copyFile( source, target, {links:"exclude"} )
      await readFolder( url, {links:"include_possible"} )

To avoid typos, you may also import the constants for the link options:

    const { LINKS } = SolidFileClient
    await copyFolder( source, target, {links: LINKS.INCLUDE_POSSIBLE} )

With readFolder()'s links:include and links:include_possible option flags, the links for the folder are a property of the folder object and the links for contained resources are in the file objects. For example:

      let folder = await readFolder( url, {links:"include"} )
      console.log(folder.links.meta || "no .meta for this folder")
      console.log(folder.files[0].links.acl) || "no .acl for this file")

See also, the getItemLinks() method which finds the possible locations of linked resources for an item. See Low-level Methods

Solid-file-client makes a special case for access control (.acl) files. These files may contain absolute links which will no longer work when the file is copied or moved. You should avoid using absolute links in .acl files. If you do use absolute links in your accessTo field, Solid-File-Client will always modify the .acl file to make the accessTo field relative to the new location. If you use absolute links for the agent field, that could change the owner of the copied or moved resource. You may control this aspect of modifying the .acl file through the agent flag as shown below.

• copyFile(),copyFolder(),moveFile(),moveFolder()

  • default - accessTo is made relative, agent, untouched
  • agent=to_source - accessTo is made relative, agent made absolute to source
  • agent=to_target - accessTo is made relative, agent made relative to target

Low-level methods

Solid-File-Client provides a number of low-level methods which either support advanced options or directly reflect the behavior of the Solid server without additional processes as are found in the high-level methods.

See the JSdoc for the API for more details of these methods.

ACL management

ACL management functions have been implemented to support creation of ACL files. (with createFile() or putFile()) It allows to create/edit programmatically the ACL content using an intermediate ACL object :

-let aclObject = await fc.aclUrlParser(url) // returns an acl object from the acl inheritance algorithm for the url.
- let aclObject = await fc.acl.contentParser(url, aclContent) // returns an aclObject from an acl file Content
- let aclContent = await fc.acl.createContent(url, aclObject, options) /// build acl content from an aclObject for url
- let aclObject = await fc.acl.addUserMode(aclObject, userAgent, userMode, userAccess) // add user, mode and access to an aclObject
- let aclObject = await fc.acl.deleteUserMode(aclobject, userAgent, userMode, userAccess) // delete user,mode and/or access from an aclObject

example 1 :

// create aclContent with the inheritance algorithm (url is not the link url)
let content = await fc.aclUrlParser(url)
  .then(agents => fc.acl.createContent(url, agents))

example 2 :

// create a block rule
let aclUsers = await fc.acl.addUserMode({}, [{ agentClass: 'Agent' }], ['Read'])
// add an other rule in the block rule
aclUsers = await fc.acl.addUserMode(aclUsers, [{ agent: 'https://example.solidcommunity.net/profile/card#me' }], ['Read', 'Write', 'Control'], ['accessTo'])

// build the aclContent
const aclBloks = [aclUsers] // array of block rules
const aclContent = await fc.acl.createContent('https://example.solidcommunity.net/public/text.txt', aclBloks)
console.log('build an aclContent ' + aclContent)


@prefix : <#>.
@prefix n0: .
@prefix n1: .
@prefix target: .

:Read
    a n0:Authorization;
    n0:accessTo target:;
    n0:agentClass n1:Agent;
    n0:mode n0:Read.

:ReadWriteControl
    a n0:Authorization;
    n0:accessTo target:;
    n0:agent ;
    n0:mode n0:Read, n0:Write, n0:Control.

to create an acl resource for a resource url :

    const { acl: aclUrl } = await fc.getItemLinks(url, { links: 'include_possible'})
    return await fc.putFile(aclUrl, aclContent, 'text/turtle')

Nota :

const aclModes = ['Read', 'Append', 'Write', 'Control']

const aclPredicates = ['agent', 'agentClass', 'agentGroup', 'origin']

const aclAccess = ['accessTo', 'default']

• userAgent : an array of objects (predicate: object). example: [{ agent: 'https://example.com/profile/card#me'}, { agentClass: 'Agent' }, { origin: 'https://solidcommunity.net' }, { default: '' }]

• userMode : an array of acl modes. example: ['Read', 'Write']. To be applied to each element of userAgent

• userAccess : an array of acl access. example : ['accessTo', 'default']

  Default is only used in the container acl. Only needed if a subset of rules are inherited.

Remark : among other functions, are available

• an ACL check function : await acl.isValidAcl (itemUrl, aclContent, options)
• and an ACL make relative : acl.makeContentRelative (aclcontent, itemUrl, toName, options)

See the JSdoc for the aclParser for more details on these methods.

Solid RDF utilities

A minimal class to query, edit and write rdf files content in/from N3 store using solid-namespace.

query(url, s, p, o, g)

• loads a Turtle file, parses it, returns an array of quads
• expects URL of a source file, if empty, uses previously loaded file
• expects Turtle strings or null for subject, predicate, object, & optional graph
• supports this non-standard syntax for Turtle strings -
  • {somePrefix:someTerm} somePrefix is then replaced using URLs from solid-namespace the special prefix thisDoc {thisDoc:me} uses current doc as namespace
  • N3 quad subject, predicate, object and optional graph

Examples :

write store relative to baseIRI : const content = fc.$rdf.write(url, { baseIRI: url })

get quads from ttlContent : const quads = fc.$rdf.queryTurtle(url, ttlContent, null, { acl: 'mode'}, { acl: 'Control' })

See the JSdoc for the rdf-query for more details on these methods.

Note on terminology

Solid servers can store data directly in a physical file system, or use a database or other storage. Instead of talking about "Files" and "Folders", it is more correct to talk about "Containers" and "Resources" - logical terms independent of the storage mechanism. In this documentation, for simplicity, we've used the file/folder terminology with no implication that it represents a physical file system.

Acknowledgements

This library was originally authored by Jeff Zucker. Version 1.0.0 includes many additions and improvements that were the results of a collaboration between Jeff, Alain Bourgeois, and Otto AA.

Many thanks for patches and issues from https://github.com/linonetwo, and https://github.com/scenaristeur.

copyright (c) 2018 Jeff Zucker may be freely used with MIT license