Skip to content

Commit

Permalink
windows: only fall back to move-remove when absolutely necessary
Browse files Browse the repository at this point in the history
  • Loading branch information
isaacs committed Jan 10, 2023
1 parent b6f7819 commit 40f64ec
Show file tree
Hide file tree
Showing 17 changed files with 1,058 additions and 172 deletions.
71 changes: 50 additions & 21 deletions README.md
Expand Up @@ -8,8 +8,9 @@ Install with `npm install rimraf`, or just drop rimraf.js somewhere.
* Built-in glob support removed.
* Functions take arrays of paths, as well as a single path.
* Native implementation used by default when available.
* New implementation on Windows, making the exponential backoff for
`EBUSY` and `ENOTEMPTY` errors no longer necessary.
* New implementation on Windows, falling back to "move then
remove" strategy when exponential backoff for `EBUSY` fails to
resolve the situation.
* Simplified implementation on Posix, since the Windows affordances are not
necessary there.

Expand All @@ -21,13 +22,24 @@ This first parameter is a path. The second argument is an options object.

Options:

* `tmp`: Temp folder to use to place files and folders for the Windows
implementation. Must be on the same physical device as the path being
deleted. Defaults to `os.tmpdir()` when that is on the same drive letter
as the path being deleted, or `${drive}:\temp` if present, or `${drive}:\`
* `preserveRoot`: If set to boolean `false`, then allow the
recursive removal of the root directory. Otherwise, this is
not allowed.
* `tmp`: Windows only. Temp folder to use to place files and
folders for the "move then remove" fallback. Must be on the
same physical device as the path being deleted. Defaults to
`os.tmpdir()` when that is on the same drive letter as the path
being deleted, or `${drive}:\temp` if present, or `${drive}:\`
if not.
* `preserveRoot`: If set to boolean `false`, then allow the recursive
removal of the root directory. Otherwise, this is not allowed.
* `retries`: Windows only. Maximum number of synchronous retry
attempts in case of `EBUSY`, `EMFILE`, and `ENFILE` errors.
Default `10`
* `backoff`: Windows only. Rate of exponential backoff for async
removal in case of `EBUSY`, `EMFILE`, and `ENFILE` errors.
Should be a number greater than 1. Default `1.2`
* `maxBackoff`: Windows only. Maximum backoff time in ms to
attempt asynchronous retries in case of `EBUSY`, `EMFILE`, and
`ENFILE` errors. Default `100`

Any other options are provided to the native Node.js `fs.rm` implementation
when that is used.
Expand Down Expand Up @@ -64,25 +76,42 @@ Synchronous form of `rimraf.manual()`
### `rimraf.windows(f, [opts])`

JavaScript implementation of file removal appropriate for Windows
platforms, where `unlink` and `rmdir` are not atomic operations.
platforms. Works around `unlink` and `rmdir` not being atomic
operations, and `EPERM` when deleting files with certain
permission modes.

Moves all files and folders to the parent directory of `f` with a temporary
filename prior to attempting to remove them.

Note that, in cases where the operation fails, this _may_ leave files lying
around in the parent directory with names like
`.file-basename.txt.0.123412341`. Until the Windows kernel provides a way
to perform atomic `unlink` and `rmdir` operations, this is unfortunately
unavoidable.

To move files to a different temporary directory other than the parent,
provide `opts.tmp`. Note that this _must_ be on the same physical device
as the folder being deleted, or else the operation will fail.
First deletes all non-directory files within the tree, and then
removes all directories, which should ideally be empty by that
time. When an `ENOTEMPTY` is raised in the second pass, falls
back to the `rimraf.moveRemove` strategy as needed.

### `rimraf.windows.sync(path, [opts])` `rimraf.windowsSync(path, [opts])`

Synchronous form of `rimraf.windows()`

### `rimraf.moveRemove(path, [opts])`

Moves all files and folders to the parent directory of `path`
with a temporary filename prior to attempting to remove them.

Note that, in cases where the operation fails, this _may_ leave
files lying around in the parent directory with names like
`.file-basename.txt.0.123412341`. Until the Windows kernel
provides a way to perform atomic `unlink` and `rmdir` operations,
this is unfortunately unavoidable.

To move files to a different temporary directory other than the
parent, provide `opts.tmp`. Note that this _must_ be on the same
physical device as the folder being deleted, or else the
operation will fail.

This is the slowest strategy, but most reliable on Windows
platforms. Used as a last-ditch fallback by `rimraf.windows()`.

### `rimraf.moveRemove.sync(path, [opts])` `rimraf.moveRemoveSync(path, [opts])`

Synchronous form of `rimraf.moveRemove()`

### Command Line Interface

```
Expand Down
6 changes: 4 additions & 2 deletions lib/bin.js
Expand Up @@ -34,8 +34,9 @@ Implementation-specific options:
const { resolve, parse } = require('path')

const main = async (...args) => {
if (process.env.__RIMRAF_TESTING_BIN_FAIL__ === '1')
if (process.env.__RIMRAF_TESTING_BIN_FAIL__ === '1') {
throw new Error('simulated rimraf failure')
}

const opt = {}
const paths = []
Expand Down Expand Up @@ -92,8 +93,9 @@ const main = async (...args) => {
console.error(`unknown option: ${arg}`)
runHelpForUsage()
return 1
} else
} else {
paths.push(arg)
}
}

if (opt.preserveRoot !== false) {
Expand Down
12 changes: 8 additions & 4 deletions lib/default-tmp.js
Expand Up @@ -27,12 +27,14 @@ const win32DefaultTmp = async path => {
const { root } = parse(path)
const tmp = tmpdir()
const { root: tmpRoot } = parse(tmp)
if (root.toLowerCase() === tmpRoot.toLowerCase())
if (root.toLowerCase() === tmpRoot.toLowerCase()) {
return tmp
}

const driveTmp = resolve(root, '/temp')
if (await isDir(driveTmp))
if (await isDir(driveTmp)) {
return driveTmp
}

return root
}
Expand All @@ -41,12 +43,14 @@ const win32DefaultTmpSync = path => {
const { root } = parse(path)
const tmp = tmpdir()
const { root: tmpRoot } = parse(tmp)
if (root.toLowerCase() === tmpRoot.toLowerCase())
if (root.toLowerCase() === tmpRoot.toLowerCase()) {
return tmp
}

const driveTmp = resolve(root, '/temp')
if (isDirSync(driveTmp))
if (isDirSync(driveTmp)) {
return driveTmp
}

return root
}
Expand Down
53 changes: 53 additions & 0 deletions lib/fix-eperm.js
@@ -0,0 +1,53 @@
const {
promises: { chmod },
chmodSync,
} = require('./fs.js')

const fixEPERM = fn => async path => {
try {
return await fn(path)
} catch (er) {
if (er.code === 'ENOENT') {
return
}
if (er.code === 'EPERM') {
try {
await chmod(path, 0o666)
} catch (er2) {
if (er2.code === 'ENOENT') {
return
}
throw er
}
return await fn(path)
}
throw er
}
}

const fixEPERMSync = fn => path => {
try {
return fn(path)
} catch (er) {
if (er.code === 'ENOENT') {
return
}
if (er.code === 'EPERM') {
try {
chmodSync(path, 0o666)
} catch (er2) {
if (er2.code === 'ENOENT') {
return
}
throw er
}
return fn(path)
}
throw er
}
}

module.exports = {
fixEPERM,
fixEPERMSync,
}
6 changes: 4 additions & 2 deletions lib/ignore-enoent.js
@@ -1,14 +1,16 @@
const ignoreENOENT = async p => p.catch(er => {
if (er.code !== 'ENOENT')
if (er.code !== 'ENOENT') {
throw er
}
})

const ignoreENOENTSync = fn => {
try {
return fn()
} catch (er) {
if (er.code !== 'ENOENT')
if (er.code !== 'ENOENT') {
throw er
}
}
}

Expand Down
7 changes: 7 additions & 0 deletions lib/index.js
Expand Up @@ -5,6 +5,7 @@ const {rimrafNative, rimrafNativeSync} = require('./rimraf-native.js')
const {rimrafManual, rimrafManualSync} = require('./rimraf-manual.js')
const {rimrafWindows, rimrafWindowsSync} = require('./rimraf-windows.js')
const {rimrafPosix, rimrafPosixSync} = require('./rimraf-posix.js')
const {rimrafMoveRemove, rimrafMoveRemoveSync} = require('./rimraf-move-remove.js')
const {useNative, useNativeSync} = require('./use-native.js')

const wrap = fn => async (path, opt) => {
Expand Down Expand Up @@ -58,4 +59,10 @@ posix.sync = posixSync
rimraf.posix = posix
rimraf.posixSync = posixSync

const moveRemove = wrap(rimrafMoveRemove)
const moveRemoveSync = wrapSync(rimrafMoveRemoveSync)
moveRemove.sync = moveRemoveSync
rimraf.moveRemove = moveRemove
rimraf.moveRemoveSync = moveRemoveSync

module.exports = rimraf
61 changes: 61 additions & 0 deletions lib/retry-busy.js
@@ -0,0 +1,61 @@
const MAXBACKOFF = 100
const RATE = 1.2
const MAXRETRIES = 10

const codes = new Set(['EMFILE', 'ENFILE', 'EBUSY'])
const retryBusy = fn => {
const method = async (path, opt, backoff = 1) => {
const mbo = opt.maxBackoff || MAXBACKOFF
const rate = Math.max(opt.backoff || RATE, RATE)
const max = opt.retries || MAXRETRIES
let retries = 0
while (true) {
try {
return await fn(path)
} catch (er) {
if (codes.has(er.code)) {
backoff = Math.ceil(backoff * rate)
if (backoff < mbo) {
return new Promise((res, rej) => {
setTimeout(() => {
method(path, opt, backoff).then(res, rej)
}, backoff)
})
}
if (retries < max) {
retries++
continue
}
}
throw er
}
}
}

return method
}

// just retries, no async so no backoff
const retryBusySync = fn => {
const method = (path, opt) => {
const max = opt.retries || MAXRETRIES
let retries = 0
while (true) {
try {
return fn(path)
} catch (er) {
if (codes.has(er.code) && retries < max) {
retries++
continue
}
throw er
}
}
}
return method
}

module.exports = {
retryBusy,
retryBusySync,
}

0 comments on commit 40f64ec

Please sign in to comment.