Remove the Active → Faulted transition

[mkeeter]

This PR removes the Active → Faulted transition when too many IO operations are in flight.

See #1252 for details on why this is desirable!

There are a bunch of changes that come along for the ride:

• The backpressure equation is changed from quadratic to the form 1 / (1 - f), i.e. it now goes to infinity at our desired maximum value. These maximum values are set at 2x the fault for the Offline → Faulted transition.
• Computing backpressure is now a standalone function on BackpressureConfig, so it can be unit tested
• Unit tests are added to confirm backpressure properties
  • The Offline → Faulted transition happens at a reasonable timescale (> 1 sec, < 2 minutes)
  • Backpressure goes to ∞ as we approach these limits
• Backpressure settings are tweaked based on these new requirements

[leftwo]

This probably should have been some test global to begin with, so you did not
have to find it and change it to match the default_config extent size.

[mkeeter]

Thanks for the nudge, done in 9760bad

[mkeeter]

I did a performance sweep on the Madrid cluster, using the new crutest rand-read/write subcommands:

#!/bin/sh
set -e

if [[ -z ${CRUTEST_BIN} ]]; then
    print "Error: must run with CRUTEST_BIN set"
    exit 1
fi

FLAGS="\
    -t [fd00:1122:3344:104::1]:28830 \
    -t [fd00:1122:3344:103::1]:28830 \
    -t [fd00:1122:3344:101::1]:28830 \
    --io-depth=8 -q --time=60 --sample-time=5 --subsample-count=2\
    --key tCw7zw0hAsPuxMOTWwnPEFYjBK9qJRtYyGdEXKEnrg0= \
"

${CRUTEST_BIN} rand-write $FLAGS --gen $(date "+%s") --io-size=1024
${CRUTEST_BIN} rand-write $FLAGS --gen $(date "+%s") --io-size=1
${CRUTEST_BIN} rand-write $FLAGS --gen $(date "+%s") --io-size=256
${CRUTEST_BIN} rand-write $FLAGS --gen $(date "+%s") --io-size=1024
${CRUTEST_BIN} rand-read $FLAGS --gen $(date "+%s") --io-size=1
${CRUTEST_BIN} rand-read $FLAGS --gen $(date "+%s") --io-size=256
${CRUTEST_BIN} rand-read $FLAGS --gen $(date "+%s") --io-size=1024

main

4M write: 806.8 MiB/sec ± 35.6 MiB/sec (201 IOPS)
4K write: 28.7 MiB/sec ± 2.3 MiB/sec (7334 IOPS)
1M write: 813.5 MiB/sec ± 29.5 MiB/sec (813 IOPS)
4M write: 779.3 MiB/sec ± 40.3 MiB/sec (194 IOPS)
4K read:  28.4 MiB/sec ± 273.8 KiB/sec (7275 IOPS)
1M read:  825.3 MiB/sec ± 43.4 MiB/sec (825 IOPS)
4M read:  733.2 MiB/sec ± 27.5 MiB/sec (183 IOPS)

This branch

4M write: 783.8 MiB/sec ± 23.2 MiB/sec (195 IOPS)
4K write: 32.1 MiB/sec ± 2 MiB/sec (8227 IOPS)
1M write: 795.8 MiB/sec ± 29.6 MiB/sec (795 IOPS)
4M write: 796.5 MiB/sec ± 34.6 MiB/sec (199 IOPS)
4K read:  27.7 MiB/sec ± 290.7 KiB/sec (7096 IOPS)
1M read:  744.3 MiB/sec ± 15.5 MiB/sec (744 IOPS)
4M read:  818.9 MiB/sec ± 28.3 MiB/sec (204 IOPS)

---

Writes consistently look about the same; reads are a little more varied but I don't see any major red flags.

[faithanalog]

All makes sense to me.