Stress and scenario testing¶
Four tools for the question "what if?". New in 0.7.0.
| Asks | Where | |
|---|---|---|
Model.stress(...) |
What is the loss distribution under each scenario? | riskpy.mc |
Result.reverse_stress(...) |
Which scenario produces the extreme outcomes? | riskpy.mc |
sensitivity |
How fast does a calculator's output change with one input? | agent tool |
scenario_table |
What does a calculator return under each named what-if? | agent tool |
Stress testing a model¶
Model.stress re-evaluates the formula with shocked inputs on the same random
draws as the base run.
from riskpy.mc import Model, Compound, LogNormal, Normal, Poisson
model = Model(
claims=Compound(Poisson(mean=120), LogNormal(mu=8.5, sigma=1.2)),
inflation=Normal(mean=0.04, sd=0.01),
)
model.correlate("claims", "inflation", 0.3)
@model.formula
def annual_loss(claims, inflation):
return claims * (1 + inflation)
stress = model.stress({
"inflation 8%": {"inflation": 0.08},
"claims +20%": {"claims": lambda x: 1.2 * x},
"heavier tail": {"claims": Compound(Poisson(120), LogNormal(8.5, 1.4))},
}, trials=100_000, seed=42)
print(stress.summary(level=0.995))
Stress test — annual_loss, 100,000 trials, level 0.995
scenario mean VaR TVaR Δ mean Δ VaR
base 1,260,374.29 2,062,312.36 2,244,163.57
inflation 8% 1,308,130.02 2,121,149.59 2,308,773.17 +47,755.73 +58,837.23
claims +20% 1,512,449.14 2,474,774.83 2,692,996.29 +252,074.86 +412,462.47
heavier tail 1,636,108.99 3,161,626.83 3,745,405.77 +375,734.70 +1,099,314.48
scenarios maps each label to {variable: shock}. A shock is one of three
things:
| Shock | Effect |
|---|---|
| a number | The variable is fixed at that value in every trial. Python and NumPy numbers both work (0.08, np.float32(0.08), np.int64(2)); True and False are refused. |
| a callable | The draws are transformed: it takes the array and returns an array of the same length. It gets a copy, so a function that changes the array in place (x *= 2) cannot leak into the base or another scenario. |
a Distribution |
The variable is redrawn from it at the same ranks as the base draw. The trial that had the worst claims still has the worst claims, so every correlation and copula is preserved. |
The remaining arguments (trials, seed, fn, vectorised, sampling,
copula) are those of Model.run, and the base is the Model.run of the same
seed, trial for trial.
The formula cannot change the draws either. It is given them read-only, and a
formula that changes an input in place (claims -= deductible, where
claims = claims - deductible was meant) is run again on copies. It gets the
answer it would get on fresh arrays, and the base, every scenario and
Result.inputs keep the draws as drawn. Changed in 0.9.0: the formula was
given the draws themselves, so such a formula applied the deductible twice in
every scenario and left its scratch values in the base's inputs. Model.run
and simulate protect the draws the same way.
How a distribution shock keeps the dependence¶
Under copula=, the new values are the shock's quantiles at the copula's own
uniforms. That is exactly the column a re-run with the new marginal, the same
copula and the same seed would draw, so a stressed default flag, claim count or
severity carries the copula's dependence value for value.
Otherwise (independent inputs, or correlate), a fresh sample of the shock is
sorted into the base ranks. Inputs with repeated values need care here: a
default flag is 0 in most trials, a claim count repeats, a low-frequency line
has many claim-free years. Trials that tie in the base are ordered by what drew
them: the normal score that correlate paired them by, or an independent
tie-break for an independent input. They are never ordered by trial number.
Before 0.8.0 they were, which dropped the dependence inside every tie group and
put all the extra defaults of a portfolio-wide PD shock on the same trials. With
20 independent obligors stressed from a 2% to a 5% PD, the 99.5% VaR came out at
20 defaults instead of 4.
The fresh sample comes from a random stream of its own for each variable, derived from the seed. A scenario's numbers therefore do not depend on which other scenarios are listed with it, or in what order, and two scenarios that redraw a variable from the same distribution use the same values.
The result is a StressResult:
baseandscenarios[label]are ordinaryResultobjects;delta(label, statistic="mean", level=0.995)is scenario minus base, for"mean","std","var"or"tvar";worst(statistic="tvar", level=0.995)names the scenario with the highest value;to_dict(level),to_frame(level)andsummary(level)tabulate everything.
Why shared draws matter¶
Running the model twice, once with each assumption, puts sampling noise from
both runs into the difference. With shared draws the noise cancels. Measured on
the model above with inflation shifted by one point, over 20 seeds of 100,000
trials: the estimated change in the mean (about 12,100) had a standard
deviation across seeds of 1,215 with two independent runs and 8.5 with
stress. The script is benchmarks/capital/run.py.
Reverse stress testing¶
A stress test starts from a scenario and asks for the outcome. A reverse stress
test starts from the outcome. Result.reverse_stress takes the trials where
the output breached a level and reports what the inputs looked like in them.
Reverse stress — annual_loss >= 2.06231e+06 (the worst 0.500%)
trials in tail 500 of 100,000 mean output there 2.24416e+06
input mean in the tail shift (sd) percentile
claims 1.21123e+06 2.13658e+06 +4.07 99.8%
inflation 0.0399921 0.050392 +1.04 85.1%
reverse_stress(level=0.995)looks at the worst1 − levelof trials: 500 of 100,000. The tail is weighted the wayResult.tvarweights it. Trials beyond the 99.5% quantile count in full. Trials tied at the quantile (a loss that is often exactly zero, or capped at a limit) share whatever weight is left. So the mean output in the tail isresult.tvar(level).threshold=takes an output value instead, and counts every trial at or beyond it in full.tail="lower"looks at the low end (profits, solvency ratios).rs.scenariois{input: mean in the tail}, the average scenario behind the extreme outcomes.rs.driverslists the inputs by how far they moved from their own mean, in standard deviations.rs.inputs[name]holdsmean,tail_mean,shift_sdandpercentile(the share of that input's own draws at or below its tail mean).rs.probability,rs.tail_trials,rs.thresholdandrs.output_meandescribe the tail;to_dict()gives it all as data.
The run must have kept its inputs (keep_inputs=True, the default).
The tail mean of an input is the conditional expectation E[input ∣ output beyond the threshold]. For a sum of normal inputs that has a closed form, μᵢ + Cov(Xᵢ, S)/σS · φ(z)/(1 − α), which is what the tests compare it with.
Sensitivities of any calculator¶
The sensitivity tool bumps one input of any calculator and revalues. It
returns the first and second derivative and the elasticity of every numeric
output.
riskpy-tools call sensitivity '{"tool": "black_scholes", "parameter": "S",
"arguments": {"S": 100, "K": 100, "T": 1, "r": 0.05, "sigma": 0.2}}'
{"tool": "black_scholes", "parameter": "S", "value": 100.0, "bump": 0.01, "scheme": "central",
"outputs": {"result": {"base": 10.450583572185565, "down": 10.444216203860648, "up": 10.456952816712189,
"derivative": 0.6368306425770243, "second_derivative": 0.018762017077733617,
"elasticity": 6.093732834900834}}}
The derivative is the option's delta (0.63683) and the second derivative its gamma (0.018762).
| Argument | Meaning |
|---|---|
tool |
the calculator to differentiate |
arguments |
its base-case arguments, as an object |
parameter |
the numeric argument to bump |
bump |
the bump in the parameter's own units; default 0.0001 × max(1, ∣value∣) for a closed form, 1% of ∣value∣ for a simulation (1% of the distance to 0 or 1 for a confidence level), or 1 for a whole-number parameter |
relative |
true to read bump as a fraction of the value (0.01 is a 1% bump) |
Things to know:
- Units. Derivatives are per one whole unit of the input. Multiply by 0.0001 for one basis point of a rate and by 0.01 for one point of volatility.
- Outputs are keyed by dotted path:
price,var.0.995, orresultwhen the calculator returns a bare number. - Elasticity is derivative × value / base, the percentage change in the
output per 1% change in the input. It is
nullwhen the base output is zero. - Edges. If one bumped value is refused (volatility below zero, say), the
tool uses a three-point one-sided difference and reports
"scheme": "forward"or"backward". - Simulations need a non-zero
seedinarguments, so that every revaluation uses the same draws. Without one the call is refused (RP113). Even with the seed fixed, a simulated output moves in steps, so the default bump for a simulation is 1% of the value rather than 0.0001 of it; repeat with another bump before relying on a simulated derivative. - Budget. The base run and the bumped runs together may ask for no more
work than one call of the calculator at its limit (RP405), and the same
holds for the base and the scenarios of a
scenario_table.
Scenario tables for any calculator¶
scenario_table runs a calculator under named sets of overrides and reports
each result against the base case.
from riskpy import tools
table = tools.call("scenario_table", {
"tool": "basel_irb_capital",
"arguments": {"pd": 0.01, "lgd": 0.45, "ead": 1_000_000},
"scenarios": {"downturn": {"pd": 0.03, "lgd": 0.6}, "typo": {"pd": 3}},
})
table["base"]["result"]["capital"] # 73,853.44
table["scenarios"]["downturn"]["change"]["capital"] # +63,146.82
table["scenarios"]["downturn"]["relative_change"]["capital"] # 0.855
table["scenarios"]["typo"]["error"]["code"] # 'RP103'
table["failed"] # 1
Each scenario's overrides replace the base arguments. For every numeric output
the table gives change (scenario minus base) and relative_change (as a
decimal). A scenario the calculator refuses is reported under its own label
with the coded error, and the others still run. An override that the calculator
does not take fails the whole call with RP402 and a suggestion. One call takes
at most 100 scenarios.
Both tools work on every calculator in riskpy-tools, need nothing beyond what
the calculator itself needs, and are available over MCP.