Hi John,

Sorry for the delay in getting back to you.

1) Assuming ESD low relaxation is set to 1 (presumably no effect), if outliers is set to 0.3, does that mean:

a) that 30% of the total pixel stack can be rejected by the combined effects of high clipping and low clipping?

ESD rejection doesn't work like sigma clipping and similar rejection methods, so the concepts of 'low' and 'high' pixels are not directly portable. The answer to this question is approximately a).

Option (a) is implied by the tooltip, "The default value is 0.3, which allows the algorithm to detect up to a 30% of outlier pixels in each pixel stack." But this would mean that high clipping would be affected by turning on or off low clipping, which would certainly be an undesirable effect.

Actually, the ESD algorithm does not 'know' whether it is rejecting low or high pixels. It works on absolute values with respect to a central value (which is not the median except for very small stacks) without taking signs into account. We introduce a relaxation factor for pixels below the central value in each iteration, which conditions the algorithm's behavior as a whole, but the decision to reject or not high/low pixels as a function of process parameters is taken at the very end of the process, when all outliers have already been detected.

This can be seen more clearly in our implementation. This is the main rejection loop:

PixInsight Class Library

gitlab.com

and here is when low and high pixels are actually rejected depending on process parameters, that is, on whether low and/or high clipping is enabled:

PixInsight Class Library

gitlab.com

As you can see, both sections of the algorithm are separate and independent each other.

2) Is the ESD significance test applied to:

a) each individual potential outlier?

b) the entire pixel stack?

The generalized ESD test algorithm is quite well described here:

If I've understood it well, the answer to your question would be a mix of a) and b). The test is repeated for the whole stack, but removing potential outliers at each successive iteration to compute the next R_i test statistic.

Note that our implementation introduces two important changes to the original algorithm that make the ESD rejection method more robust and versatile. For robustness, we don't use the mean and the standard deviation of the sample, but a trimmed mean and the standard deviation of the trimmed sample at each iteration. In all of our tests the algorithm behaves much more consistently with this variation. For versatility, we introduce a relaxation factor that multiplies the standard deviation (the s variable in the algorithm description above) for pixels with values smaller than the trimmed mean. This allows us to apply a more tolerant rejection for dark pixels.

That is, the tooltip implies that the test is all or nothing, either there are outliers that can be rejected or there are not.

This is not what this tooltip tries to communicate. The ESD significance parameter does not define a Boolean condition (reject or don't reject) over the whole stack. It works as a limit to compute critical values (lambda_i variables), which are compared to test statistics (R_i) in order to find the number of outliers. By increasing the ESD significance parameter more pixels will be rejected because the algorithm will allow more mistakes made by rejecting the null hypothesis (no outliers in the stack). It's a bit convoluted but that's how it works.

The ESD outliers parameter defines an upper bound for the number of outliers that can be rejected in the sample. For example, for a stack of 10 pixels, if ESD outliers is equal to 0.3 this means that 0, 1, 2 or 3 outliers can be detected. Of course, this also means that in case there were an additional fourth outlier, it would pass unnoticed.

3) What parameter is relaxation applied to:

a) a divisor for the outliers parameter?

b) a divisor for the significance parameter?

c) something else?

The answer is c). As I've noted above, the relaxation parameter multiplies the standard deviation of the (trimmed) sample at each iteration to compute test statistics for pixels with values smaller than the trimmed mean. Since the low relaxation parameter is >= 1, it causes the algorithm to be more tolerant for rejection of low pixels, since the algorithm 'sees' a higher dispersion for these low pixels. In a sense, what we are doing here is telling lies to the ESD test for a subset of pixels where we want less rejection.