overall_stats¶

Variables

struct [anonymous] get_overall_accuracy¶

Overall Accuracy.

overall_accuracy = sum(TP) / POP

Param TP: array of true positives for each class.
Param TOP: array of positive predicted values for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall accuracy score.

struct [anonymous] get_overall_random_accuracy_unbiased¶

Overall random accuracy unbiased.

overall_random_accuracy = sum(random_accuracy_unbiased)

Param RACCU: array of random accuracies unbiased for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall random accuracy unbiased score.

struct [anonymous] get_overall_random_accuracy¶

Overall random accuracy.

overall_random_accuracy = sum(random_accuracy)

Param RACC: array of random accuracies for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall random accuracy score.

struct [anonymous] get_overall_kappa¶

Kappa.

Kappa is a statistic that measures inter-rater agreement for qualitative (categorical) items. It is generally thought to be a more robust measure than simple percent agreement calculation, as kappa takes into account the possibility of the agreement occurring by chance.

Kappa = (overall_ACC - overall_RACC) / (1 - overall_RACC)

Param overall_random_accuracy: Overall RACC value.
Param overall_accuracy: Overall ACC value.
Return: The Kappa overall score.

struct [anonymous] get_PC_PI¶

Utility function for future scores.

It compute the simple formula:

pc = sum((TOP + P)/(2 * POP)**2)

Param P: array of the number of positive samples for each class.
Param TOP: array of positive predicted values for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The PC_PI score.

struct [anonymous] get_PC_AC1¶

Utility function for future scores.

It compute the simple formula:

pi = (TOP * P) / (2 * POP)
pc = 1 / (abs(C) - 1) * sum(pi * (1 - pi))

Param P: array of the number of positive samples for each class.
Param TOP: array of positive predicted values for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The PC_AC1 score.

struct [anonymous] get_PC_S¶

One over classes.

It is just an utility function for future scores.

Param classes: array of classes found (this variable is useless in the computation but it is necessary for the graph estimation).
Param Nclass: size of classes array (aka number of classes)
Return: The PC_S score

struct [anonymous] get_PI¶

Scott’s Pi.

Scott’s pi (named after William A. Scott) is a statistic for measuring inter-rater reliability for nominal data in communication studies. Textual entities are annotated with categories by different annotators, and various measures are used to assess the extent of agreement between the annotators, one of which is Scott’s pi. Since automatically annotating text is a popular problem in natural language processing, and the goal is to get the computer program that is being developed to agree with the humans in the annotations it creates, assessing the extent to which humans agree with each other is important for establishing a reasonable upper limit on computer performance.

pc = sum((TOP + P) / (2 * POP)**2)
π = (overall_accuracy - pc) / (1 - pc)

Param PC_PI: pc score in the formula.
Param overall_accuracy: Overall accuracy score.
Return

struct [anonymous] get_AC1¶

Gwet’s AC1.

AC1 was originally introduced by Gwet in 2001 (Gwet, 2001). The interpretation of AC1 is similar to generalized kappa (Fleiss, 1971), which is used to assess inter-rater reliability when there are multiple raters. Gwet (2002) demonstrated that AC1 can overcome the limitations that kappa is sensitive to trait prevalence and rater’s classification probabilities (i.e., marginal probabilities), whereas AC1 provides more robust measure of inter-rater reliability.

AC1 = (overall_accuracy - pc) / (1 - pc)

Param PC_AC1: the pc in the formula.
Param overall_accuracy: Overall accuracy score.
Return: The Gwet’s AC1 score.

struct [anonymous] get_S¶

Bennett’s S.

Bennett, Alpert & Goldstein’s S is a statistical measure of inter-rater agreement. It was created by Bennett et al. in 1954. Bennett et al. suggested adjusting inter-rater reliability to accommodate the percentage of rater agreement that might be expected by chance was a better measure than a simple agreement between raters.

pc = 1 / abs(C)
S = (overall_accuracy - pc) / (1 - pc)

Param PC_S: one over classes.
Param overall_accuracy: Overall accuracy score.
Return: The Bennett’s S score.

struct [anonymous] get_kappa_SE¶

Kappa standard error.

The standard error(s) of the Kappa coefficient was obtained by Fleiss (1969)

Kappa_se = sqrt((overall_accuracy * (1 - overall_random_accuracy)) / (1 - overall_random_accuracy)**2)

Param overall_accuracy: Overall ACC value.
Param overall_random_accuracy_unbiased: Overall RACCU value.
Param POP: array of total samples for each class.
Return: The Kappa standard error overall score.

struct [anonymous] get_kappa_unbiased¶

Kappa unbiased.

The unbiased kappa value is defined in terms of total accuracy and a slightly different computation of expected likelihood that averages the reference and response probabilities.

Kappa_unbiased = (overall_ACC - overall_RACCU) / (1 - overall_RACC)

Param overall_random_accuracy_unbiased: Overall RACCU value.
Param overall_accuracy: Overall ACC value.
Return: The Kappa unbiased overall score.

struct [anonymous] get_kappa_no_prevalence¶

Kappa no prevalence.

The kappa statistic adjusted for prevalence.

Kappa_no_prevalence = 2 * overall_ACC - 1

Param overall_accuracy: Overall ACC value.
Return: The Kappa without prevalence overall score.

struct [anonymous] get_kappa_CI_up¶

Kappa 95% CI.

Kappa 95% Confidence (upper) Interval

Kappa_ci = overall_kappa + 1.96 * kappa_SE

Param overall_kappa: Kappa overall score.
Param kappa_SE: kappa standard error score.
Return: The Kappa 95% CI (upper bound) score.

struct [anonymous] get_kappa_CI_down¶

Kappa 95% CI.

Kappa 95% Confidence (lower) Interval

Kappa_ci = overall_kappa - 1.96 * kappa_SE

Param overall_kappa: Kappa overall score.
Param kappa_SE: kappa standard error score.
Return: The Kappa 95% CI (lower bound) score.

struct [anonymous] get_overall_accuracy_se¶

Standard error.

The standard error (SE) of a statistic (usually an estimate of a parameter) is the standard deviation of its sampling distribution or an estimate of that standard deviation.

SE =sqrt(overall_accuracy * (1 - overall_accuracy) / POP)

Param overall_accuracy: Overall accuracy score.
Param POP: array of total samples for each class.
Return: The accuracy standard error score.

struct [anonymous] get_overall_accuracy_ci_up¶

95% CI (upper bound)

In statistics, a confidence interval (CI) is a type of interval estimate (of a population parameter) that is computed from the observed data. The confidence level is the frequency (i.e., the proportion) of possible confidence intervals that contain the true value of their corresponding parameter. In other words, if confidence intervals are constructed using a given confidence level in an infinite number of independent experiments, the proportion of those intervals that contain the true value of the parameter will match the confidence level.

CI = overall_accuracy + 1.96 * overall_accuracy_se

Param overall_accuracy: Overall accuracy score.
Param overall_accuracy_se: Overall standard error score.
Return: The upper bound CI.

struct [anonymous] get_overall_accuracy_ci_down¶

95% CI (lower bound)

In statistics, a confidence interval (CI) is a type of interval estimate (of a population parameter) that is computed from the observed data. The confidence level is the frequency (i.e., the proportion) of possible confidence intervals that contain the true value of their corresponding parameter. In other words, if confidence intervals are constructed using a given confidence level in an infinite number of independent experiments, the proportion of those intervals that contain the true value of the parameter will match the confidence level.

CI = overall_accuracy - 1.96 * overall_accuracy_se

Param overall_accuracy: Overall accuracy score.
Param overall_accuracy_se: Overall standard error score.
Return: The lower bound CI.

struct [anonymous] get_chi_square¶

Chi-squared.

Pearson’s chi-squared test is a statistical test applied to sets of categorical data to evaluate how likely it is that any observed difference between the sets arose by chance. It is suitable for unpaired data from large samples.

Param confusion_matrix: Confusion matrix.
Param TOP: array of positive predicted values for each class.
Param P: array of true positive rates for each class.
Param POP: array of true positive rates for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The Chi-Squared score.

struct [anonymous] get_phi_square¶

Phi-squared.

In statistics, the phi coefficient (or mean square contingency coefficient) is a measure of association for two binary variables. Introduced by Karl Pearson, this measure is similar to the Pearson correlation coefficient in its interpretation. In fact, a Pearson correlation coefficient estimated for two binary variables will return the phi coefficient.

phi**2 = chi**2 / POP

Param chi_square: Chi-Squared score.
Param POP: array of true positive rates for each class.
Return: The Phi-squared score.

struct [anonymous] get_cramer_V¶

Cramer’s V.

In statistics, Cramér’s V (sometimes referred to as Cramér’s phi) is a measure of association between two nominal variables, giving a value between 0 and +1 (inclusive). It is based on Pearson’s chi-squared statistic and was published by Harald Cramér in 1946.

V = sqrt((phi**2) / (abs(C) - 1)

Param phi_square: Phi-Squared score.
Param Nclass: size of classes array (aka number of classes)
Return: The Cramer’s V score.

struct [anonymous] get_response_entropy¶

Response entropy.

The entropy of the response distribution. The entropy of a distribution is the average negative log probability of outcomes.

likelihood_response = TOP/POP
entropy_response = - sum(likelihood_response * log2(likelihood_response))

Param TOP: array of positive predicted values for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The response entropy score.

struct [anonymous] get_reference_entropy¶

Reference entropy.

The entropy of the decision problem itself as defined by the counts for the reference. The entropy of a distribution is the average negative log probability of outcomes.

likelihood_reference = P/POP
entropy_reference = - sum(likelihood_reference * log2(likelihood_reference))

Param P: array of the number of positive samples for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The reference entropy score.

struct [anonymous] get_cross_entropy¶

Cross entropy.

The cross-entropy of the response distribution against the reference distribution. The cross-entropy is defined by the negative log probabilities of the response distribution weighted by the reference distribution.

likelihood_reference = P/POP
likelihood_response = TOP/POP
entropy_cross = - sum(likelihood_reference * log2(likelihood_response))

Param TOP: array of positive predicted values for each class.
Param P: array of the number of positive samples for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The cross entropy score.

struct [anonymous] get_join_entropy¶

Joint entropy.

The entropy of the joint reference and response distribution as defined by the underlying matrix.

P = confusion_matrix / POP
entropy_joint = - sum(P * log2(P))

Param confusion_matrix: Confusion matrix.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The joint entropy score.

struct [anonymous] get_conditional_entropy¶

Conditional entropy.

The entropy of the distribution of categories in the response given that the reference category was as specified.

Param confusion_matrix: Confusion matrix.
Param P: array of the number of positive samples for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The conditional entropy score.

struct [anonymous] get_mutual_information¶

Mutual information.

Mutual information is defined as Kullback-Leibler divergence, between the product of the individual distributions and the joint distribution. Mutual information is symmetric. We could also subtract the conditional entropy of the reference given the response from the reference entropy to get the same result.

MI = response_entropy - conditional_entropy

Param response_entropy: The response entropy score.
Param conditional_entropy: The conditional entropy score.
Return: The mutual information score.

struct [anonymous] get_kl_divergence¶

Kullback-Leibler divergence.

In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy) is a measure of how one probability distribution diverges from a second, expected probability distribution.

likelihood_response = TOP/POP
likelihood_reference = P/POP
KL = - sum(likelihood_reference * log2(likelihood_reference/likelihood_response))

Param P: array of the number of positive samples for each class.
Param TOP: array of positive predicted values for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The Kullback-Leibler divergence score.

struct [anonymous] get_lambda_B¶

Goodman & Kruskal’s lambda B.

In probability theory and statistics, Goodman & Kruskal’s lambda is a measure of proportional reduction in error in cross tabulation analysis.

Param confusion_matrix: Confusion matrix.
Param TOP: array of positive predicted values for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The lambda B score.

struct [anonymous] get_lambda_A¶

Goodman & Kruskal’s lambda A.

In probability theory and statistics, Goodman & Kruskal’s lambda is a measure of proportional reduction in error in cross tabulation analysis.

Param confusion_matrix: Confusion matrix.
Param P: array of the number of positive samples for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The lambda A score.

struct [anonymous] get_DF¶

Chi-squared DF.

Number of degrees of freedom of this confusion matrix for the chi-squared statistic.

DF = (abs(C) - 1)**2

Param classes: array of classes found (this variable is useless in the computation but it is necessary for the graph estimation).
Param Nclass: size of classes array (aka number of classes)
Return: The Chi-squared DF score.

struct [anonymous] get_overall_jaccard_index¶

Overall Jaccard index.

overall_jaccard_index = sum(jaccard_index)

Param jaccard_index: array of Jaccard indexes for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall Jaccard index score.

struct [anonymous] get_hamming_loss¶

Hamming loss.

The average Hamming loss or Hamming distance between two sets of samples.

hamming_loss = 1 / POP * sum(TP)

Param TP: array of true positives for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall Hamming loss score.

struct [anonymous] get_zero_one_loss¶

Zero-one loss.

Zero-one loss is a common loss function used with classification learning. It assigns 0 to loss for a correct classification and 1 for an incorrect classification.

L0 = POP - sum(TP)

Param TP: array of true positives for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall Zeor-one loss score.

struct [anonymous] get_NIR¶

No information rate.

Largest class percentage in the data.

NIR = 1/POP max(P)

Param P: array of number of positive samples for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall No informative rate score.

struct [anonymous] get_p_value¶

P-Value.

In statistical hypothesis testing, the p-value or probability value is, for a given statistical model, the probability that, when the null hypothesis is true, the statistical summary (such as the absolute value of the sample mean difference between two compared groups) would be greater than or equal to the actual observed results. Here a one-sided binomial test to see if the accuracy is better than the no information rate.

x = sum(TP)
p = NIR
n = POP
p_value = 1 - sum([binomial(n, i) * p**i * (1 - p)**(n - i) for i in range(x)])

Param TP: array of number of true positives for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Param NIR: the overall No informative rate score.
Return: The p-value score.

struct [anonymous] get_overall_CEN¶

Overall CEN.

Param TOP: array of positive predicted values for each class.
Param P: array of number of positive samples for each class.
Param CEN: array of confusion entropies for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall CEN score.

struct [anonymous] get_overall_MCEN¶

Overall MCEN.

Param TP: array of true positive values for each class.
Param TOP: array of positive predicted values for each class.
Param P: array of number of positive samples for each class.
Param MCEN: array of modified confusion entropies for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall MCEN score.

struct [anonymous] get_overall_MCC¶

Overall MCC.

Param confusion_matrix: Confusion matrix.
Param TOP: array of positive predicted values for each class.
Param P: array of number of positive samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The overall MCC score.

struct [anonymous] get_RR¶

Global performance index.

RR = 1 / abs(C) * sum(confusion_matrix)

Param TOP: array of positive predicted values for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The RR index score.

struct [anonymous] get_CBA¶

Class balance accuracy.

As an evaluation tool, CBA creates an overall assessment of model predictive power by scrutinizing measures simultaneously across each class in a conservative manner that guarantees that a model’s ability to recall observations from each class and its ability to do so efficiently won’t fall below the bound.

CBA = sum(confusion_matrix / max(TOP, P)) / abs(C)

Param confusion_matrix: Confusion matrix.
Param TOP: array of positive predicted values for each class.
Param P: array of number of positive samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The CBA score.

struct [anonymous] get_AUNU¶

AUNU.

When dealing with multiclass problems, a global measure of classification performances based on the ROC approach (AUNU) has been proposed as the average of single-class measures.

AUNU = sum(AUC) / abs(C)

Param AUC: array of area under the curve score for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The arrau of AUNU score.

struct [anonymous] get_AUNP¶

AUNP.

Another option (AUNP) is that of averaging the AUCi values with weights proportional to the number of samples experimentally belonging to each class, that is, the a priori class distribution.

AUNP = sum(P / POP * AUC)

Param P: array of number of positive samples for each class.
Param POP: array of total samples for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The arrau of AUNP score.

struct [anonymous] get_RCI¶

Relative classifier information.

Performance of different classifiers on the same domain can be measured by comparing relative classifier information while classifier information (mutual information) can be used for comparison across different decision problems.

Param mutual_information: array of MI scores for each class.
Param reference_entropy: array of reference entropy scores for each class.
Return: The RCI score.

struct [anonymous] get_CSI¶

Classification success index.

The Classification Success Index (CSI) is an overall measure defined by averaging ICSI over all classes.

CSI = 1 / abs(C) * sum(ICSI)

Param ICIS: array of individual classification success indexes for each class.
Param Nclass: size of classes array (aka number of classes)
Return

struct [anonymous] get_overall_pearson_C¶

Pearson’s C.

The contingency coefficient is a coefficient of association that tells whether two variables or data sets are independent or dependent of/on each other. It is also known as Pearson’s coefficient (not to be confused with Pearson’s coefficient of skewness).

C = sqrt(chi**2 / (chi**2 + POP))

Param chi_square: Chi-squared score.
Param POP: array of total samples for each class.
Return: The Pearson’s C score.

struct [anonymous] get_TPR_PPV_F1_micro¶

Utility function for micro score evaluation (it is valid for TPR micro, PPV micro, F1 micro)

The function computes the score according to the formula:

res = sum(TP) / (sum(TP) + sum(FN))

Param TP: array of true positives for each class.
Param FN: array of false negatives for each class.
Param Nclass: size of classes array (aka number of classes)
Return: The required score.

struct [anonymous] get_MCC_analysis¶

Matthews’s benchmark.

MCC is a confusion matrix method of calculating the Pearson product-moment correlation coefficient (not to be confused with Pearson’s C). Therefore, it has the same interpretation. The score follows the ranges

SOA6 < 0.3 Negligible
0.3 < SOA6 < 0.5 Weak
0.5 < SOA6 < 0.7 Moderate
0.7 < SOA6 < 0.9 Strong
0.9 < SOA6 < 1.0 Very strong

Param overall_MCC: Overall Matthews correlation coefficient.
Return: The value of SOA6 score.

struct [anonymous] get_kappa_analysis_cicchetti¶

Cicchetti’s benchmark.

The score follows the ranges

SOA4 < 0.4 Poor
0.4 < SOA4 < 0.6 Fair
0.6 < SOA4 < 0.75 Good
0.75 < SOA4 < 1.0 Excellent

Param overall_kappa: Overall Kappa score.
Return: The value of SOA4 score.

struct [anonymous] get_kappa_analysis_koch¶

Landis & Koch’s benchmark.

The score follows the ranges

SOA1 < 0 Poor
0 < SOA1 < 0.2 Slight
0.2 < SOA1 < 0.4 Fair
0.4 < SOA1 < 0.6 Moderate
0.6 < SOA1 < 0.8 substantial
0.8 < SOA1 < 1.0 Almost perfect

Param overall_kappa: Overall Kappa score.
Return: The value of SOA1 score.

struct [anonymous] get_kappa_analysis_fleiss¶

Fleiss’ benchmark.

The score follows the ranges

SOA2 < 0.4 Poor
0.4 < SOA2 < 0.75 Intermediate to Good
SOA2 > 0.75 Excellent

Param overall_kappa: Overall Kappa score.
Return: The value of SOA2 score.

struct [anonymous] get_kappa_analysis_altman¶

Altman’s benchmark.

The score follows the ranges

SOA3 < 0.2 Poor
0.2 < SOA3 < 0.4 Fair
0.4 < SOA3 < 0.6 Moderate
0.6 < SOA3 < 0.8 Good
0.8 < SOA3 < 1.0 Very good

Param overall_kappa: Overall Kappa score.
Return: The value of SOA3 score.

struct [anonymous] get_V_analysis¶

Cramer’s benchmark.

The score follows the ranges

SOA5 < 0.1 Negligible
0.1 < SOA5 < 0.2 Weak
0.2 < SOA5 < 0.4 Moderate
0.4 < SOA5 < 0.6 Relatively Strong
0.6 < SOA5 < 0.8 Strong
0.8 < SOA5 < 1.0 Very strong

Param cramer_V: Overall Cramer’s V score.
Return: The value of SOA5 score.

struct [anonymous] get_TPR_macro¶

TPR_Macro.

TPR_macro = 1/abs(C) * sum(TP / (TP + FN))

Param TPR: array of true positive rates for each class.
Param Nclass: size of classes array (aka number of classes)
Return: the TPR macro score.

struct [anonymous] get_PPV_macro¶

PPV_Macro.

PPV_macro = 1/abs(C) * sum(TP / (TP + FN))

Param PPV: array of true positive predicted values for each class.
Param Nclass: size of classes array (aka number of classes)
Return: the PPV macro score.

struct [anonymous] get_ACC_macro¶

ACC_Macro.

ACC_macro = 1/abs(C) * sum(ACC)

Param ACC: array of accuracies for each class.
Param Nclass: size of classes array (aka number of classes)
Return: the ACC macro score.

struct [anonymous] get_F1_macro¶

F1_Macro.

F1_macro = 2/abs(C) * sum((TPR * PPV) / (TPR + PPV))

Param F1_score: array of F1 scores for each class.
Param Nclass: size of classes array (aka number of classes)
Return: the F1 macro score.