scorer

class scorer.scorer.scorer.Scorer

Bases: dict

Multi-class score computation.

This class represents an optimized and extended version of the PyCM library. The full list of scores are evaluated using C++ functions wrapped into a single score object. The evaluation of the score functions can be performed into a parallel environment using OMP multhithreading. The C++ code is in fact auto-generated using the scripts provided into the utils directory and the optimal dependency graph is computed to allow the work distribution among the available threads.

Example

>>> from scorer import Scorer
>>>
>>> y_true = ['a', 'b', 'a', 'a', 'b', 'c', 'c', 'a', 'a', 'b', 'c', 'a']
>>> y_pred = ['b', 'b', 'a', 'c', 'b', 'a', 'c', 'b', 'a', 'b', 'a', 'a']
>>>
>>> scorer = Scorer()
>>> scorer.evaluate(y_true, y_pred)

References

• Haghighi, S., Jasemi, M., Hessabi, S. and Zolanvari, A. (2018). PyCM: Multiclass confusion matrix library in Python. Journal of Open Source Software, 3(25), p.729.

__getattr__(stat)

Access to score stats as attribute

Parameters

stat (name) – Name of the score

Examples

>>> from scorer import scorer
>>> size = 10
>>> y_true = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>> y_pred = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>>
>>> scorer = Scorer()
>>> scorer.evaluate(y_true, y_pred)
>>> print(scorer.ACC, scorer.TP, scorer.FP)

Notes

Note

In many cases the string related to the score is very long and it includes information about the mathematical meaning of that score. To facilitate the usage of the class the search of the attributes is performed using a “regex” search. In this way it is possible to access member values as in the following example

np.testing.assert_allclose(scorer['ACC(Accuracy)'], scorer.ACC)
np.testing.assert_allclose(scorer['FP(False positive/type 1 error/false alarm)'], scorer.FP)
np.testing.assert_allclose(scorer['TOP(Test outcome positive)'], scorer.TOP)
np.testing.assert_allclose(scorer['FDR(False discovery rate)'], scorer.FDR)

If the attribute is not found an AttributeError is raised.

__getitem__(stat)

Get the value of the required score

Parameters

stat (str) – Name of the score

Examples

>>> from scorer import scorer
>>> size = 10
>>> y_true = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>> y_pred = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>>
>>> scorer = Scorer()
>>> scorer.evaluate(y_true, y_pred)
>>>
>>> print(scorer['accuracy_score'])

Notes

Note

The search of the score name is performed using the key name of the dictionary. This function is different from __getattr__.

__init__()

Default constructor

__setitem__(stat, values)

Set a score variable.

Parameters

• stat (str) – Key as name of the new score

• values (float or list) – Value(s) of the new score

Examples

>>> from scorer import scorer
>>> size = 10
>>> y_true = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>> y_pred = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>>
>>> scorer = Scorer()
>>> scorer.evaluate(y_true, y_pred)
>>>
>>> scorer['dummy'] = 'dummy'
  UserWarning: Setting new statistics does not enable
  the computation of the dependencies

__str__()

Print the object as table of scores

_check_params(true, pred)

Check input dimension shapes

Parameters

• true (array-like) – True label array

• pred (array-like) – Predicted label array

Notes

Note

The array of true labels and predicted ones mush have the same length. If the given arrays have different shapes a ValueError is raised.

_label2numbers(arr)

Convert labels to numerical values

Parameters

arr (array_like) – The array of labels

Returns

numeric_labels – Array of numerical labels obtained by the LabelEncoder transform

Return type

np.ndarray

Notes

Note

The C++ function allows only numerical (integer) values as labels in input. For more general support refers to the C++ example.

Examples

>>> from scorer import scorer
>>> y = ('A', 'A', 'B', 'B')
>>> num_y = scorer()._label2numbers(y)
>>> print(num_y)
  [0, 0, 1, 1]

evaluate(lbl_true, lbl_pred)

Evaluate scores of prediction labels vs true labels

Parameters

• lbl_true (array-like) – List of true labels

• lbl_pred (array-like) – List of predicted labels

Return type

self

Examples

>>> from scorer import scorer
>>> size = 10
>>> y_true = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>> y_pred = np.random.choice([0., 1.], p=[.5, .5], size=(size, ))
>>>
>>> scorer = Scorer()
>>> scorer.evaluate(y_true, y_pred)
>>>
>>> # Or using simple lists
>>>
>>> y_true = y_true.tolist()
>>> y_pred = y_pred.tolist()
>>>
>>> scorer.evaluate(y_true, y_pred)

Notes

Note

The score evaluation is possible only with integer labels. The input labels are encoded in integers using the C++ version of the label encoder (_label2numbers).

property num_classes

Return the number of classes identified. If the scores are not yet evaluated the return value is 0.

property score

Return the score list as dictionary.