Skip to content

[MRG] FIX top_k_accuracy_score ignoring labels for "multiclass" case #19721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 26, 2021
Merged

[MRG] FIX top_k_accuracy_score ignoring labels for "multiclass" case #19721

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
adc32fe
Typo
joclement Jan 28, 2021
6d5b13d
Add test case for multiclass top_k_accuracy_score
joclement Jan 29, 2021
3a2f840
Fix top_k_accuracy_score for multiclass case
joclement Jan 28, 2021
47a4098
Impmrove check on labels
joclement Jan 29, 2021
ab1dea2
Avoid duplicate tests
joclement Jan 29, 2021
9b0322b
Test strings in labels
joclement Jan 29, 2021
b46b3f0
Test labels as ndarray
joclement Jan 29, 2021
ca32af9
Add comment explaining test
joclement Jan 29, 2021
feb62fe
Add changelog entry
joclement Jan 29, 2021
cc23ce3
Improve description of test
joclement Apr 3, 2021
9c38c23
Update v0.24.rst
glemaitre Apr 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions doc/whats_new/v0.24.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,14 @@ Changelog
- |Fix|: Fixed a bug in :class:`linear_model.LogisticRegression`: the
sample_weight object is not modified anymore. :pr:`19182` by
:user:`Yosuke KOBAYASHI <m7142yosuke>`.

:mod:`sklearn.metrics`
......................

- |Fix| :func:`metrics.top_k_accuracy_score` now supports multiclass
problems where only two classes appear in `y_true` and all the classes
are specified in `labels`.
:pr:`19721` by :user:`Joris Clement <flyingdutchman23>`.

:mod:`sklearn.model_selection`
..............................
Expand Down
4 changes: 3 additions & 1 deletion sklearn/metrics/_ranking.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1598,7 +1598,7 @@ def top_k_accuracy_score(y_true, y_score, *, k=2, normalize=True,
non-thresholded decision values (as returned by
:term:`decision_function` on some classifiers). The binary case expects
scores with shape (n_samples,) while the multiclass case expects scores
with shape (n_samples, n_classes). In the nulticlass case, the order of
with shape (n_samples, n_classes). In the multiclass case, the order of
the class scores must correspond to the order of ``labels``, if
provided, or else to the numerical or lexicographical order of the
labels in ``y_true``.
Expand Down Expand Up @@ -1655,6 +1655,8 @@ def top_k_accuracy_score(y_true, y_score, *, k=2, normalize=True,
y_true = check_array(y_true, ensure_2d=False, dtype=None)
y_true = column_or_1d(y_true)
y_type = type_of_target(y_true)
if y_type == "binary" and labels is not None and len(labels) > 2:
y_type = "multiclass"
y_score = check_array(y_score, ensure_2d=False)
y_score = column_or_1d(y_score) if y_type == 'binary' else y_score
check_consistent_length(y_true, y_score, sample_weight)
Expand Down
24 changes: 24 additions & 0 deletions sklearn/metrics/tests/test_ranking.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1657,6 +1657,30 @@ def test_top_k_accuracy_score_binary(y_score, k, true_score):
assert score == score_acc == pytest.approx(true_score)


@pytest.mark.parametrize('y_true, true_score, labels', [
(np.array([0, 1, 1, 2]), 0.75, [0, 1, 2, 3]),
(np.array([0, 1, 1, 1]), 0.5, [0, 1, 2, 3]),
(np.array([1, 1, 1, 1]), 0.5, [0, 1, 2, 3]),
(np.array(['a', 'e', 'e', 'a']), 0.75, ['a', 'b', 'd', 'e']),
])
@pytest.mark.parametrize("labels_as_ndarray", [True, False])
def test_top_k_accuracy_score_multiclass_with_labels(
y_true, true_score, labels, labels_as_ndarray
):
"""Test when labels and y_score are multiclass."""
if labels_as_ndarray:
labels = np.asarray(labels)
y_score = np.array([
[0.4, 0.3, 0.2, 0.1],
[0.1, 0.3, 0.4, 0.2],
[0.4, 0.1, 0.2, 0.3],
[0.3, 0.2, 0.4, 0.1],
])

score = top_k_accuracy_score(y_true, y_score, k=2, labels=labels)
assert score == pytest.approx(true_score)


def test_top_k_accuracy_score_increasing():
# Make sure increasing k leads to a higher score
X, y = datasets.make_classification(n_classes=10, n_samples=1000,
Expand Down