BUG: always warn when on_bad_lines callable returns extra fields with index_col in read_csv (Python engine) (GH#61837)

[skalwaghe-56]

• closes BUG: read_csv() on_bad_lines callable does not raise ParserWarning when index_col is set #61837 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

---

This PR fixes a regression in the CSV parsers when using on_bad_lines as a callable.

Thanks!

[skalwaghe-56]

@jbrockmendel @rhshadrach If you could please guide me further.

[skalwaghe-56]

@rhshadrach @jorisvandenbossche When I ran the test locally for the changes 1 test xpassed. Related to #10153 I think.
Its this test

@pytest.mark.parametrize("dtype", [{"b": "category"}, {1: "category"}])
def test_categorical_dtype_single(all_parsers, dtype, request):
    # see gh-10153
    parser = all_parsers
    data = """a,b,c
1,a,3.4
1,a,3.4
2,b,4.5"""
    expected = DataFrame(
        {"a": [1, 1, 2], "b": Categorical(["a", "a", "b"]), "c": [3.4, 3.4, 4.5]}
    )
    if parser.engine == "pyarrow":
        mark = pytest.mark.xfail(
            strict=False,
            reason="Flaky test sometimes gives object dtype instead of Categorical",
        )
        request.applymarker(mark)

    actual = parser.read_csv(StringIO(data), dtype=dtype)
    tm.assert_frame_equal(actual, expected)

I would like you guys to check this out and check the PR too!

Thanks!

[skalwaghe-56]

I have fixed the tests too now. The CI should be successful now.

[rhshadrach]

Looking good!

[rhshadrach]

Looks good - I suspect merging main will fix the CI.

[skalwaghe-56]

Looks good - I suspect merging main will fix the CI.

Yep you're right!

[skalwaghe-56]

I guess the errors are still present, will rebase and push once these are all fixed :)

[rhshadrach]

lgtm

[rhshadrach]

Thanks @skalwaghe-56!

[skalwaghe-56]

Thanks!