Fix package dependency issues

[ChristopherGS]

Update: For students checking this PR, the fixes have now been applied to the commit history. Please update your local copy from the current master branch

Fix for the Unit Testing a Production ML Model section

[ChristopherGS]

@solegalli previously I was able to use the _fit method to access the transformed dataframe of the pipeline. This method seems to have been updated (possibly here: scikit-learn/scikit-learn@88ce8cd) and is throwing an error.

Any idea how to access the transformed dataframe for the GradientBoostingRegressor (since it doesn't have a transform method?)

[solegalli]

Not straightforward. Here some potential solutions:

https://stackoverflow.com/questions/54332654/how-to-analyse-the-intermediate-steps-of-sklearn-pipeline
https://stackoverflow.com/questions/48743032/get-intermediate-data-state-in-scikit-learn-pipeline?

In short, I think you need to apply all the transformations one after the other, just up to the one before the model.

[solegalli]

Ok, I asked the sklearn community through the email list and one of the developers told me to do this:

Using slicing: model[:-1].transform(X)

[ChristopherGSTest]

That worked perfectly :)

from gradient_boosting_model import pipeline
from gradient_boosting_model.config.core import config

from functools import reduce


def test_pipeline_drops_unnecessary_features(pipeline_inputs):
    # Given
    X_train, X_test, y_train, y_test = pipeline_inputs
    assert config.model_config.drop_features in X_train.columns
    pipeline.price_pipe.fit(X_train, y_train)

    # When
    # We access the transformed inputs with slicing
    transformed_inputs = pipeline.price_pipe[:-1].transform(X_train)

    # Then
    assert config.model_config.drop_features in X_train.columns
    assert config.model_config.drop_features not in transformed_inputs.columns


def test_pipeline_transforms_temporal_features(pipeline_inputs):
    # Given
    X_train, X_test, y_train, y_test = pipeline_inputs

    # When
    # We access the transformed inputs with slicing
    transformed_inputs = pipeline.price_pipe[:-1].transform(X_train)

    # Then
    assert (
        transformed_inputs.iloc[0]["YearRemodAdd"]
        == X_train.iloc[0]["YrSold"] - X_train.iloc[0]["YearRemodAdd"]
    )

[ChristopherGS]

Key Section 5 fix

[ChristopherGS]

Key section 4 fix 1

[ChristopherGS]

Key section 4 fix 2

[ChristopherGS]

key fix for section 9

[ChristopherGS]

Key fix for section 9 (2/2)

Note required new publishing of gradient boosting model (so sklearn versions are compatible)