Unable to apply SFS to a sklearn-wrapped Keras classifier neuralnet

[leowang396]

Hi there, I would really appreciate any help or pointers on this issue.

I'm attempting to leverage on SFS to select the minimal suitable list of features for my 3-class classification neuralnet, and I'm using KerasClassifier as the wrapper.
Currently, I would need some help to figure out why my input data does not seem to reach the first layer of the network in the right shape, which is causing a ValueError: Input 0 of layer sequential is incompatible with the layer: expected axis -1 of input shape to have value 4 but received input with shape [None, 1]. I've tried to address this by aligning input_shape of the first layer to train_input.shape[1], but it still doesn't seem to work.

I looked into the error message as much as I could, and it seems that the width of input is changed to 1 during processing, maybe that's causing the error, but I do not know how to fix it...
(At the start, input has 4 columns)

(After a few steps, the input seems to have only 1 column)

The error is recreated below using iris dataset from sklearn:

from keras.wrappers.scikit_learn import KerasClassifier
from mlxtend.feature_selection import SequentialFeatureSelector as SFS
import sklearn
from sklearn.datasets import load_iris
from tensorflow import keras
iris_train, iris_test = load_iris(return_X_y=True, as_frame=True)

EPOCHS_IRIS = 100
BATCH_SIZE_IRIS = 16

def create_model(train_input):
    # 1 ReLU layer + 1 Dropout layer + 1 softmax layer for 3 classes
    model = keras.Sequential([
        keras.layers.Dense(16,
                           activation='relu',
                           input_shape=((train_input.shape[1]),)),
        keras.layers.Dropout(0.5),
        keras.layers.Dense(3,
                           activation='softmax')
    ])
    
    model.compile(optimizer=keras.optimizers.Adam(lr=1e-3),
                  loss=keras.losses.SparseCategoricalCrossentropy)

    return model

keras.backend.clear_session()
# Wrap Keras nn and generating SFS object
skwrapped_model = KerasClassifier(build_fn=create_model,
                                  train_input=iris_train,
                                  epochs=EPOCHS_IRIS,
                                  batch_size=BATCH_SIZE_IRIS,
                                  validation_split=1-TRAIN_TEST_SPLIT,
                                  verbose=0)
sffs = SFS(skwrapped_model,
           k_features=(1, iris_train.shape[1]),
           floating=True,
           clone_estimator=False,
           cv=0,
           n_jobs=1,
           scoring='accuracy')

# Apply SFS to identify best feature subset
sffs = sffs.fit(iris_train,
                iris_test)

Help is much appreciated, thank u!

[rasbt]

Hi @shengdiWang ,

I was trying to reproduce your Iris example but can't get it to reproduce your error because of a different error:

logs---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-15-e0bffb350382> in <module>
     42 
     43 # Apply SFS to identify best feature subset
---> 44 sffs = sffs.fit(iris_train,
     45                 iris_test)

...


    110     # Running inside `run_distribute_coordinator` already.

~/miniconda3/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1102               end_step = step + data_handler.step_increment
   1103               callbacks.on_train_batch_end(end_step, )
-> 1104         epoch_logs = copy.copy()
   1105 
   1106         # Run validation.

UnboundLocalError: local variable 'logs' referenced before assignmentlogs

Regarding your error,

a)

One hypothesis is that it has something to do with your X_train, x_test inputs to

# Apply SFS to identify best feature subset
sffs = sffs.fit(iris_train,
                iris_test)

because the SFS fit function usually works with 1 dataset, not two. E.g., typical inputs would be

from sklearn.model_selection import train_test_split

dataset = load_iris()
X_train, X_test, y_train, y_test = train_test_split(dataset.data, dataset.target, test_size=0.2)

sfs.fit(X_train, y_train)

b)

The other hypothesis is that keras may return incompatible outputs. I remember there was some reshaping necessary in a PR recently: https://github.com/rasbt/mlxtend/pull/725/files

[rasbt]

Sorry, I am new to GitHub discussions and I think I accidentally deleted parts of it. Trying to reconstruct what I wrote:

I think one of your issues was that you were using

    model.compile(optimizer=keras.optimizers.Adam(lr=1e-3),
                  loss=keras.losses.SparseCategoricalCrossentropy)

instead of

    model.compile(optimizer=keras.optimizers.Adam(lr=1e-3),
                  loss=keras.losses.SparseCategoricalCrossentropy())

The second issue was that the model was created upfront with a fixed size of input features. This is a problem because the sequential feature selector is supposed to try different feature dimensions.

The following modified version might work:

import keras
from keras.wrappers.scikit_learn import KerasClassifier
import mlxtend
from mlxtend.feature_selection import SequentialFeatureSelector as SFS
import sklearn
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from tensorflow import keras

dataset = load_iris()
X_train, X_test, y_train, y_test = train_test_split(dataset.data,
                                                    dataset.target,
                                                    test_size=0.2)


EPOCHS_IRIS = 100
BATCH_SIZE_IRIS = 16
TRAIN_TEST_SPLIT = 0.8

def create_model(train_input):
    # 1 ReLU layer + 1 Dropout layer + 1 softmax layer for 3 classes
    model = keras.Sequential([
        keras.layers.Dense(16,
                           activation='relu',
                           input_shape=(train_input.shape[1],)),
        keras.layers.Dropout(0.5),
        keras.layers.Dense(3,
                           activation='softmax')
    ])
    
    model.compile(optimizer=keras.optimizers.Adam(lr=1e-3),
                  loss=keras.losses.SparseCategoricalCrossentropy())

    return model

keras.backend.clear_session()
# Wrap Keras nn and generating SFS object


def make_model(X_train):
    skwrapped_model = KerasClassifier(build_fn=create_model,
                                      train_input=X_train,
                                      epochs=EPOCHS_IRIS,
                                      batch_size=BATCH_SIZE_IRIS,
                                      validation_split=1-TRAIN_TEST_SPLIT,
                                      verbose=0)
    return skwrapped_model
    

    
class MakeModel(object):

    def __init__(self, X=None, y=None):
        pass

    def predict(self, X):
        y_pred = self.model.predict(X)
        return y_pred
        #return np.argmax(y_pred, axis=1)
    
    def fit(self, X, y):
        skwrapped_model = KerasClassifier(build_fn=create_model,
                                          train_input=X,
                                          epochs=EPOCHS_IRIS,
                                          batch_size=BATCH_SIZE_IRIS,
                                          validation_split=1-TRAIN_TEST_SPLIT,
                                          verbose=0)
        self.model = skwrapped_model
        self.model.fit(X, y)
        return self.model

        

sffs = SFS(MakeModel(),
           k_features=(1, 4),
           floating=True,
           clone_estimator=False,
           cv=0,
           n_jobs=1,
           scoring='accuracy',
           verbose=1)

# Apply SFS to identify best feature subset
sffs = sffs.fit(X_train, y_train)

[leowang396]

Thanks @rasbt ! After applying the two fixes, happy to report that SFS is working well now 👏 My final code is:

import keras
from keras.wrappers.scikit_learn import KerasClassifier
import mlxtend
from mlxtend.feature_selection import SequentialFeatureSelector as SFS
import sklearn
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from tensorflow import keras

dataset = load_iris()
X_train, X_test, y_train, y_test = train_test_split(dataset.data,
                                                    dataset.target,
                                                    test_size=0.2)

EPOCHS_IRIS = 100
BATCH_SIZE_IRIS = 16
TRAIN_TEST_SPLIT = 0.8

def create_model(train_input):
    # 1 ReLU layer + 1 Dropout layer + 1 softmax layer for 3 classes
    model = keras.Sequential([
        keras.layers.Dense(16,
                           activation='relu',
                           input_shape=((train_input.shape[1]),)),
        keras.layers.Dropout(0.5),
        keras.layers.Dense(3,
                           activation='softmax')
    ])
    
    model.compile(optimizer=keras.optimizers.Adam(lr=1e-3),
                  loss=keras.losses.SparseCategoricalCrossentropy())

    return model

keras.backend.clear_session()

# Wrap Keras nn and generating SFS object
class MakeModel(object):

    def __init__(self, X=None, y=None):
        pass

    def predict(self, X):
        y_pred = self.model.predict(X)
        return y_pred
        #return np.argmax(y_pred, axis=1)
    
    def fit(self, X, y):
        skwrapped_model = KerasClassifier(build_fn=create_model,
                                          train_input=X,
                                          epochs=EPOCHS_IRIS,
                                          batch_size=BATCH_SIZE_IRIS,
                                          validation_split=1-TRAIN_TEST_SPLIT,
                                          verbose=0)
        self.model = skwrapped_model
        self.model.fit(X, y)
        return self.model

sffs = SFS(MakeModel(),
           k_features=(1, X_train.shape[1]),
           floating=True,
           clone_estimator=False,
           cv=0,
           n_jobs=1,
           scoring='accuracy')

# Apply SFS to identify best feature subset
sffs = sffs.fit(X_train, y_train)

[rasbt]

Sorry for the late response. Didn't get a notification about this message. But that's awsome to hear. Glad it works now. Actually, I will add this to the documentation later today so it can be used as a reference by others who'd like to use Keras!