GridSearch for XGBoost

Good evening,
I’m training a XGBoost model for a regression problem.
I need to do a Grid Search to find the best hyperparameters of the model.
In Knime I’ve only found the “parameter optimization loop start” node which allows to change the parameter using a fixed step.
In my case, most of the values for the hyperparameters don’t have a fixed distance, for example I want to try the Gamma parameter with values of {0.01, 0.1, 3, 7}.

How can I do that?
Thanks in advance!

Welcome to the KNIME forum @AxelLexa,
if you want to specify the exact values you can always create a table holding the configuration you want to try with the Table Creator followed by the Table Row to Variable Loop Start.
In order to get the best configuration, just train and score your model within the loop and collect the scores with the Variable Loop End. Once the loop is completed you can select the best configuration with the Top k Selector and train the final model.

Cheers,
Adrian

3 Likes

Hello Nemad and thanks for the fast reply.
If I have 5 different parameter to tune using the gridsearch, how can I do it without innesting 5 loops?
I have to try all the possible combinations of the 5 parameters, I don’t want to tune only one single parameter at each time.

Thank you,
Axel

Hi Axel,
you can create an individual table for each parameter you want to optimize and then create the configurations table by using multiple Cross Joiner nodes.
The resulting table can then be used in a similar way as I described above.

Cheers,
Adrian

2 Likes

Why do you know these are good values? why not 0.5?

The optimization node also has options for strategies how to do the optimization. I suggest to use bayesian based option (TPE). With that you can also choose to ignore step size and hence only the boundaries you define will matter. The node description explains more how this works.

Note: Always use the same CV split to evaluate your parameter setting. This split can be per-genereated which is much faster than using a fix seed CV loop.

2 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.