BERT model not working (KeyError: 'sample_weight_mode')

Partner Extensions
1

I trained a set of BERT models about a year ago (in December 2020) and I am only now trying to use one for prediction today. The model reads OK, and the predictor node seems to load the table of sentences (467 rows/sentences) OK, with the progress meter getting to 10%, but then (after a brief pause) it stops with this message:

ERROR BERT Predictor 0:393 Execute failed: ‘sample_weight_mode’
Traceback (most recent call last):
File “”, line 10, in
File “C:\Program Files\KNIME423\plugins\se.redfield.bert_0.0.1.v202012081121\py\BertClassifier.py”, line 155, in run_predict
model = tf.keras.models.load_model(file_store)
File “C:\Users\FirstName.LastName\anaconda3\envs\py3_knime_tf2\lib\site-packages\tensorflow\python\keras\saving\save.py”, line 190, in load_model
return saved_model_load.load(filepath, compile)
File “C:\Users\FirstName.LastName\anaconda3\envs\py3_knime_tf2\lib\site-packages\tensorflow\python\keras\saving\saved_model\load.py”, line 126, in load
training_config))
File “C:\Users\FirstName.LastName\anaconda3\envs\py3_knime_tf2\lib\site-packages\tensorflow\python\keras\saving\saving_utils.py”, line 230, in compile_args_from_training_config
sample_weight_mode = training_config[‘sample_weight_mode’]
KeyError: ‘sample_weight_mode’

I found a similar issue discussed (KeyError: 'sample_weight_mode' · Issue #14040 · keras-team/keras · GitHub) which recommends turning off model compilation, however I don’t know how to do this in Knime, looks like it is not an option.

My Python and TensorFlow/DeepLearning environments are unchanged since a year ago (Python version 3.6.12), the only major change that I can remember is updating Knime itself (from 4.2.3 [I think] to 4.3.3) in May this year.

This failing model was created on 12 Dec 2020, so as an experiment, I tested another of the BERT models that I produced last December, from a few days earlier (7 Dec 2020) and this one DOES work OK (processing 74 sentences)! So I’m wondering why one model works and another (from a few days later) doesn’t. I tried reducing the number of rows from 467 to 74, and then to just one row, but it still fails.

I checked my conda environment in Anaconda (against the list of required versions here: BERT Text Classification for Everyone | KNIME) and the only differences are that I have slightly more recent versions of these three packages, for some reason:

numpy (I have 1.19.4, should be 1.19.1)
tokenizers (0.8.1rc1, should be 0.7.0)
tqdm (4.54.0, should be 4.48.0)

Could these make a difference, or cause such an error? I can’t remember how to revert to earlier versions in Python, but it took me days to get this close to the required versions, so I’m a bit wary of fiddling any more, especially as one of the models has worked.

Any ideas welcome! Would upgrading Knime to the latest version help?

Thanks

Mike

2

Hello @Mick

Could you please downgrade the versions of Python packages to the recommended ones? You can do it with pip command something like this “pip install tokenizers==0.7.0”.

Is it possible for you to share the model so I can try it on my environment?

And the last thing, you can try to use this workflow that has Conda Environment Propagation nodes with the required packages. Just install a new environment and select it in Python Deep Learning settings for TensorFlow2 section.

Please let me know if any of the advice help.

Best,
Artem.

1 Like
3

Thank you Artem. I have managed to get the model working by using a different environment (‘test-pip’) that I created a year ago, at the same time as I built the models using the ‘py3_knime_tf2’ environment. I can’t remember now why I created test-pip or what it is, but I think I was probably trying to side-step using Anaconda, which seemed to create deadlocks and incompatabilities when trying to install the full set of required python packages of the recommended versions. Looking at test-pip’s package versions, it is notable that tensorflow is 2.3.0.

I have tested my ‘py3_knime_tf2’ environment and it is still capable of building models (without error messages). And as I said, one model has worked OK under it, but not this other one that I am currently needing to run. But this latter runs under ‘test-pip’ so the immediate problem is ‘solved’.

But when I get time I will try downgrading those particular packages in ‘py3_knime_tf2’, to see if I can get it to work with this model (and I will also try the Conda ‘create environment’ node, it looks very useful).

Thanks again

Mike

2 Likes
4

I am glad you found a workaround for your issue. Unfortunately package management is a big problem for Python and BERT in particular.

In case it is not sensitive could you please send me the model and list of packages with versions for both environments - where the model and where does not?
This information might be helpful for us to make our node more flexible and avoid similar issues in the future.

Please feel free to get back to this thread in case you have some questions or updates.

Best,
Artem.

1 Like