Use get_max_new_tokens() insted of max_new_tokens field when stopping…

[michalkulakowski]

… generation

[ilya-lavrenov]

Not all places inside src/cpp are changed

[mzegla]

One more thing - I believe max_length is now loaded from generation config. Isn't it a model property that is not meant to be a per generation configuration? @michalkulakowski I know you have logic to read that value in OVMS now. Maybe we could move it here and make it a pipeline member. This way it could be used in both OVMS and standalone GenAI app.

[michalkulakowski]

One more thing - I believe max_length is now loaded from generation config. Isn't it a model property that is not meant to be a per generation configuration? @michalkulakowski I know you have logic to read that value in OVMS now. Maybe we could move it here and make it a pipeline member. This way it could be used in both OVMS and standalone GenAI app.

That makes sense to me. @ilya-lavrenov what do you think?

[ilya-lavrenov]

One more thing - I believe max_length is now loaded from generation config. Isn't it a model property that is not meant to be a per generation configuration? @michalkulakowski I know you have logic to read that value in OVMS now. Maybe we could move it here and make it a pipeline member. This way it could be used in both OVMS and standalone GenAI app.

I suppose it depends on the model:

• For example for meta-llama/Meta-Llama-3-8B-Instruct we have this value in generation config https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/generation_config.json#L6
• For other models like default value for max_length is used, which is 20. See https://github.com/huggingface/transformers/blob/e5fd865ebae062b7cf03a81b8c6affeb39f30bec/src/transformers/generation/configuration_utils.py#L127-L129

Looks like max_model_length (which is config.max_position_embeddings, example is https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/config.json#L13) and max_length from generation_config.json are different things, are not they?

Maybe we can have similar behavior for GenAI and add some defaults similar to HF?

@Wovchena @pavel-esir @as-suvorov what is your opinion?

[michalkulakowski]

@Wovchena @pavel-esir @as-suvorov please share your opinion

[ilya-lavrenov]

@Wovchena @pavel-esir @as-suvorov please share your opinion

I think even w/o default value for max new tokens, we can go with other changes which will respect max_length

[ilya-lavrenov]

build_jenkins

[ilya-lavrenov]

Please, fix compilation

[ilya-lavrenov]

build_jenkins