-
Notifications
You must be signed in to change notification settings - Fork 124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add support got-ocr2 #1202
add support got-ocr2 #1202
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
self, model: Union["PreTrainedModel", "TFPreTrainedModel"], model_kwargs: Optional[Dict[str, Any]] = None | ||
): | ||
model_kwargs = model_kwargs or {} | ||
if self._behavior != LlavaConfigBehavior.VISION_EMBEDDINGS: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is VISION_EMBEDDINGS
not added directly to ConfigBehavior
? using LlavaConfigBehavior
looks wrong here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure that I understand your comment, there is no separated config behaviour for this model class. it inherits it from parent class - LllavaOpenVINOConfig. vision embeddings is part of it and its LlavaConfigBehavior. This overriding required to use got-ocr specific patching only for vision_embeddings part instead of llava (because they are different).
if inverted logic looks clearer, I can invert it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
my point is that those behaviors should probably be added to the original ConfigBehavior
class, like decoder/encoder. Having them in LlavaConfigBehavior
makes less sense the more we add multi-modal models.
Anyways this can be addressed in a separate PR if it makes sense for you as well, I will proceed with merging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
every multimodal model may have own partition. it is difficult to maintain everything using only one enum class and will required additional logic for maintenance and validation of model specific structure.
it may be refactored later if you have better solution
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just one question/suggestion to address.
What does this PR do?
Fixes # #918
Before submitting