Add support to create index with mapping definition #89
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Anderson Carubelli <acarubelli@gmail.com>
Signed-off-by: Anderson Carubelli <acarubelli@gmail.com>
Signed-off-by: Anderson Carubelli <acarubelli@gmail.com>
Signed-off-by: Anderson Carubelli <acarubelli@gmail.com>
simahawk
reviewed
May 7, 2021
| @@ -88,6 +89,10 @@ def from_crawler(cls, crawler): | |||
|
|
|||
| cls.validate_settings(ext.settings) | |||
| ext.es = cls.init_es_client(crawler.settings) | |||
|
|
|||
| if 'ELASTICSEARCH_INDEX_MAPPING' in ext.settings: | |||
There was a problem hiding this comment.
not an expert on this tool yet but seems a good place to do it (right after ES init)
| logging.debug("index already exists, mapping will not be updated") | ||
| else: | ||
| logging.debug("will create index with mapping definition") | ||
| file = open(mapping_file, "r") |
There was a problem hiding this comment.
use a ctx manager to close the file right after 😉
| ext.es.indices.create( | ||
| index=index_name, ignore=400, body=mapping_data) | ||
|
|
||
| def create_index_name(ext): |
There was a problem hiding this comment.
Not mandatory but I would extrapolate from index_item index name computation and rename this method like _get_index_name.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi!
I'm facing a problem with my current project where I need to create a rotating index and for this, I need to obey some mapping definitions, that's the reason I crating this PR collaborating with this project and also trying to help features requests here and here.
I've decided to use property "ELASTICSEARCH_INDEX_MAPPING" to set file location of mapping definition instead of writing the definition itself inside the "settings.py" file, that's because these definitions tend to grow a lot and can easily pollute the code, eg:
{ "settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 1 }, "analysis": { "analyzer": { "my_analyzer": { "type": "snowball" } } } }, "mappings": { "dynamic": "strict", "properties": { "source": { "type": "text" }, "url": { "type": "text" }, "name": { "type": "text", "analyzer": "my_analyzer", "search_analyzer": "my_analyzer" }, "phone": { "type": "text" }, "photo": { "type": "text" }, "details": { "type": "text", "analyzer": "my_analyzer", "search_analyzer": "my_analyzer" }, "created_at": { "type": "date" } } } }I decided to make it simple and clean just setting the file path of the index definition.
DISCLAIMER: I'm not sure if is the right decision to call the function "create_index_with_mapping" inside the function called "from_crawler". I would love to know your option about that.
Thanks for your project it's helping me a lot and I hope I can help a little bit with it too.
Cheers
Anderson Carubelli