update from Master #655
Merged
update from Master #655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…toffe BGDIDIC-1596: update field names
…s-hist BGDIDIC-541: adapted sphinx search towards mat view
…verordnung_eisenbahnanlagen BGDIDIC-400: add name index
BGDIDIC-1596: add edkinds field
- BGDIDIC-2712 in kraft - BGDIDIC-2713 in anhoerung
…r-guetertransport BGDIDIC-2709 add search config for:
New Release 2023-12-20-rc1
service container in the infra vhost environment. After a scan for orphaned indexes the container (running in service mode) tried to remove orphaned indexes on EFS which as mounted read-only on the host. in service mode (as backend for search wsgi) the service-sphinxsearch container needs read-only access to the EFS in maintenance mode (triggered by data deploy on geodatasync) the service-sphinxsearch container needs read-write access to the EFS with this change the clean-up of orphaned indexes in the EFS will be done only in maintenance mode, during the sphinx index createion. this will allow us to strictly mount the geodata efs in read-only mode on all our kubernetes or infra vhosts instances.
this will fix an issue that has ocurred during the initial start of the
Fixing index with time
update from master
activate all year from 1850 - 2024 to return results when querying and not only from the most recent year. number_of_communes * num_of_years = 2200 * (2024-1850) ~= 430K entries https://sys-api3.dev.bgdi.ch/rest/services/ech/SearchServer?sr=2056&searchText=bern&lang=de&type=featuresearch&features=ch.swisstopo.swissboundaries3d-gemeinde-flaeche.fill&timeEnabled=true&timeStamps=2024
fix/suppress warnings during the scan of the efs with find
Update Master
swissboundaries3d-gemeinden: Even though there are going to be created a large index, this pr will
needed during the parallel operations of service-search-sphinx on
infra-vhost and k8s.
on k8s we are forced to mount the efs indexes with the full path:
so this will be the folder with the efs index files:
* /var/local/geodata/service-sphinxsearch/${DBSTAGING}/index
this is due to a limitation of the aws efs csi driver which is not
supporting subpaths as volumemount config. we would have to create new
peristant volumes with terraform which is unreasonable for this special
use-case.
PB-82: prepare migration to k8s
BGDIDIC-1804: fix index skitouren
…chuh BGDIDIC-1804: nicer label schneeschuhe
…winter BGDIDIC-2872: add sphinx search index
New Release 2024-11-13-rc1
charachters, that's why we have to auomtaically expand keywords on index generation
the fuzzy string search Text will be built without wildcards by the wsgi application. we have to enable automated wildcard expansion for infix matches in the index configuration
PB-1167: improve fuzzy search index configuration
the limit will be set per docker run --memory parameter, by default 50% of the currently available memory can be used for the index creation.
the table lebensraumkarte.lebensraukarte_schweiz: 64 GB compared with: database solarkataster: 160 GB table solarenergie_daecher: 4 GB table size towards sphinx index size table bfs.gwr_chsdi: 4.71 GB -> sphinx index size: 10 GB table bfs.landschaftswandel: 1.5 MB -> Index size: ~ 5 MB So the index of the lebensraumkarte_schweiz will most probably be > 130 GB That is the reason why we remove the index.
This layer seems to big to calculate an index on it
increase percentage of accessible memory to 70
limit the available memory for the sphinx index creation
as decided with @ltrea we will only enable global cgroup limitations in a first step. if this is working fine a docker run limit is not necessary.
disable docker run memory limitation
New Release 2024-12-18-rc1
some dummy changes for the creation of a new base branch
New Release 2025-02-05-rc1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.