Singleton log spam on startup with many Peppol IDs

[mirvnillith]

Previously I did a performance test where I registered 100 000 Peppol IDs through the API (without any performance degrading at all, very nice) and now when I start the application it spams the log waiting for the SMPMetaManager singleton:

[2025-01-14T11:50:34,916] [SMP-SERVER] [INFO ] [main] Started running migration to ensure all used transport profiles are automatically created -- com.helger.phoss.smp.domain.SMPMetaManager.lambda$_performMigrations$0(SMPMetaManager.java:154)
[2025-01-14T11:50:35,925] [SMP-SERVER] [WARN ] [main] Singleton 'com.helger.phoss.smp.domain.SMPMetaManager' is not usable - please check your calling order: [SMPMetaManager@0x1926e42e: Status={0}] -- com.helger.scope.singleton.AbstractSingleton.getSingleton(AbstractSingleton.java:564)
...
[2025-01-14T11:51:48,906] [SMP-SERVER] [WARN ] [main] Singleton 'com.helger.phoss.smp.domain.SMPMetaManager' is not usable - please check your calling order: [SMPMetaManager@0x09caa933: Status={0}] -- com.helger.scope.singleton.AbstractSingleton.getSingleton(AbstractSingleton.java:564)
[2025-01-14T11:52:26,663] [SMP-SERVER] [INFO ] [main] Finished performing migration 'ensure-transport-profiles-128' -- com.helger.photon.core.sysmigration.SystemMigrationManager.performMigrationIfNecessary(SystemMigrationManager.java:200)

It took me a while to realize that this was just a slow startup and not an actual failure, but I guess this "migration" takes ~2 minutes for 100 000 registered IDs, every startup. Sounds ominous for us looking at many times that but it looks like there could be a straight-forward SELECT to only load the entities that's actually needed?

Rough SQL from following the relations of SMPMetaManager._performMigrations:

SELECT sm.*
FROM smp_service_metadata sm
JOIN smp_process p
ON p.businessidentifierscheme=sm.businessidentifierscheme
AND p.businessidentifier=sm.businessidentifier
AND p.documentidentifierscheme=sm.documentidentifierscheme
AND p.documentidentifier=sm.documentidentifier
JOIN smp_endpoint ep
ON ep.businessidentifierscheme=p.businessidentifierscheme
AND ep.businessidentifier=p.businessidentifier
AND ep.documentidentifierscheme=p.documentidentifierscheme
AND ep.documentidentifier=p.documentidentifier
AND ep.processidentifiertype=p.processidentifiertype
AND ep.processidentifier=p.processidentifier
LEFT OUTER JOIN smp_tprofile tp
ON tp.id=ep.transportprofile
WHERE tp.id IS NULL

I get that this does not universally apply across backends, but either make this migration persistent/optional or move the actual migration into the backends so that they can optimize if/where possible.

[phax]

I think your problem is related to #298
Eventually you are running the SQL backend as a Docker image and you need a persistent volume for the "system migrations". This will avoid that the system migrations happening over and over again.
I will create an issue out of it

[mirvnillith]

Correct, I’m running an image and would also like to avoid a volume (especially as we’ll eventually be running multiple instances). tis 14 jan. 2025 kl. 14:39 skrev Philip Helger ***@***.***>:

…

I think your problem is related to #298 <#298> Eventually you are running the SQL backend as a Docker image and you need a persistent volume for the "system migrations". This will avoid that the system migrations happening over and over again. I will create an issue out of it — Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOQLOITXYO3HIJVJA35YU32KUHSPAVCNFSM6AAAAABVEZQIJCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOBTGE4DEMI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[phax]

Okay. As a workaround, please use the volume for now. It will be mostly read-only after the initial migrations.