feat(matrix): speed up query to refresh index

Author: bethesque

DEVELOPER_DOCUMENTATION.md

@@ -58,4 +58,31 @@ Domain classes are found in `lib/pact_broker/domain`. Many of these classes are
 
 * `matrix` - The matrix of every pact publication and verification. Includes every pact revision (eg. publishing to the same consumer version twice, or using PATCH) and every verification (including 'overwritten' ones. eg. when the same provider build runs twice.)
 
-* `latest_matrix` - This view is a subset of, and has the same columns as, the `matrix`. It removes 'overwritten' pacts and verifications from the matrix (ie. only show latest pact revision for each consumer version and latest verification for each provider version)
+* `latest_matrix_for_consumer_version_and_provider_version` - This view is a subset of, and has the same columns as, the `matrix`. It removes 'overwritten' pacts and verifications from the matrix (ie. only show latest pact revision for each consumer version and latest verification for each provider version)
+
+### Materialized Views
+
+We can't use proper materialized views because we have to support MySQL :|
+
+So as a hacky solution, there are two tables which act as materialized views to speed up the performance of the matrix and index queries. These tables are updated after any resource is published with a `consumer_name`, `provider_name` or `pacticipant_name` in the URL (see lib/pact_broker/api/resources/base_resource.rb#finish_request).
+
+* `materialized_matrix` table - is populated from the `matrix` view.
+
+* `materialized_head_matrix` table - is populated from `head_matrix` view, and is based on `materialized_matrix`.
+
+### Dependencies
+
+materialized_head_matrix table (is populated from...)
+  = head_matrix view
+    -> latest_matrix_for_consumer_version_and_provider_version view
+      -> materialized_matrix table (is populated from...)
+        = matrix view
+          -> verifications table
+          -> versions table
+          -> all_pact_publications view
+            -> pact_versions table
+            -> pact_publications table
+            -> pacticipants table
+            -> versions table
+      -> latest_verification_id_for_consumer_version_and_provider_version view
+      -> latest_pact_publication_revision_numbers view

db/migrations/20180210_fix_latest_matrix_for_cv_and_pv_again.rb

@@ -8,10 +8,21 @@
     # lose the unverified pacts.
     # This view used to be (stupidly) called latest_matrix
 
-    # Fixes mistakenly copied definition in 20180209_recreate_latest_matrix_for_cv_and_pv_union_all.rb
-    # which missed the join to latest_pact_publication_revision_numbers
+    # Fix mistakenly copied definition in 20180209_recreate_latest_matrix_for_cv_and_pv_union_all.rb
+    # which missed the join to latest_pact_publication_revision_numbers.
+
+    # Change this view to be based on materialized_matrix instead of matrix
+    # to speed it up.
+    # Note! This does mean there is a dependency on having updated
+    # materialized_matrix FIRST that may cause problems. Will see how it goes.
+
+    alter_table(:materialized_matrix) do
+      add_index [:verification_id], name: 'ndx_mm_verif_id'
+      add_index [:pact_revision_number], name: 'ndx_mm_pact_rev_num'
+    end
+
     create_or_replace_view(:latest_matrix_for_consumer_version_and_provider_version,
-      "SELECT matrix.* FROM matrix
+      "SELECT matrix.* FROM materialized_matrix matrix
       inner join latest_pact_publication_revision_numbers lr
       on matrix.consumer_id = lr.consumer_id
       and matrix.provider_id = lr.provider_id
@@ -24,7 +35,7 @@
 
       UNION ALL
 
-      select matrix.* from matrix
+      select matrix.* from materialized_matrix matrix
       inner join latest_pact_publication_revision_numbers lr
       on matrix.consumer_id = lr.consumer_id
       and matrix.provider_id = lr.provider_id