Skip to content

Commit

Permalink
Add first three migration phases pages
Browse files Browse the repository at this point in the history
Signed-off-by: Archer <naarcha@amazon.com>
  • Loading branch information
Naarcha-AWS committed Nov 29, 2024
1 parent 8cd95c2 commit eaeb2a6
Show file tree
Hide file tree
Showing 13 changed files with 311 additions and 345 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
layout: default
title: Assessing your cluster for migration
nav_order: 60
has_children: true
parent: Migration phases
---

# Assessing your cluster for migration


The goal of the Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster.

## Understanding breaking changes

Before performing any upgrade or migration, you should review any documentation of breaking changes. Even if the cluster is migrated there might be changes required for clients to connect to the new cluster

Check failure on line 16 in _migrations/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/assessing-your-cluster-for-migration.md#L16

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'changes. Even'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'changes. Even'.", "location": {"path": "_migrations/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 16, "column": 93}}}, "severity": "ERROR"}

## Upgrade and breaking changes guides

For migrations paths between Elasticsearch 6.8 and OpenSearch 2.x users should be familiar with documentation in the links below that apply to their specific case:

Check warning on line 20 in _migrations/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/assessing-your-cluster-for-migration.md#L20

[OpenSearch.DirectionAboveBelow] Use 'following or later' instead of 'below' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions.
Raw output
{"message": "[OpenSearch.DirectionAboveBelow] Use 'following or later' instead of 'below' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions.", "location": {"path": "_migrations/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 20, "column": 124}}}, "severity": "WARNING"}

* [Upgrading Amazon Service Domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html)

* [Changes from Elasticsearch to OpenSearch fork](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html)

* [OpenSearch Breaking Changes](https://opensearch.org/docs/latest/breaking-changes/)

The next step is to set up a proper test bed to verify that your applications will work as expected on the target version.

## Data Transformations and Client Impact

Check failure on line 30 in _migrations/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/assessing-your-cluster-for-migration.md#L30

[OpenSearch.HeadingCapitalization] 'Data Transformations and Client Impact' is a heading and should be in sentence case.
Raw output
{"message": "[OpenSearch.HeadingCapitalization] 'Data Transformations and Client Impact' is a heading and should be in sentence case.", "location": {"path": "_migrations/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 30, "column": 4}}}, "severity": "ERROR"}

Any time you apply a transformation to your data, such as:

- Changing index names
- Modifying field names or field mappings
- Splitting indices with type mappings

Check failure on line 36 in _migrations/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/assessing-your-cluster-for-migration.md#L36

[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.
Raw output
{"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_migrations/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 36, "column": 13}}}, "severity": "ERROR"}

These changes might need to be reflected in your client configurations. For example, if your clients are reliant on specific index or field names, you must ensure that their queries are updated accordingly.

We recommend running production-like queries against the target cluster before switching over actual production traffic. This helps verify that the client can:

- Communicate with the target cluster
- Locate the necessary indices and fields

Check failure on line 43 in _migrations/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/assessing-your-cluster-for-migration.md#L43

[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.
Raw output
{"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_migrations/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 43, "column": 24}}}, "severity": "ERROR"}
- Retrieve the expected results

For complex migrations involving multiple transformations or breaking changes, we highly recommend performing a trial migration with representative, non-production data (e.g., in a staging environment) to fully test client compatibility with the target cluster.



7 changes: 0 additions & 7 deletions _migrations/migration-phases/assessment/index.md

This file was deleted.

41 changes: 0 additions & 41 deletions _migrations/migration-phases/assessment/required-client-changes.md

This file was deleted.

This file was deleted.

56 changes: 0 additions & 56 deletions _migrations/migration-phases/metadata/Snapshot-Creation.md

This file was deleted.

7 changes: 0 additions & 7 deletions _migrations/migration-phases/metadata/index.md

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,17 +1,59 @@
---
layout: default
title: Metadata migration
title: Migrating metadata
nav_order: 85
parent: Metadata
grand_parent: Migration phases
parent: Migration phases
---

# Metadata migration
# Migrating metadata

Metadata migration is a relatively fast process to execute so we recommend attempting this workflow as early as possible to discover any issues which could impact longer running migration steps.
Metadata migration involves creating a snapshot of you cluster than migrating the metadata from the snapshot using the Migration Console.

## Prerequisites
A snapshot of the cluster state will need to be taken, [[guide to create a snapshot|Snapshot Creation]].

This tool gathers information from a source cluster, through a snapshot or through HTTP requests against the source cluster. These snapshots are fully compatible with the backfill process for Reindex-From-Snapshot (RFS) scenarios.

Check failure on line 13 in _migrations/migration-phases/migrating-metadata.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/migrating-metadata.md#L13

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'cluster. These'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'cluster. These'.", "location": {"path": "_migrations/migration-phases/migrating-metadata.md", "range": {"start": {"line": 13, "column": 117}}}, "severity": "ERROR"}

After collecting information on the source cluster comparisons are made on the target cluster. If running a migration, any metadata items do not already exist will be created on the target cluster.

Check failure on line 15 in _migrations/migration-phases/migrating-metadata.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/migrating-metadata.md#L15

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'cluster. If'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'cluster. If'.", "location": {"path": "_migrations/migration-phases/migrating-metadata.md", "range": {"start": {"line": 15, "column": 87}}}, "severity": "ERROR"}

## Creating the snapshot

Creating a snapshot of the source cluster capture all the metadata and documents to migrate onto a new target cluster.

Create the initial snapshot on the source cluster with the following command:

```shell
console snapshot create
```

To check the progress of the snapshot in real-time, use the following command:

```shell
console snapshot status --deep-check
```

You should receive the following response when the snapshot is created:

```shell
SUCCESS
Snapshot is SUCCESS.
Percent completed: 100.00%
Data GiB done: 29.211/29.211
Total shards: 40
Successful shards: 40
Failed shards: 0
Start time: 2024-07-22 18:21:42
Duration: 0h 13m 4s
Anticipated duration remaining: 0h 0m 0s
Throughput: 38.13 MiB/sec
```


### Limitations

Incremental or "delta" snapshots are not yet supported. For more information, refer to the [tracking issue MIGRATIONS-1624](https://opensearch.atlassian.net/browse/MIGRATIONS-1624). A single snapshot must be used for a backfill.

### Dealing with slow snapshot speeds

Depending on the size of the data on the source cluster and the bandwidth allocated for snapshots, the snapshot creation process can take some time. You can adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. If not specified, the default rate for the source cluster's version will be used.

## Command Arguments

Expand Down Expand Up @@ -47,22 +89,19 @@ INFO:console_link.models.metadata:Migrating metadata with command: /root/metadat
.
.
```
</details>

## Metadata verification with evaluate command

## Using the `evaluate` commonad

By scanning the contents of the source cluster, applying filtering, and applying modifications a list of all items that will be migrated will be created. Any items not seen in this output will not be migrated onto the target cluster if the migrate command was to be run. This is a safety check before making modifications on the target cluster.

```shell
console metadata evaluate [...]
```

<details>
<summary>
<b>Example evaluate command output<b>
</summary>
You should receive a response similar to the following:

```
```bash
Starting Metadata Evaluation
Clusters:
Source:
Expand All @@ -89,20 +128,17 @@ Migration Candidates:
Results:
0 issue(s) detected
```
</details>

## Metadata migration with migrate command

## Using the migrate command

Running through the same data as the evaluate command all of the migrated items will be applied onto the target cluster. If re-run multiple times items that were previously migrated will not be recreated. If any items do need to be re-migrated, please delete them from the target cluster and then rerun the evaluate then migrate commands to ensure the desired changes are made.

```shell
console metadata migrate [...]
```

<details>
<summary>
<b>Example migrate command output</b>
</summary>
You should receive a response similar to the following:

```
Starting Metadata Migration
Expand Down Expand Up @@ -138,9 +174,12 @@ Results:

Before moving on to additional migration steps, it is recommended to confirm details of your cluster. Depending on your configuration, this could be checking the sharding strategy or making sure index mappings are correctly defined by ingesting a test document.

## Troubleshoot issues
## Troubleshooting

Use these instructions to help troubleshoot the following issues.

### Access detailed logs

### Access Detailed Logs
Metadata migration creates a detailed log file that includes low level tracing information for troubleshooting. For each execution of the program a log file is created inside a shared volume on the migration console named `shared-logs-output` the following command will list all log files, one for each run of the command.

```shell
Expand All @@ -153,23 +192,20 @@ To inspect the file within the console `cat`, `tail` and `grep` commands line to
tail /shared-logs-output/migration-console-default/*/metadata/*.log
```

### Warnings / Errors inline
### Warnings/errors inline

Check failure on line 195 in _migrations/migration-phases/migrating-metadata.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/migrating-metadata.md#L195

[OpenSearch.HeadingCapitalization] 'Warnings/errors inline' is a heading and should be in sentence case.
Raw output
{"message": "[OpenSearch.HeadingCapitalization] 'Warnings/errors inline' is a heading and should be in sentence case.", "location": {"path": "_migrations/migration-phases/migrating-metadata.md", "range": {"start": {"line": 195, "column": 5}}}, "severity": "ERROR"}

There might be `WARN` or `ERROR` elements inline the output, they will be accompanied by a short message, such as `WARN - my_index already exists`. Full information will be in the detailed logs associated with this warnings or errors.

### OpenSearch running in compatibility mode
There might be an error about being unable to update an ES 7.10.2 cluster, this can occur when compatibility mode has been enabled on an OpenSearch cluster please disable it to continue, see [Enable compatibility mode](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html#rename-upgrade) ↗.

## How the tool works

This tool gathers information from a source cluster, through a snapshot or through HTTP requests against the source cluster. These snapshots are fully compatible with the backfill process for Reindex-From-Snapshot (RFS) scenarios, [[learn more|Backfill-Execution]].

After collecting information on the source cluster comparisons are made on the target cluster. If running a migration, any metadata items do not already exist will be created on the target cluster.

### Breaking change compatibility

Metadata migration needs to modify data from the source to the target versions to recreate items. Sometimes these features are no longer supported and have been removed from the target version. Sometimes these features are not available on the target version, which is especially true when downgrading. While this tool is meant to make this process easier, it is not exhaustive in its support. When encountering a compatibility issue or an important feature gap for your migration, please [search the issues](https://github.com/opensearch-project/opensearch-migrations/issues) and comment + upvote or a [create a new](https://github.com/opensearch-project/opensearch-migrations/issues/new/choose) issue if one cannot be found.
Metadata migration needs to modify data from the source to the target versions to recreate items. Sometimes these features are no longer supported and have been removed from the target version. Sometimes these features are not available on the target version, which is especially true when downgrading. While this tool is meant to make this process easier, it is not exhaustive in its support. When encountering a compatibility issue or an important feature gap for your migration, [search the issues](https://github.com/opensearch-project/opensearch-migrations/issues) or [create a new](https://github.com/opensearch-project/opensearch-migrations/issues/new/choose) issue if one cannot be found.

Check failure on line 205 in _migrations/migration-phases/migrating-metadata.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/migrating-metadata.md#L205

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'items. Sometimes'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'items. Sometimes'.", "location": {"path": "_migrations/migration-phases/migrating-metadata.md", "range": {"start": {"line": 205, "column": 92}}}, "severity": "ERROR"}

Check failure on line 205 in _migrations/migration-phases/migrating-metadata.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/migrating-metadata.md#L205

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'version. Sometimes'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'version. Sometimes'.", "location": {"path": "_migrations/migration-phases/migrating-metadata.md", "range": {"start": {"line": 205, "column": 187}}}, "severity": "ERROR"}

Check failure on line 205 in _migrations/migration-phases/migrating-metadata.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/migrating-metadata.md#L205

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'downgrading. While'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'downgrading. While'.", "location": {"path": "_migrations/migration-phases/migrating-metadata.md", "range": {"start": {"line": 205, "column": 293}}}, "severity": "ERROR"}

Check failure on line 205 in _migrations/migration-phases/migrating-metadata.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-phases/migrating-metadata.md#L205

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'support. When'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'support. When'.", "location": {"path": "_migrations/migration-phases/migrating-metadata.md", "range": {"start": {"line": 205, "column": 389}}}, "severity": "ERROR"}

#### Deprecation of Mapping Types

In Elasticsearch 6.8 the mapping types feature was discontinued in Elasticsearch 7.0+ which has created complexity in migrating to newer versions of Elasticsearch and OpenSearch, [learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html) ↗.

As Metadata migration supports migrating from ES 6.8 on to the latest versions of OpenSearch this scenario is handled by removing the type mapping types and restructuring the template or index properties. Note that, at the time of this writing multiple type mappings are not supported, [tracking task](https://opensearch.atlassian.net/browse/MIGRATIONS-1778) ↗.
Expand Down Expand Up @@ -203,4 +239,4 @@ As Metadata migration supports migrating from ES 6.8 on to the latest versions o
}
```

*Technical details are available, [view source code](https://github.com/opensearch-project/opensearch-migrations/blob/main/transformation/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java).*
For additional technical details, [view the mapping type removal source code](https://github.com/opensearch-project/opensearch-migrations/blob/main/transformation/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java).

This file was deleted.

Loading

0 comments on commit eaeb2a6

Please sign in to comment.