Skip to content

Commit

Permalink
update branch and fix conflicts
Browse files Browse the repository at this point in the history
Merge branch 'master' into justinegeffen-edu-474
  • Loading branch information
justinegeffen committed Feb 25, 2025
2 parents 8b23fef + c2cfa7d commit f363fe9
Show file tree
Hide file tree
Showing 21 changed files with 867 additions and 109 deletions.
2 changes: 1 addition & 1 deletion fusion_docs/guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Fusion is a virtual, lightweight, distributed file system designed to optimise t
Fusion enables seamless filesystem I/O to cloud object stores via a standard POSIX interface resulting in simpler pipeline logic and faster, more efficient pipeline execution.

:::note
Fusion requires a license for use beyond limited testing and validation within Seqera Platform or directly within Nextflow. [Contact Seqera](https://seqera.io/contact-us/) for more details.
Fusion requires a license for use in Seqera Platform compute environments or directly in Nextflow. See [Fusion licensing](https://seqera.io/contact-us) for more information.
:::

## Getting started
Expand Down
35 changes: 28 additions & 7 deletions fusion_docs/guide/snapshots.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ More specifically, the first use case for this feature is for Seqera Platform us
Fusion Snapshots v1.0.0 requires the following [Seqera compute environment](https://docs.seqera.io/platform/latest/compute-envs/aws-batch) configuration:

- **Provider**: AWS Batch
- **Pipeline work directory**: An S3 bucket in the same region as the compute environment
- **Pipeline work directory**: An S3 bucket located in the same region as your AWS Batch compute resources
- **Enable Wave containers**
- **Enable Fusion v2**
- **Enable fast instance storage**
Expand All @@ -43,18 +43,39 @@ fusion.containerConfigUrl = '<CUSTOM_CONTAINER_URL>'

`maxSpotAttempts` must be a value higher than `0`.

### Recommended instance sizes
### EC2 instance selection guidelines

Fusion Snapshots require EC2 Spot instances with enough memory and network bandwidth to dump the cache of task intermediate files to S3 storage before AWS terminates an instance. When AWS issues a Spot instance reclamation notice, Fusion has two minutes to complete this transfer.
- Choose EC2 Spot instances with sufficient memory and network bandwidth to dump the cache of task intermediate files to S3 storage before AWS terminates an instance.
- Select instances with guaranteed network bandwidth (not instances with bandwidth "up to" a maximum value).
- Maintain a 5:1 ratio between memory (GiB) and network bandwidth (Gbps).
- Recommended instance families: `c6id`, `r6id`, or `m6id` series instances work optimally with Fusion fast instance storage.

It is recommended to select instances with guaranteed network bandwidth (as opposed to bandwidth _up to_ a maximum value) and maintain a ratio of 5:1 between memory and network bandwidth.
:::info Example
A c6id.8xlarge instance provides 64 GiB memory and 12.5 Gbps guaranteed network bandwidth. This configuration can transfer the entire memory contents to S3 in approximately 70 seconds, well within the 2-minute reclamation window.

For example, taking into account the bandwidth and compute necessary to create a snapshot, a `c6i.8xlarge` instance with 64 GIB memory and a guaranteed network bandwidth of 12.5 Gbps can take approximately 70 seconds to dump the entire instance to S3 storage before instance reclamation occurs.
Instances with memory:bandwitdth ratios over 5:1 may not complete transfers before termination, potentially resulting in task failures.
:::

#### Recommended instance types

| Instance type | Memory (GiB) | Network bandwidth (Gbps) | Memory:bandwidth ratio | Est. Snapshot time|
|----------------|--------------|--------------------------|------------------------|-------------------|
| c6id.4xlarge | 32 | 12.5 | 2.56:1 | ~45 seconds |
| c6id.8xlarge | 64 | 12.5 | 5.12:1 | ~70 seconds |
| r6id.2xlarge | 64 | 12.5 | 5.12:1 | ~70 seconds |
| m6id.4xlarge | 64 | 12.5 | 5.12:1 | ~70 seconds |
| c6id.12xlarge | 96 | 18.75 | 5.12:1 | ~70 seconds |
| r6id.4xlarge | 128 | 12.5 | 10.24:1 | ~105 seconds |
| m6id.8xlarge | 128 | 25 | 5.12:1 | ~70 seconds |

### Amazon Linux 2023 ECS-optimized AMI
### (Seqera Enterprise only) Select an Amazon Linux 2023 ECS-optimized AMI

To obtain sufficient performance, Fusion Snapshots require instances with Amazon Linux 2023 (which ships with Linux Kernel 6.1), with an ECS Container-optimized AMI.

:::note
Selecting a custom Amazon Linux 2023 ECS-optimized AMI is only required for compute environments in Seqera Enterprise deployments. Seqera Cloud AWS Batch compute environments use Amazon Linux 2023 AMIs by default.
:::

To find the recommended AL2023 ECS-optimized AMI for your region, run the following (replace `eu-central-1` with your AWS region):

```bash
Expand All @@ -77,4 +98,4 @@ The result for the `eu-central-1` region is similar to the following:
}
```
Note the `image_id` in your result (in this example, `ami-0281c9a5cd9de63bd`). Specify this ID in the **AMI ID** field under **Advanced options** when you create your Seqera compute environment.
Note the `image_id` in your result (in this example, `ami-0281c9a5cd9de63bd`). Specify this ID in the **AMI ID** field under **Advanced options** when you create your Seqera compute environment.
4 changes: 2 additions & 2 deletions fusion_docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ storage. Fusion enables seamless filesystem I/O to cloud object stores via a sta
simpler pipeline logic and faster, more efficient pipeline execution.

:::note
Fusion requires a license for use in Seqera Platform compute environments or directly in Nextflow. Fusion can be trialed at no cost. [Contact Seqera](https://seqera.io/contact-us/) for more details.
Fusion requires a license for use in Seqera Platform compute environments or directly in Nextflow. See [Fusion licensing](https://seqera.io/contact-us) for more information.
:::

### Transparent, automated installation
Expand Down Expand Up @@ -43,4 +43,4 @@ The result is significant overhead for every task. Fusion enables direct file ac

While some open-source projects provide a POSIX interface over cloud storage, they require developers to install and configure additional software and add packages to containers or VMs.

Unlike third-party solutions, Fusion is optimized for Nextflow and handles these tasks automatically. Fusion delivers fast, seamless access to cloud object storage.
Unlike third-party solutions, Fusion is optimized for Nextflow and handles these tasks automatically. Fusion delivers fast, seamless access to cloud object storage.
27 changes: 27 additions & 0 deletions fusion_docs/licensing.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
title: Fusion licensing
description: "Understand how Fusion licensing works"
date: "20 Feb 2025"
tags: [fusion, storage, compute, file system, license, quotas]
---

:::info
Fusion licensing requirements and free tier quotas will be enforced starting June 30th, 2025. [Contact Seqera](https://seqera.io/contact-us) with any questions or licensing requests.
:::

A Fusion license sets a specific quota for Fusion use, measured by total throughput per month (defined by [Nextflow’s I/O metrics](https://www.nextflow.io/docs/latest/metrics.html#i-o-usage)).

### For Seqera Cloud

- Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform.
- The default quota for free use within Seqera Cloud is 100TB.
- Once your quota is met or exceeded, you will be prevented from running additional Nextflow pipelines with Fusion.

### For Seqera Enterprise instances with connectivity to Seqera’s license server

- Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform.
- Once your quota is met or exceeded, you will be prevented from running additional Nextflow pipelines with Fusion.

### For Seqera Enterprise instances without network connectivity, and direct Fusion use with Nextflow

[Contact Seqera](https://seqera.io/contact-us) for a license that can be integrated into your workflow, or to set up an evaluation.
1 change: 1 addition & 0 deletions fusion_docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
}
]
},
"licensing",
"reference",
"troubleshooting",
"faq",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,37 @@ $ python3

### DNS allowlist

In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:
In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:

- `cloud.seqera.io`
- `api.cloud.seqera.io`
- `user-data.cloud.seqera.io`
- `tower.nf`
- `api.tower.nf`
- `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io`
- `hub.seqera.io`
- `intern.seqera.io`
- `wave.seqera.io`
- `community.wave.seqera.io`
- `cerbero.seqera.io`
- `public.cr.seqera.io`
- `auth.cr.seqera.io`
- `cr.seqera.io`
- `licenses.seqera.io`
- `api.multiqc.info`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `community-cr-prod.seqera.io`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `public-cr-prod.seqera.io`
- `wave-cache-prod-cloudflare.seqera.io`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `community-cr-prod.seqera.io.cdn.cloudflare.net`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `public-cr-prod.seqera.io.cdn.cloudflare.net`
- `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net`

If you chose to filter by specific DNS records, please note that new services may be added in the future.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,37 @@ $ python3

### DNS allowlist

In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:
In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:

- `cloud.seqera.io`
- `api.cloud.seqera.io`
- `user-data.cloud.seqera.io`
- `tower.nf`
- `api.tower.nf`
- `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io`
- `hub.seqera.io`
- `intern.seqera.io`
- `wave.seqera.io`
- `community.wave.seqera.io`
- `cerbero.seqera.io`
- `public.cr.seqera.io`
- `auth.cr.seqera.io`
- `cr.seqera.io`
- `licenses.seqera.io`
- `api.multiqc.info`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `community-cr-prod.seqera.io`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `public-cr-prod.seqera.io`
- `wave-cache-prod-cloudflare.seqera.io`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `community-cr-prod.seqera.io.cdn.cloudflare.net`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `public-cr-prod.seqera.io.cdn.cloudflare.net`
- `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net`

If you chose to filter by specific DNS records, please note that new services may be added in the future.
Expand Down
35 changes: 18 additions & 17 deletions platform_versioned_docs/version-23.4/limits/limits.mdx
Original file line number Diff line number Diff line change
@@ -1,34 +1,35 @@
---
title: "Usage limits"
description: "An overview of Seqera Cloud usage limits"
date: "21 Apr 2023"
date: "19 Feb 2025"
tags: [limits]
---

Seqera Platform elements and features have default limits per workspace and organization.
Seqera Platform elements and features have default limits per organization and workspace.

### Organizations

| Description | Basic | Cloud Pro + Enterprise |
| ----------------------- | ----- | ---------------------- |
| Members | 3 | 50, or per license |
| Workspaces | 50 | 50, or per license |
| Teams | 20 | 20, or per license |
| Run history | 250 | 250, or per license |
| Active runs | 3 | 100, or per license |

### Workspaces

| Description | Free | Cloud Pro + Enterprise |
| ------------ | ---- | ---------------------- |
| Active runs | 5 | 100, or per license |
| Members | 50 | 50, or per license |
| Participants | 50 | 50, or per license |
| Pipelines | 100 | 100, or per license |
| Datasets | 100 | 1000, or per license |
| Labels | 1000 | 1000, or per license |
| Description | Basic | Cloud Pro + Enterprise |
| ------------ | ----- | ---------------------- |
| Participants | 3 | 50, or per license |
| Pipelines | 100 | 100, or per license |
| Datasets | 100 | 1000, or per license |
| Labels | 1000 | 1000, or per license |

:::note
Some Enterprise instances on older licenses are limited to 100 labels per workspace. [Contact support](mailto:support@seqera.io) to upgrade your license.
:::

### Organizations

| Description | Free | Cloud Pro + Enterprise |
| ----------- | ---- | ---------------------- |
| Workspaces | 50 | 50, or per license |
| Teams | 20 | 20, or per license |

### Datasets

| Description | Default limit |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,37 @@ $ python3

### DNS allowlist

In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:
In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following:

- `cloud.seqera.io`
- `api.cloud.seqera.io`
- `user-data.cloud.seqera.io`
- `tower.nf`
- `api.tower.nf`
- `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io`
- `hub.seqera.io`
- `intern.seqera.io`
- `wave.seqera.io`
- `community.wave.seqera.io`
- `cerbero.seqera.io`
- `public.cr.seqera.io`
- `auth.cr.seqera.io`
- `cr.seqera.io`
- `licenses.seqera.io`
- `api.multiqc.info`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `community-cr-prod.seqera.io`
- `fusionfs.seqera.io`
- `nf-xpack.seqera.io`
- `public-cr-prod.seqera.io`
- `wave-cache-prod-cloudflare.seqera.io`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `community-cr-prod.seqera.io.cdn.cloudflare.net`
- `fusionfs.seqera.io.cdn.cloudflare.net`
- `nf-xpack.seqera.io.cdn.cloudflare.net`
- `public-cr-prod.seqera.io.cdn.cloudflare.net`
- `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net`

If you chose to filter by specific DNS records, please note that new services may be added in the future.
Expand Down
42 changes: 19 additions & 23 deletions platform_versioned_docs/version-24.1/limits/limits.mdx
Original file line number Diff line number Diff line change
@@ -1,45 +1,41 @@
---
title: "Usage limits"
description: "An overview of Seqera Cloud usage limits"
date: "21 Apr 2023"
date: "19 Feb 2025"
tags: [limits]
---

Seqera Platform elements and features have default limits per workspace and organization.
Seqera Platform elements and features have default limits per organization and workspace.

### Organizations

| Description | Basic | Cloud Pro + Enterprise |
| ----------------------- | ----- | ---------------------- |
| Members | 3 | 50, or per license |
| Workspaces | 50 | 50, or per license |
| Teams | 20 | 20, or per license |
| Run history | 250 | 250, or per license |
| Active runs | 3 | 100, or per license |
| Running Studio sessions | 1 | 1000, or per license |

### Workspaces

| Description | Free | Cloud Pro + Enterprise |
| ------------ | ---- | ---------------------- |
| Active runs | 5 | 100, or per license |
| Members | 50 | 50, or per license |
| Participants | 50 | 50, or per license |
| Pipelines | 100 | 100, or per license |
| Datasets | 100 | 1000, or per license |
| Labels | 1000 | 1000, or per license |
| Description | Basic | Cloud Pro + Enterprise |
| ------------ | ----- | ---------------------- |
| Participants | 3 | 50, or per license |
| Pipelines | 100 | 100, or per license |
| Datasets | 100 | 1000, or per license |
| Labels | 1000 | 1000, or per license |

:::note
Some Enterprise instances on older licenses are limited to 100 labels per workspace. [Contact support](mailto:support@seqera.io) to upgrade your license.
:::

### Organizations

| Description | Free | Cloud Pro + Enterprise |
| ----------- | ---- | ---------------------- |
| Workspaces | 50 | 50, or per license |
| Teams | 20 | 20, or per license |

### Datasets

| Description | Default limit |
| -------------------- | ------------- |
| File size | 10 MB |
| Versions per dataset | 100 |

### Data Studios

| Description | Free | Cloud Pro + Enterprise |
| ---------------------------- | ---- | ---------------------- |
| Running data studio sessions | 1 | 1000, or per license |

If you need higher limits and capabilities, [contact us](https://seqera.io/contact-us/) to discuss your application requirements.
Loading

0 comments on commit f363fe9

Please sign in to comment.