Skip to content

Commit

Permalink
Merge branch 'opensearch-project:main' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
@rishabh6788
rishabh6788 authored Sep 17, 2024
2 parents 2beea05 + 379611d commit 3e38f4a
Show file tree
Hide file tree
Showing 6 changed files with 309 additions and 87 deletions.
24 changes: 23 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ In order to deploy both the stacks the user needs to provide a set of required a

| Name | Requirement | Type | Description |
|-------------------------------|:------------|:----------|:------------|
| contextKey | Optional | string | A top-level key that the rest of the parameters can be nested within for organized configuration. This is used to parse config from a specific key in the `cdk.context.json` or in the `context` block in the `cdk.json` file. |
| distVersion | Required | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| securityDisabled | Required | boolean | Enable or disable security plugin |
| adminPassword | Optionally required | string | This value is required when security plugin is enabled and the cluster version is greater or equal to `2.12.0`|
Expand Down Expand Up @@ -78,6 +79,7 @@ In order to deploy both the stacks the user needs to provide a set of required a
| certificateArn | Optional | string | Add ACM certificate to the any listener (OpenSearch or OpenSearch-Dashboards) whose port is mapped to 443. e.g., `--context certificateArn=arn:1234`|
| mapOpensearchPortTo | Optional | integer | Load balancer port number to map to OpenSearch. e.g., `--context mapOpensearchPortTo=8440` Defaults to 80 when security is disabled and 443 when security is enabled |
| mapOpensearchDashboardsPortTo | Optional | integer | Load balancer port number to map to OpenSearch-Dashboards. e.g., `--context mapOpensearchDashboardsPortTo=443` Always defaults to 8443 |
| loadBalancerType | Optional | string | The type of load balancer to deploy. Valid values are nlb for Network Load Balancer or alb for Application Load Balancer. Defaults to nlb. e.g., `--context loadBalancerType=alb` |

* Before starting this step, ensure that your AWS CLI is correctly configured with access credentials.
* Also ensure that you're running these commands in the current directory
Expand Down Expand Up @@ -108,6 +110,26 @@ cdk synth "*" --context securityDisabled=false \
--context distVersion=2.3.0 --context serverAccessType=ipv4 --context restrictServerAccessTo=10.10.10.10/32
```

Alternatively, you can use the `contextKey` to provide the configuration.

For example, to synthesize the CloudFormation templates using a context key:
```sh
cdk synth "*" --context contextKey=devConfig
```
You would include the configuration in your `cdk.json` file like this:
```js
// cdk.json
{
"context": {
"devConfig": {
"securityDisabled": false,
// ...
}
}
}
```
This approach allows you to manage multiple configurations easily by defining different context keys for each environment.

#### Sample command to set up multi-node cluster with security enabled on x64 AL2 machine

Please note that as of now we only support instances backed by Amazon Linux-2 amis.
Expand Down Expand Up @@ -169,7 +191,7 @@ All the ec2 instances are hosted in private subnet and can only be accessed usin

## Port Mapping

The ports to access the cluster are dependent on the `security` parameter value
The ports to access the cluster are dependent on the `security` parameter value and are identical whether using an Application Load Balancer (ALB) or a Network Load Balancer (NLB):
* If `security` is `disable` (HTTP),
* OpenSearch 9200 is mapped to port 80 on the LB
* If `security` is `enable` (HTTPS),
Expand Down
11 changes: 11 additions & 0 deletions bin/app.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,17 @@ import { InfraStack } from '../lib/infra/infra-stack';
import { NetworkStack } from '../lib/networking/vpc-stack';

const app = new App();

const contextKey = app.node.tryGetContext('contextKey');
if (contextKey) {
const nestedContext = app.node.tryGetContext(contextKey);
if (nestedContext && typeof nestedContext === 'object') {
Object.entries(nestedContext).forEach(([nestedKey, nestedValue]) => {
app.node.setContext(nestedKey, nestedValue);
});
}
}

const region = app.node.tryGetContext('region') ?? process.env.CDK_DEFAULT_REGION;
const account = app.node.tryGetContext('account') ?? process.env.CDK_DEFAULT_ACCOUNT;

Expand Down
216 changes: 159 additions & 57 deletions lib/infra/infra-stack.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@ import {
SubnetType,
} from 'aws-cdk-lib/aws-ec2';
import {
ApplicationListener,
ApplicationLoadBalancer,
ApplicationProtocol,
BaseApplicationListenerProps,
BaseListener,
BaseLoadBalancer,
BaseNetworkListenerProps,
ListenerCertificate,
NetworkListener, NetworkLoadBalancer, Protocol,
} from 'aws-cdk-lib/aws-elasticloadbalancingv2';
Expand All @@ -55,6 +62,11 @@ enum cpuArchEnum{
ARM64='arm64'
}

export enum LoadBalancerType {
NLB = 'nlb',
ALB = 'alb'
}

const getInstanceType = (instanceType: string, arch: string) => {
if (arch === 'x64') {
if (instanceType !== 'undefined') {
Expand Down Expand Up @@ -133,10 +145,13 @@ export interface InfraProps extends StackProps {
readonly mapOpensearchPortTo ?: number
/** Map opensearch-dashboards port on load balancer to */
readonly mapOpensearchDashboardsPortTo ?: number
/** Type of load balancer to use (e.g., 'nlb' or 'alb') */
readonly loadBalancerType?: LoadBalancerType
}

export class InfraStack extends Stack {
public readonly nlb: NetworkLoadBalancer;
public readonly elb: NetworkLoadBalancer | ApplicationLoadBalancer;
public readonly elbType: LoadBalancerType;

private instanceRole: Role;

Expand Down Expand Up @@ -200,8 +215,6 @@ export class InfraStack extends Stack {

constructor(scope: Stack, id: string, props: InfraProps) {
super(scope, id, props);
let opensearchListener: NetworkListener;
let dashboardsListener: NetworkListener;
let managerAsgCapacity: number;
let dataAsgCapacity: number;
let clientNodeAsg: AutoScalingGroup;
Expand Down Expand Up @@ -398,11 +411,28 @@ export class InfraStack extends Stack {

const certificateArn = `${props?.certificateArn ?? scope.node.tryGetContext('certificateArn')}`;

this.nlb = new NetworkLoadBalancer(this, 'clusterNlb', {
vpc: props.vpc,
internetFacing: (!this.isInternal),
crossZoneEnabled: true,
});
// Set the load balancer type, defaulting to NLB if not specified
const loadBalancerTypeStr = scope.node.tryGetContext('loadBalancerType') ?? 'nlb'
this.elbType = props?.loadBalancerType ?? LoadBalancerType[(loadBalancerTypeStr).toUpperCase() as keyof typeof LoadBalancerType];
switch (this.elbType) {
case LoadBalancerType.NLB:
this.elb = new NetworkLoadBalancer(this, 'clusterNlb', {
vpc: props.vpc,
internetFacing: (!this.isInternal),
crossZoneEnabled: true,
});
break;
case LoadBalancerType.ALB:
this.elb = new ApplicationLoadBalancer(this, 'clusterAlb', {
vpc: props.vpc,
internetFacing: (!this.isInternal),
crossZoneEnabled: true,
securityGroup: props.securityGroup,
});
break;
default:
throw new Error('Invalid load balancer type provided. Valid values are ' + Object.values(LoadBalancerType).join(', '));
}

const opensearchPortMap = `${props?.mapOpensearchPortTo ?? scope.node.tryGetContext('mapOpensearchPortTo')}`;
const opensearchDashboardsPortMap = `${props?.mapOpensearchDashboardsPortTo ?? scope.node.tryGetContext('mapOpensearchDashboardsPortTo')}`;
Expand All @@ -428,34 +458,27 @@ export class InfraStack extends Stack {
+ ` Current mapping is OpenSearch:${this.opensearchPortMapping} OpenSearch-Dashboards:${this.opensearchDashboardsPortMapping}`);
}

if (!this.securityDisabled && !this.minDistribution && this.opensearchPortMapping === 443 && certificateArn !== 'undefined') {
opensearchListener = this.nlb.addListener('opensearch', {
port: this.opensearchPortMapping,
protocol: Protocol.TLS,
certificates: [ListenerCertificate.fromArn(certificateArn)],
});
} else {
opensearchListener = this.nlb.addListener('opensearch', {
port: this.opensearchPortMapping,
protocol: Protocol.TCP,
});
}
const useSSLOpensearchListener = !this.securityDisabled && !this.minDistribution && this.opensearchPortMapping === 443 && certificateArn !== 'undefined';
const opensearchListener = InfraStack.createListener(
this.elb,
this.elbType,
'opensearch',
this.opensearchPortMapping,
(useSSLOpensearchListener) ? certificateArn : undefined
);

let dashboardsListener: NetworkListener | ApplicationListener;
if (this.dashboardsUrl !== 'undefined') {
if (!this.securityDisabled && !this.minDistribution && this.opensearchDashboardsPortMapping === 443 && certificateArn !== 'undefined') {
dashboardsListener = this.nlb.addListener('dashboards', {
port: this.opensearchDashboardsPortMapping,
protocol: Protocol.TLS,
certificates: [ListenerCertificate.fromArn(certificateArn)],
});
} else {
dashboardsListener = this.nlb.addListener('dashboards', {
port: this.opensearchDashboardsPortMapping,
protocol: Protocol.TCP,
});
}
const useSSLDashboardsListener = !this.securityDisabled && !this.minDistribution
&& this.opensearchDashboardsPortMapping === 443 && certificateArn !== 'undefined';
dashboardsListener = InfraStack.createListener(
this.elb,
this.elbType,
'dashboards',
this.opensearchDashboardsPortMapping,
(useSSLDashboardsListener) ? certificateArn : undefined
);
}

if (this.singleNodeCluster) {
console.log('Single node value is true, creating single node configurations');
singleNodeInstance = new Instance(this, 'single-node-instance', {
Expand Down Expand Up @@ -483,19 +506,23 @@ export class InfraStack extends Stack {
});
Tags.of(singleNodeInstance).add('role', 'client');

opensearchListener.addTargets('single-node-target', {
port: 9200,
protocol: Protocol.TCP,
targets: [new InstanceTarget(singleNodeInstance)],
});
// Disable target security for now, can be provided as an option in the future
InfraStack.addTargetsToListener(
opensearchListener,
this.elbType,
'single-node-target',
9200,
new InstanceTarget(singleNodeInstance),
false);

if (this.dashboardsUrl !== 'undefined') {
// @ts-ignore
dashboardsListener.addTargets('single-node-osd-target', {
port: 5601,
protocol: Protocol.TCP,
targets: [new InstanceTarget(singleNodeInstance)],
});
InfraStack.addTargetsToListener(
dashboardsListener!,
this.elbType,
'single-node-osd-target',
5601,
new InstanceTarget(singleNodeInstance),
false);
}
new CfnOutput(this, 'private-ip', {
value: singleNodeInstance.instancePrivateIp,
Expand Down Expand Up @@ -625,7 +652,7 @@ export class InfraStack extends Stack {
requireImdsv2: true,
signals: Signals.waitForAll(),
});
Tags.of(clientNodeAsg).add('cluster', scope.stackName);
Tags.of(clientNodeAsg).add('cluster', this.stackName);
}

Tags.of(clientNodeAsg).add('role', 'client');
Expand Down Expand Up @@ -660,23 +687,27 @@ export class InfraStack extends Stack {
Tags.of(mlNodeAsg).add('role', 'ml-node');
}

opensearchListener.addTargets('opensearchTarget', {
port: 9200,
protocol: Protocol.TCP,
targets: [clientNodeAsg],
});
// Disable target security for now, can be provided as an option in the future
InfraStack.addTargetsToListener(
opensearchListener,
this.elbType,
'opensearchTarget',
9200,
clientNodeAsg,
false);

if (this.dashboardsUrl !== 'undefined') {
// @ts-ignore
dashboardsListener.addTargets('dashboardsTarget', {
port: 5601,
protocol: Protocol.TCP,
targets: [clientNodeAsg],
});
InfraStack.addTargetsToListener(
dashboardsListener!,
this.elbType,
'dashboardsTarget',
5601,
clientNodeAsg,
false);
}
}
new CfnOutput(this, 'loadbalancer-url', {
value: this.nlb.loadBalancerDnsName,
value: this.elb.loadBalancerDnsName,
});

if (this.enableMonitoring) {
Expand Down Expand Up @@ -1013,4 +1044,75 @@ export class InfraStack extends Stack {

return cfnInitConfig;
}

/**
* Creates a listener for the given load balancer.
* If a certificate is provided, the protocol will be set to TLS/HTTPS.
* Otherwise, the protocol will be set to TCP/HTTP.
*/
private static createListener(elb: BaseLoadBalancer, elbType: LoadBalancerType, id: string, port: number,
certificateArn?: string): ApplicationListener | NetworkListener {
const useSSL = !!certificateArn;

let protocol: ApplicationProtocol | Protocol;
switch(elbType) {
case LoadBalancerType.ALB:
protocol = useSSL ? ApplicationProtocol.HTTPS : ApplicationProtocol.HTTP;
break;
case LoadBalancerType.NLB:
protocol = useSSL ? Protocol.TLS : Protocol.TCP;
break;
default:
throw new Error('Unsupported load balancer type.');
}

const listenerProps: BaseApplicationListenerProps | BaseNetworkListenerProps = {
port: port,
protocol: protocol,
certificates: useSSL ? [ListenerCertificate.fromArn(certificateArn)] : undefined,
};

switch(elbType) {
case LoadBalancerType.ALB: {
const alb = elb as ApplicationLoadBalancer;
return alb.addListener(id, listenerProps as BaseApplicationListenerProps);
}
case LoadBalancerType.NLB: {
const nlb = elb as NetworkLoadBalancer;
return nlb.addListener(id, listenerProps as BaseNetworkListenerProps);
}
default:
throw new Error('Unsupported load balancer type.');
}
}

/**
* Adds targets to the given listener.
* Works for both Application Load Balancers and Network Load Balancers.
*/
private static addTargetsToListener(listener: BaseListener, elbType: LoadBalancerType, id: string, port: number, target: AutoScalingGroup | InstanceTarget,
securityEnabled: boolean) {
switch(elbType) {
case LoadBalancerType.ALB: {
const albListener = listener as ApplicationListener;
albListener.addTargets(id, {
port: port,
protocol: securityEnabled ? ApplicationProtocol.HTTPS : ApplicationProtocol.HTTP,
targets: [target],
});
break;
}
case LoadBalancerType.NLB: {
const nlbListener = listener as NetworkListener;
nlbListener.addTargets(id, {
port: port,
protocol: securityEnabled ? Protocol.TLS : Protocol.TCP,
targets: [target],
});
break;
}
default:
throw new Error('Unsupported load balancer type.');
}
}
}
Loading

0 comments on commit 3e38f4a

Please sign in to comment.