Skip to content

Commit 5ace3e8

Browse files
alekoduhannesmann
alekodu
and
hannesmann
authored
Merge development (#89)
* Separate image URLs into production and development * Add parameter to settings for image URL and policy * Remove Python application emulator * Target Go 1.20 and upgrade all packages This shouldn't change anything since k8s.io/apimachinery already requires Go 1.20 * Move the application model into a separate directory * Add placeholder Go emulator * Move Dockerfile to the repo root This is necessary because the emulator needs to access both emulator/ and model/ * Read config in emulator (test) * Set GOMAXPROCS to the number of processes in configmap * Shebang should always be on the first line * Add script to redeploy emulator in dev environment * Move HTTP server into goroutine * Use encoding/json instead of sending string * Add notFoundHandler * Add endpointHandler * Clean up emulator code * Add testing config map for running emulator outside of K8s * Change JSON response to better match old Python emulator * Use struct instead of creating functions * Export response structs * Add CPU stressor * Move CPU time function into util * Add basic logging for endpoint calls * Add ExecParallel for stressors * Read SERVICE_NAME from env * Remove unused istio.go and redis.go * Use Go workspace to separate modules * Add CPU task response * Add network task function * Refactor stressors to use interface * Indent JSON response * Forward the HTTP request to NetworkTask * Propagate headers from inbound to outbound * Move payload generation into separate function * Format time when logging * Add a restful.POST function for forwarding requests * Restructure code to avoid import cycles * Change called service port type to int * Add function for forwarding requests to endpoints * Set the slice capacity on init * Call other endpoints in NetworkTask * Combine network responses I don't like this code... * Omit port if zero This seems to match what Python does * Simplify code for network complexity * Fix for NetworkTask == nil * Don't append services and statuses twice * Format status properly * Fix duplicated service name in network complexity * Embed task responses in RESTResponse * Add ForwardParallel function * Move threads parameter into CPU complexity * Support CPU stressor on multiple goroutines * Rename response.go to api.go Since it contains requests now * Log CPU tasks if logging=true * Move documentation from wiki into repo * Move documentation from wiki into repo * Worker image should not be built in model/ anymore * Worker image should not be built in model/ anymore * Add logging for network tasks * Also remove threads from model.Service * Remove the workaround for null network complexity In Go, a nil slice is an empty slice, so it works without panicking * Ensure CPU complexity uses at least one thread in generator * Allow processes = 0 in service * Panic if input contains unknown fields * Show protocol in network task log * Set execution mode and forward requests in validation.go * Update simple example to work with the new emulator * Generate a new complex example * Log configuration on startup * Add documentation for adding a new stressor * Add stressor documentation to generator-parameters.md * Fix issue with execution time rounding * Remove lock_threads from cpu complexity It always needs to be on or the stressor doesn't work properly * Fix image link in complex example * Add comment to clarify that HTTP server should always be running * Launch gRPC server on demand * Convert requests and responses to protobufs gRPC works with protobufs so this makes it easier to share types between the HTTP and gRPC server * Move protobuf structs to generated package * Add a generated placeholder gRPC service to the repo * Set AllowPartial to true Should be faster * Unconditionally use HTTP server as readiness probe * Compile the service at runtime This is necessary once gRPC support is added * Set run.sh executable bit * Clean caches after build * More edits to try to save space * Add placeholder file for registering services * Remove status from response HTTP provides 200 OK, 404 Not Found, etc and gRPC provides OK, INVALID_ARGUMENT, etc * Start gRPC server * Update stressors.md for new API * Start generating emulator code * Add function for converting K8s name to Go name * Use goname to generate service.go * Add logging to stressors.md * Generate gRPC code in run.sh * Implement gRPC health service * Generate service name when Check is called This is faster since we only have one service anyway * Do not update modules in run.sh The emulator is supposed to use the modules in the base image * Download protoc in run.sh * Fix protoc not finding packages * Use goname in service.tmpl * Fix segfault in generator * Fix go build not working * Expose gRPC on port 81 * Add function CallGeneratedEndpoint * Enable gRPC reflection service For grpcurl, etc * Only generate protobufs for gRPC services * Install grpc_health_probe in base image * Check for empty response data * Also check ResponseData.Tasks != nil * Specify protocol for microservice instead of endpoint As discussed with Aleksandra, running two servers on two ports is complicated and unrealistic for a microservice * Forward requests to gRPC endpoints * Add protocols to network task response * Update impl.tmpl to match new definition * Print protocols in LogNetworkTask * Fix parallel gRPC request * Update stressors.md to match current implementation * Create Dockerfile for application-generator * Write generated files to k8s/generated * Build Docker image in generator * Write Docker output to stdout/stderr * Fix error in Docker build * Do not download modules for model Since emulator depends on model * Delete placeholder files from base layer * Copy grpc_health_probe into final image * Fix grpc_health_probe path * Improve endpoint response format to be less ambiguous * Add build ID to Docker image and config to make sure they match * Warn instead of panicking on build ID mismatch * Change all errors to lowercase https://github.com/golang/go/wiki/CodeReviewComments#error-strings * Update documentation to match new response format * Fix inconsistent endpoint keys * Fix buildID not being set in emulator * Remove most TODOs * Fix buildID not being set in Dockerfile * Move generated files out of k8s/ Fixes warning when running deploy.sh * Remove the development config map * Document api.proto * Use strcase package * Ignore go.work.sum This file includes other modules installed by the user, such as gopls * Separate generated files into client and server Prevents import cycle error * Remove old run.sh script * Ensure that gRPC status codes are returned on error * Return INVALID_ARGUMENT if endpoint doesn't exist * Fix api.proto comment * Update documentation for Docker base and layered images * Add DefaultProtocol back into generator * Split CreateK8sYaml into two functions * Name docker images after hostname This ensures they won't be pulled down from the Docker registry * Remove buildID and give all images unique tag * Delete old hydragen-emulator images * Never cache results from image build * Add auto-deployment of images for Kind * Add break to deploy.sh * Add containerd-push-image-to-clusters.sh * Pass sudo password to remote script * Fix containerd-push-image-to-clusters.sh * Remove old code from deploy.sh * Fix ssh commands in containerd-push-image-to-clusters.sh * Document helper scripts for kind and containerd * Run goimports after generating gRPC code * Ensure TrafficForwardRatio is positive * Implement TrafficForwardRatio * Don't allow traffic forward ratio under 1 Otherwise the default is 0 * Make sure multiple responses from endpoint can be returned * Change container name back to "app" * Add SSH and sudo password options to containerd-push-image-to-clusters.sh * Set GOMEMLIMIT in k8s manifest * Add debug output to containerd script * Only use contexts in containerd script * Don't reference loop variable in HTTP server https://github.com/golang/go/wiki/CommonMistakes#using-reference-to-loop-iterator-variable * Set default emulator base image to busybox Provides a shell and utilities without increasing image size by much * Rename "base image" to "source image" * Add a base_image option in config * Set BASEIMAGE in Dockerfile * Document base_image parameter * Automatically determine port and protocol of called service * Add an option to use the development image with the random preset * Fix port and protocol not applying in network complexity * Update simple example for new Go emulator * Update complex example by generating new application --------- Co-authored-by: Hannes Mann <hannesmann2000@gmail.com>
1 parent f2d2da3 commit 5ace3e8

File tree

123 files changed

+5154
-3911
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

123 files changed

+5154
-3911
lines changed

‎.gitignore

+2-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
1-
.idea/
1+
.idea/
2+
go.work.sum

‎Dockerfile

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#
2+
# Copyright 2023 Ericsson AB
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
# This is the source image for the application emulator
18+
# It contains the source code, Go compiler and protobuf compiler
19+
# The generator will compile a unique layered image for the current configuration
20+
21+
FROM golang:1.20
22+
23+
# Install protoc
24+
RUN apt update && apt install -y protobuf-compiler
25+
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.31.0
26+
RUN go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.3.0
27+
28+
# Copy relevant parts of the source tree to the new source dir
29+
COPY emulator /usr/src/emulator/emulator
30+
COPY model /usr/src/emulator/model
31+
# Delete placeholder files
32+
RUN rm -Rf /usr/src/emulator/emulator/src/generated
33+
34+
WORKDIR /usr/src/emulator
35+
36+
# Create Go workspace
37+
RUN go work init
38+
RUN go work use ./emulator
39+
RUN go work use ./model
40+
41+
# Download as many modules as possible to be shared between compilations
42+
RUN cd emulator && go mod download -x

‎README.md

+9-4
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
# HydraGen
2+
23
HydraGen is a tool that allows generating a wide range of microservice benchmarks in a systematic and flexible way. This tool facilitates evaluating performance and scalability of various resource management strategies entailing microservice-based architectures hosted in cloud environments. Currently, HydraGen can generate benchmarks emulating web-based applications with HTTP or gRPC servers.
34

45
## License and copyright
@@ -8,16 +9,20 @@ HydraGen is a free software. You can use, distribute and/or modify it under the
89
HydraGen's developement is driven by both Ericsson Research and Umea University.
910

1011
## Papers and scientific reports
12+
1113
The design and evaluation of HydraGen is being published at IEEE Cloud 2023.
1214
> M. R. Saleh Sedghpour, A. Obeso Duque, X. Cai, B. Skubic, E. Elmroth, C. Klein and J. Tordsson, "HydraGen: A Microservice Benchmark Generator," in IEEE International Conference on Cloud Computing, vol. X, no. Y, pp. Z, Day Month 2023, doi: N.
1315
14-
## Want to use our work?
15-
Check our documentation in our [Wiki](https://github.com/EricssonResearch/cloud-native-app-simulator/wiki) pages.
16+
## Want to use our work?
17+
18+
Check our documentation [here](docs/home.md).
1619
If you use our tool, please cite our work.
1720

1821
## Want to contribute?
22+
1923
We welcome new contributions to our project via pull requests.
20-
If you are interested in contribution, please check the list of open issues and visit the [Development Environment](https://github.com/EricssonResearch/cloud-native-app-simulator/wiki/Development-Environment) section under our wiki pages to learn about how to setup a development environment.
24+
If you are interested in contribution, please check the list of open issues and visit the [Development Environment](docs/development-environment.md) section in our documentation to learn about how to setup a development environment.
2125

2226
## Acknowledgements
23-
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
27+
28+
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

‎community/containerd-import-image.sh

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/bash
2+
#
3+
# Copyright 2023 Ericsson AB
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# This script is run over SSH and caches the image being sent over the network in containerd
18+
19+
password="$1"
20+
21+
if [[ -z "$password" ]]; then
22+
ctr -n=k8s.io images import -
23+
else
24+
# First authorize (timeout is usually 15 minutes)
25+
echo "$password" | sudo -S -v -p ""
26+
# Now read from ssh stdin
27+
sudo ctr -n=k8s.io images import -
28+
fi

‎community/containerd-push-image-to-clusters.sh

+113
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
#!/bin/bash
2+
#
3+
# Copyright 2023 Ericsson AB
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
has_sudo_password=false
19+
sudo_password=""
20+
has_ssh_password=false
21+
ssh_password=""
22+
23+
while getopts ":s:p:n" option; do
24+
case "${option}" in
25+
s)
26+
has_sudo_password=true
27+
sudo_password="$OPTARG"
28+
;;
29+
p)
30+
has_ssh_password=true
31+
ssh_password="$OPTARG"
32+
;;
33+
n)
34+
has_sudo_password=true
35+
sudo_password=""
36+
;;
37+
*)
38+
echo "Usage: $0 -s <sudo password> -p <ssh password> -n"
39+
echo "Parameters:"
40+
echo " -s: Set sudo password to argument"
41+
echo " -p: Set ssh password to argument (if sshpass is installed)"
42+
echo " -n: Skip sudo password prompt"
43+
exit 0
44+
;;
45+
esac
46+
done
47+
48+
name="$(hostname -f)/hydragen-emulator"
49+
image="$(docker images $name --format '{{.Repository}}:{{.Tag}}')"
50+
51+
cd "$(git rev-parse --show-toplevel)/generator/k8s"
52+
contexts="$(echo *)"
53+
#contexts="$(kubectl config get-contexts --output=name | tr '\n' ' ')"
54+
55+
echo "Contexts: $contexts"
56+
echo "Trying to discover all nodes that need an updated image..."
57+
echo ""
58+
59+
nodes=()
60+
61+
# Try every context
62+
# TODO: Does not check for the "node" property in configmap
63+
for ctx in $contexts; do
64+
echo "Trying to access context $ctx"
65+
cmd="kubectl get nodes -o custom-columns=:metadata.name,:spec.taints[].effect --no-headers --context $ctx"
66+
output="$($cmd 2>&1)"
67+
if [[ $? == 0 ]]; then
68+
echo " Kubectl returned nodes: $(echo $output | tr '\n' ' ')"
69+
ctxnodes="$(echo "$output" | grep -v 'NoSchedule' | cut -d ' ' -f 1 | tr '\n' ' ')"
70+
echo " Nodes that can have pods scheduled: $ctxnodes"
71+
for node in $ctxnodes; do
72+
nodes+=("$ctx/$node")
73+
done
74+
else
75+
echo " Command failed (exit status $?): $(echo $output | tr '\n' ' ')"
76+
fi
77+
echo ""
78+
done
79+
80+
echo "Nodes: ${nodes[@]}"
81+
82+
if [[ $has_sudo_password == false ]]; then
83+
read -s -p "Sudo password (leave blank if '$(whoami)' has administrative access to containerd): " sudo_password
84+
if [[ -z "$sudo_password" ]]; then
85+
echo -n "(not using sudo)"
86+
fi
87+
echo ""
88+
fi
89+
90+
for node in "${nodes[@]}"; do
91+
IFS="/" read -r ctx name <<< $node
92+
# https://kubernetes.io/docs/reference/kubectl/cheatsheet/
93+
jsonpath="{.status.addresses[?(@.type=='InternalIP')].address}"
94+
ip="$(kubectl get nodes $name --context $ctx -o jsonpath=$jsonpath)"
95+
file="/tmp/containerd-import-image.sh"
96+
97+
# Start ssh in background
98+
if [[ $has_ssh_password == true ]]; then
99+
sshpass -p "$ssh_password" ssh -M -S /tmp/containerd-import-ssh-socket -fnNT "$(whoami)@$ip"
100+
else
101+
ssh -M -S /tmp/containerd-import-ssh-socket -fnNT "$(whoami)@$ip"
102+
fi
103+
104+
# Copy script to remote machine
105+
scp -o "ControlPath=/tmp/containerd-import-ssh-socket" ../../community/containerd-import-image.sh "$(whoami)@$ip:/tmp/containerd-import-image.sh"
106+
# Execute script with archive coming from stdin
107+
ssh -S /tmp/containerd-import-ssh-socket "$(whoami)@$ip" "chmod +x /tmp/containerd-import-image.sh"
108+
# Add space at the start to prevent password from being saved in bash history
109+
cat ../generated/hydragen-emulator.tar | ssh -S /tmp/containerd-import-ssh-socket -C "$(whoami)@$ip" " /tmp/containerd-import-image.sh "$sudo_password""
110+
ssh -S /tmp/containerd-import-ssh-socket "$(whoami)@$ip" "rm /tmp/containerd-import-image.sh"
111+
# Close ssh session
112+
ssh -S /tmp/containerd-import-ssh-socket -O exit "$(whoami)@$ip"
113+
done

‎community/push-image-to-clusters.sh ‎community/kind-push-image-to-clusters.sh

+6-5
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
#!/bin/bash
12
#
23
# Copyright 2021 Ericsson AB
34
#
@@ -13,16 +14,16 @@
1314
# See the License for the specific language governing permissions and
1415
# limitations under the License.
1516
#
16-
#!/bin/bash
1717

1818
DEFAULT_NUM=2
1919
if [ -z "$1" ]; then
20-
NUM=$DEFAULT_NUM
20+
NUM=$DEFAULT_NUM
2121
else
22-
NUM=$1
22+
NUM=$1
2323
fi
2424

25-
# Push the image to the all clusters
25+
# Push the image to all clusters
2626
for i in $(seq ${NUM}); do
27-
kind load docker-image app-demo --name=cluster-${i}
27+
name="$(hostname -f)/hydragen-emulator"
28+
kind load docker-image "$(docker images $name --format '{{.Repository}}:{{.Tag}}')" --name=cluster-${i}
2829
done

‎community/kind-setup-clusters.sh

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
#!/bin/bash
12
#
23
# Copyright 2021 Ericsson AB
34
#
@@ -13,14 +14,13 @@
1314
# See the License for the specific language governing permissions and
1415
# limitations under the License.
1516
#
16-
#!/bin/bash
1717

1818
DEFAULT_NUM=2
1919
DEFAULT_CONFIG="kind-cluster-3-nodes.yaml"
2020
if [ -z "$1" ]; then
21-
NUM=$DEFAULT_NUM
21+
NUM=$DEFAULT_NUM
2222
else
23-
NUM=$1
23+
NUM=$1
2424
fi
2525

2626
if [ -z "$2" ]; then
@@ -34,5 +34,5 @@ fi
3434
# Create the kind multi-node clusters based on the given config
3535
for i in $(seq ${NUM}); do
3636
kind create cluster --name cluster-${i} --config $CONFIG
37-
kind load docker-image app-demo --name=cluster-${i}
37+
kind load docker-image hydragen-emulator --name=cluster-${i}
3838
done

‎docs/development-environment.md

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Development Environment
2+
3+
This document helps you get started developing code for HydraGen.
4+
If you follow this guide and find a problem, please take a few minutes to update this page.
5+
6+
HydraGen's build system is designed to run with minimal dependencies:
7+
8+
- kind
9+
- docker
10+
- git
11+
12+
These dependencies need to be set up before building and running the code.
13+
14+
- [Setting Up Docker](#setting-up-docker)
15+
- [Build the worker image](#build-the-worker-image)
16+
- [Setting Up Kind](#setting-up-kind)
17+
18+
## Setting up Docker
19+
20+
To use docker to build required images you will need:
21+
22+
- **docker tools:** To download and install Docker follow [these instructions](https://docs.docker.com/install/).
23+
24+
## Setting up Kind
25+
26+
To be able to run the *hydragen-emulator container* on a sample cluster, we use
27+
[Kind](https://kind.sigs.k8s.io/docs/user/quick-start/).
28+
29+
- **Installation:** To download and install Kind follow [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/).
30+
- **Setup the clusters:** To setup the clusters, you could simply run the [`kind-setup-clusters.sh`](kind-setup-clusters.sh)
31+
script.
32+
33+
```bash
34+
#
35+
# This will create multiple Kind clusters (default 2)
36+
# The naming of each cluster is followed by the number (ie, cluster-1, cluster-2, etc.)
37+
# Each of the created clusters has 3 worker nodes and one control plane by default.
38+
#
39+
40+
cd community
41+
./kind-setup-clusters.sh [number of clusters (default 2)] [config of each cluster (default kind-cluster-3-nodes.yaml)]
42+
```
43+
44+
## Build the source image
45+
46+
The source image, containing code and compilers, needs to be built from the local source code.
47+
48+
```bash
49+
docker build -t "$(hostname -f)/hydragen-base" .
50+
```
51+
52+
By default, HydraGen will use a release image from GitHub Packages as the base when building your image.
53+
To use the development source image instead set this option in the input JSON configuration:
54+
55+
```json
56+
{
57+
...
58+
"settings": {
59+
"development": true
60+
},
61+
...
62+
}
63+
```
64+
65+
## Pushing the image to a cluster
66+
67+
The generated image needs to be pushed to all clusters after you have run `generator.sh`.
68+
69+
```bash
70+
cd community
71+
./push-image-to-clusters.sh [number of clusters (default 2)]
72+
```
73+
74+
## Logging
75+
76+
To be able to have logging, simply follow the instructions [here](logging.md).

0 commit comments

Comments
 (0)