Benchmark S3 gateway #102

roman-khimov · 2021-06-22T19:19:44Z

Let's try to see how our gateway performs under load. I'd take Wasabi's test: https://github.com/wasabi-tech/s3-benchmark (the one used by MinIO in https://blog.min.io/s3-benchmark-using-hdd/), but we can also consider https://github.com/dvassallo/s3-benchmark. We need to:

make sure the benchmark works with our gateway
if it doesn't, submit/fix bugs until it does, it's a good compatibility test
make sure the results are stable
if they're not, we need to figure out if it's a benchmark or gateway problem
add some documentation on how to configure node/run the test against it

Then our colleagues could take the document and run the same tests on more fancy hardware to get some representative results.

masterSplinter01 · 2021-06-30T09:54:20Z

The benchmark https://github.com/chinglinwen/s3-benchmark works.

Example how to execute:

./s3-benchmark -a %ACCESS_KEY_ID -s %SECRET_KEY -u %ENDPOINT

Result:

Wasabi benchmark program v2.0
Parameters: url=https://127.0.0.1:8084, bucket=loadgen, region=us-east-1, duration=60, threads=1, loops=1, size=1M
Loop 1: PUT time 60.1 secs, objects = 278, speed = 4.6MB/sec, 4.6 operations/sec. Slowdowns = 0
Loop 1: GET time 39.2 secs, objects = 278, speed = 7.1MB/sec, 7.1 operations/sec. Slowdowns = 0
Loop 1: DELETE time 61.8 secs, 4.5 deletes/sec. Slowdowns = 0
result title: name-concurrency-size, uloadspeed, downloadspeed
result csv: https://127-1-1M,4.63,7.08

masterSplinter01 · 2021-06-30T09:59:07Z

The benchmark https://github.com/wasabi-tech/s3-benchmark doesn't work because of #119
The benchmark https://github.com/dvassallo/s3-benchmark doesn't work because it receives a response with a non-expected error code #118 (internal server error instead of not found) and raises panic.

roman-khimov · 2021-06-30T19:57:47Z

Benchmark build:

$ git clone https://github.com/chinglinwen/s3-benchmark
$ go mod init github.com/chinglinwen/s3-benchmark
$ go mod download
$ go mod tidy
$ go build .

Intended key setup and test run (new key for the gateway, standard dev env key for the client, it doesn't work):

$ neo-go wallet init -a -w s3w.json
$ neo-go wallet dump-keys -w s3w.json
$ neofs-authmate issue-secret --wallet ../neofs-dev-env/wallets/wallet.json --peer 192.168.130.71:8080 --gate-public-key 033c4d2a3e35e385c9120e7d10bdc0ad129d39200e157b021b3f809f87d3b99324 --create-session-token
$ neofs-s3-gw -p 192.168.130.71:8080 -p 192.168.130.72:8080 -w s3w.json
$ s3-benchmark -a 5VyF3WBLziqHBpPvGayq9G8w7Bir2DTXed2s7R494cXa_8QEvgzPDTeqA8R8DjEMnBdHxFoDVv1LSMU3X6RqzAhXD -s 60dfc3143e5d48c7478eb874f1923979954db9d4d082ff4d5b4fc0f001d98540 -u http://localhost:8080 -t 4 -z 64K

Real key setup and test run (dev env key for the gateway):

$ neo-go wallet dump-keys -w ../neofs-dev-env/wallets/wallet.json
$ neofs-authmate issue-secret --wallet s3w.json --peer 192.168.130.71:8080 --gate-public-key 033c4d2a3e35e385c9120e7d10bdc0ad129d39200e157b021b3f809f87d3b99324 --create-session-token
$ neofs-s3-gw -p 192.168.130.71:8080 -p 192.168.130.72:8080 -w ../neofs-dev-env/wallets/wallet.json
$ ./s3-benchmark -a FENzs5oo1qVsfzrj3Q38xNajv98spKGDkVdUu27Xk2Np_CFXQXSmkz7CXYpBenajhXeQ78wBV73LPqxTx4jpBfZkz -s 6b9d8483c7fb204d33a5c4ad236e74c26be1400051d15609b4b9d3987bd0ea1b -u http://localhost:8080 -t 4 -z 64K

masterSplinter01 · 2021-07-06T14:30:58Z

During this #137 I found that Wasabi's benchmark uses Signature Version 2.

Headers of requests which Wasabi benchmark creates, seems like the example of header here:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html

This version is deprecated:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#UsingAWSSDK-sig2-deprecation

The gateway receives headers of Signature Version 4:
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

The fork of this benchmark -- https://github.com/chinglinwen/s3-benchmark -- uses the new version.

What do you think, @realloc ?

masterSplinter01 · 2021-07-06T14:54:48Z

Also Wasabi's benchmark looks abandoned.
For example, I couldn't build the project without fixes because of problems with imports. Relevant fixes were made in: wasabi-tech/s3-benchmark#8 , but the PR still not be merged since 2018.

alexvanin · 2021-07-09T14:13:31Z

One more option from @realloc: https://github.com/ceph/s3-tests

realloc · 2021-07-12T14:15:02Z

For the benchmark, let's fork wasabi and make all the small changes we need to make it work with our system.

We need the benchmark to compare the S3 Protocol Gateway performance against plain NeoFS gRPC API, hence it' ok to have a slightly modified version in use.

masterSplinter01 · 2021-07-14T16:37:30Z

One more option, but it's too complex for now, maybe it will be used later: https://github.com/minio/warp

masterSplinter01 · 2021-07-15T10:19:08Z

I tried to launch Chinglinwen's benchmark with neofs-dev-env commit 9c275da0 : [#115] Update neofs-node to v0.22.3.

Current results:

Looks like there is some issues with concurrency.

Executed a command with a self-signed token and 4 threads:

 ./s3-benchmark -a 8RSwu1G7pHW5Tu8tsBYqyFFLrPJ2tXEuDpVvUT9ZoQN_8pRBz5VD63R3TVbTaLEt5mhPNZmNRBbemjroSD6ztszf  -s 8d24de38587217d58cefc006939153edb33181154287a0c9978372832c28bd99  -u https://127.0.0.1:8084 -t 4
Wasabi benchmark program v2.0
Parameters: url=https://127.0.0.1:8084, bucket=loadgen, region=us-east-1, duration=60, threads=4, loops=1, size=1M
upload err InternalError: object exists
        status code: 500, request id: 441e4310-960f-42eb-bef3-b6b7cc2f00f8, host id: 
upload err InternalError: object exists
        status code: 500, request id: 2224aee8-2947-422a-ba23-64c512209d17, host id: 
upload err InternalError: object exists
        status code: 500, request id: d1e361dc-4269-4210-bd7f-8aadb2b6b1f3, host id: 
upload err InternalError: object exists
        status code: 500, request id: 4657969c-b2ac-411c-8695-7c58306c84e9, host id: 
upload err InternalError: object exists
        status code: 500, request id: 69e48b3b-2a9e-4b42-bed6-b54c5dda6957, host id: 
upload err InternalError: object exists
        status code: 500, request id: 1ba45593-4979-4995-88a6-3b113f1fca3b, host id: 
upload err InternalError: object exists
        status code: 500, request id: 416a7bf6-2c82-4b95-9e7b-22cc2820b359, host id: 
upload err InternalError: object exists
        status code: 500, request id: 78c3c3fb-3629-4fb9-9e8a-c8a266bc1903, host id: 
upload err InternalError: object exists
        status code: 500, request id: 01cb4d48-feaa-4bca-ae14-829a680e6db5, host id: 
upload err InternalError: object exists
        status code: 500, request id: 4197e13e-50e5-4437-88a5-afe0b3c1051a, host id: 
upload err InternalError: object exists
        status code: 500, request id: ccd5d824-c845-4b7c-b830-452ed552e77d, host id: 
upload err InternalError: object exists
        status code: 500, request id: 42010a10-04a4-445c-8506-7f051cc76060, host id: 
Loop 1: PUT time 52.1 secs, objects = 967, speed = 18.5MB/sec, 18.5 operations/sec. Slowdowns = 12
download err InternalError: several objects with the same name found
        status code: 500, request id: 3fde81c0-9f68-4d9a-97da-8d7760ec6156, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: dc511cfa-56d9-4d6e-b06b-8fc721b2b199, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 1caf831c-2bc3-462b-9e06-7a01c7e0424b, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 9b8657cc-2c32-45de-b359-15fd314f27fc, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: a78fef40-df13-459c-8184-2692782a268f, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 09e2070f-748b-4f62-bd99-a8f4bdf86378, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 0336aaee-45a7-4d3d-adc9-6b106edd6f2a, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 447655eb-6977-4ebc-8cdf-85550e9e3e00, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: b532b4a9-7a98-4129-bbaf-1ff8843b3401, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 96d15b53-0120-4ab3-8dca-50a02edfc4a8, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: f4f2bc02-52bb-4015-8331-b0ce18f6f835, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 5d3ccf7f-b926-4445-aa61-75a2d15d7ce2, host id: 
Loop 1: GET time 47.8 secs, objects = 1420, speed = 29.7MB/sec, 29.7 operations/sec. Slowdowns = 12
Loop 1: DELETE time 26.8 secs, 36.1 deletes/sec. Slowdowns = 0
result title: name-concurrency-size, uloadspeed, downloadspeed
result csv: https://127-4-1M,18.55,29.72

Errors in logs of the gateway:

PutObject:

2021-07-14T21:32:45.161+0300    error   handler/put.go:48       could not upload object {"request_id": "1af719eb-c5c9-4457-a029-89a852da568f", "bucket_name": "loadgen", "object_name": "Object-878", "error": "closing the stream failed: rpc error: code = Unknown desc = could not receive response: could not close stream and receive response: (*object.putStreamResponser) could not receive response: (*response.ClientMessageStreamer) could not close stream and receive response: (*putsvc.streamer) could not object put stream: (*putsvc.Streamer) could not close object target: could not close target: could not close next target: incomplete object put"}

2021-07-14T21:32:51.270+0300    error   handler/put.go:48       could not upload object {"request_id": "42010a10-04a4-445c-8506-7f051cc76060", "bucket_name": "loadgen", "object_name": "Object-968", "error": "object exists"}
2021-07-14T21:32:51.271+0300    error   api/router.go:144       something went wrong    {"status": 500, "request_id": "42010a10-04a4-445c-8506-7f051cc76060", "method": "PutObject", "description": "Internal Server Error"}

GetObject :

2021-07-14T21:33:38.748+0300    error   api/router.go:144       something went wrong    {"status": 500, "request_id": "e8ccf073-8c9c-4d72-b876-923ecbeb337b", "method": "GetObject", "description": "Internal Server Error"}
2021-07-14T21:33:39.050+0300    error   layer/layer.go:402      could not find object id        {"error": "several objects with the same name found"}
2021-07-14T21:33:39.050+0300    error   handler/get.go:180      could not find object   {"request_id": "5d3ccf7f-b926-4445-aa61-75a2d15d7ce2", "bucket_name": "loadgen", "object_name": "Object-866", "error": "several objects with the same name found"}
2021-07-14T21:33:39.050+0300    error   api/router.go:144       something went wrong    {"status": 500, "request_id": "5d3ccf7f-b926-4445-aa61-75a2d15d7ce2", "method": "GetObject", "description": "Internal Server Error"}

DeleteObject:

2021-07-14T21:34:00.135+0300    error   handler/delete.go:54    could not delete object {"request_id": "aed2165a-a8ce-4cea-8529-56d242c70172", "bucket_name": "loadgen", "object_name": "Object-764", "error": "sending the request failed: rpc error: code = Unknown desc = could not handle request: could not handle request: (*putsvc.Streamer) could not close object target: could not close target: could not close next target: incomplete object put (Object-764)"}

alexvanin · 2021-07-29T10:11:37Z

Let's try to check these errors on v0.16.0

masterSplinter01 · 2021-08-03T10:11:22Z

Tried with dev-env: a011d670 and s3-gw: a0fb14d, received the same result:

Parameters: url=http://127.0.0.1:8084, bucket=loadgen, region=us-east-1, duration=60, threads=4, loops=1, size=1M
upload err InternalError: Object: loadgen#Object-4 already exists
        status code: 500, request id: e247c0f5-2cdf-4c5c-ad05-663b90d42124, host id: 
upload err InternalError: Object: loadgen#Object-78 already exists
        status code: 500, request id: f62fbcc0-d616-4351-a4b6-65826e8e2e0e, host id: 
upload err InternalError: Object: loadgen#Object-88 already exists
        status code: 500, request id: d59ab822-1b8c-4338-a398-18b99bb51f81, host id: 
upload err InternalError: Object: loadgen#Object-276 already exists
        status code: 500, request id: 9ff05a17-024a-4137-935c-210294e42bb5, host id: 
upload err InternalError: Object: loadgen#Object-372 already exists
        status code: 500, request id: 461e46fc-d4a7-48e2-91da-21c594941fbd, host id: 
upload err InternalError: Object: loadgen#Object-795 already exists
        status code: 500, request id: 3f74c91b-13e9-4f20-afc5-b5bb70963104, host id: 
upload err InternalError: Object: loadgen#Object-805 already exists
        status code: 500, request id: 37b04256-a4c4-4541-9e02-6f74e8544240, host id: 
upload err InternalError: Object: loadgen#Object-811 already exists
        status code: 500, request id: 005e6cac-6f33-48f8-9049-48ba50d9407e, host id: 
Loop 1: PUT time 60.1 secs, objects = 958, speed = 15.9MB/sec, 15.9 operations/sec. Slowdowns = 8
download err InternalError: several objects with the same name found
        status code: 500, request id: 767e3834-0ed2-4c3a-a180-84ff84dc19b8, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 0e1907cf-9d62-4105-9ca2-f96826af5782, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: ee1747bf-df6d-439f-a0cd-faac4d354c48, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: d4be93cd-ca2d-41ed-bc3f-a870839a0fe6, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: f9ba0d69-c531-4b33-9ffc-329e881d0a8b, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 3dae443c-d7a7-4bcb-9930-c451279a7718, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 3a79b687-4de7-407d-90ba-93fbab827d57, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: d185e7fa-94de-46b4-8630-8dac01adaa40, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 155ebebd-c114-4e62-9eca-dfcefffed952, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: d047c575-5855-4792-9e2e-9175838d7864, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 2e2b701b-5390-4185-bae6-a9126d7850af, host id: 
download err InternalError: several objects with the same name found
        status code: 500, request id: 0beeb1b5-1da1-49c1-87ec-b654d7470165, host id: 
Loop 1: GET time 58.3 secs, objects = 1247, speed = 21.4MB/sec, 21.4 operations/sec. Slowdowns = 12
Loop 1: DELETE time 37.9 secs, 25.3 deletes/sec. Slowdowns = 0
result title: name-concurrency-size, uloadspeed, downloadspeed
result csv: 127-4-1M,15.93,21.41

Logs of the gw:

2021-08-03T12:59:22.487+0300    error   handler/put.go:54       could not upload object {"request_id": "ae1d5507-9232-4085-8bc2-d0e190c264dc", "bucket_name": "loadgen", "object_name": "Object-78", "error": "closing the stream failed: rpc error: code = Unknown desc = could not receive response: could not close stream and receive response: (*object.putStreamResponser) could not receive response: (*response.ClientMessageStreamer) could not close stream and receive response: (*putsvc.streamer) could not object put stream: (*putsvc.Streamer) could not close object target: could not close target: could not close next target: incomplete object put"}

2021-08-03T13:00:18.848+0300    info    api/router.go:153       call method     {"status": 0, "request_id": "d075bb9f-a6f6-40d0-8c2a-164cd85c0e4a", "method": "GetObject", "description": ""}
2021-08-03T13:00:18.850+0300    error   layer/layer.go:268      could not find object id        {"error": "several objects with the same name found"}
2021-08-03T13:00:18.850+0300    error   handler/get.go:179      could not find object   {"request_id": "767e3834-0ed2-4c3a-a180-84ff84dc19b8", "bucket_name": "loadgen", "object_name": "Object-383", "error": "several objects with the same name found"}

2021-08-03T13:01:20.674+0300    info    api/router.go:153       call method     {"status": 204, "request_id": "0d7f4b3b-5231-4e07-89d9-caa523d57128", "method": "DeleteObject", "description": "No Content"}
2021-08-03T13:01:20.703+0300    error   handler/delete.go:53    could not delete object {"request_id": "8633584c-e302-4854-bd3f-75245f28a3ef", "bucket_name": "loadgen", "object_name": "Object-119", "error": "sending the request failed: rpc error: code = Unknown desc = could not handle request: could not handle request: (*putsvc.Streamer) could not close object target: could not close target: could not close next target: incomplete object put (Object-119)"}

alexvanin · 2021-08-04T11:17:30Z

Tried with dev-env: a011d670 and s3-gw: a0fb14d, received the same result:

It seems that some of the errors are NeoFS related errors.

could not upload object {"request_id": "ae1d5507-9232-4085-8bc2-d0e190c264dc", "bucket_name": "loadgen", "object_name": "Object-78", "error": "closing the stream failed: rpc error: code = Unknown desc = could not receive response: could not close stream and receive response: (*object.putStreamResponser) could not receive response: (*response.ClientMessageStreamer) could not close stream and receive response: (*putsvc.streamer) could not object put stream: (*putsvc.Streamer) could not close object target: could not close target: could not close next target: incomplete object put"}

2021-08-03T13:01:20.703+0300    error   handler/delete.go:53    could not delete object {"request_id": "8633584c-e302-4854-bd3f-75245f28a3ef", "bucket_name": "loadgen", "object_name": "Object-119", "error": "sending the request failed: rpc error: code = Unknown desc = could not handle request: could not handle request: (*putsvc.Streamer) could not close object target: could not close target: could not close next target: incomplete object put (Object-119)"}

I have two concerns:

NeoFS Storage nodes has bugs during parallel data upload,
Docker environment affects NeoFS Storage node performance that lead to these errors.

Maybe it is related to nspcc-dev/neofs-node#676 somehow.

Is it hard to reproduce it in local environments? Are there any instructions for it?

masterSplinter01 · 2021-08-04T11:36:49Z

No, it's not hard, I executed like here, but appended -t 4.

alexvanin · 2021-12-09T10:27:15Z

We did all the tests we need for now. For automatic runs of s3 compatibility tests there will be different issue.

roman-khimov added the documentation Improvements or additions to documentation label Jun 22, 2021

roman-khimov assigned masterSplinter01 Jun 22, 2021

This was referenced Jun 30, 2021

Panic in nspcc-dev/neofs-api-go/v2/session.(*RequestMetaHeader).StableMarshal during benchmark #123

Closed

Multithreaded upload errors #124

Closed

Deletion errors on benchmark rerun #125

Closed

This was referenced Jul 5, 2021

null-versioning and corresponding handler #134

Merged

Forbidden put-object in Wasabi's benchmark #137

Closed

KirillovDenis mentioned this issue Jul 12, 2021

Try ceph s3 compatibility tests #150

Closed

masterSplinter01 mentioned this issue Jul 14, 2021

Try to launch Chinglinwen's benchmark #162

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark S3 gateway #102

Benchmark S3 gateway #102

roman-khimov commented Jun 22, 2021

masterSplinter01 commented Jun 30, 2021

masterSplinter01 commented Jun 30, 2021

roman-khimov commented Jun 30, 2021

masterSplinter01 commented Jul 6, 2021 •

edited

Loading

masterSplinter01 commented Jul 6, 2021

alexvanin commented Jul 9, 2021

realloc commented Jul 12, 2021

masterSplinter01 commented Jul 14, 2021 •

edited

Loading

masterSplinter01 commented Jul 15, 2021

alexvanin commented Jul 29, 2021

masterSplinter01 commented Aug 3, 2021 •

edited

Loading

alexvanin commented Aug 4, 2021

masterSplinter01 commented Aug 4, 2021

alexvanin commented Dec 9, 2021

Benchmark S3 gateway #102

Benchmark S3 gateway #102

Comments

roman-khimov commented Jun 22, 2021

masterSplinter01 commented Jun 30, 2021

masterSplinter01 commented Jun 30, 2021

roman-khimov commented Jun 30, 2021

masterSplinter01 commented Jul 6, 2021 • edited Loading

masterSplinter01 commented Jul 6, 2021

alexvanin commented Jul 9, 2021

realloc commented Jul 12, 2021

masterSplinter01 commented Jul 14, 2021 • edited Loading

masterSplinter01 commented Jul 15, 2021

alexvanin commented Jul 29, 2021

masterSplinter01 commented Aug 3, 2021 • edited Loading

alexvanin commented Aug 4, 2021

masterSplinter01 commented Aug 4, 2021

alexvanin commented Dec 9, 2021

masterSplinter01 commented Jul 6, 2021 •

edited

Loading

masterSplinter01 commented Jul 14, 2021 •

edited

Loading

masterSplinter01 commented Aug 3, 2021 •

edited

Loading