Multithreaded upload errors

[roman-khimov]

The same setup from #102 (comment), benchmark:

Parameters: url=http://localhost:8080, bucket=loadgen, region=us-east-1, duration=10, threads=4, loops=1, size=64K
upload err InternalError: object exists
        status code: 500, request id: 96d880ac-141a-466c-867a-99de254491ba, host id: 
upload err AccessDenied: Access Denied.
        status code: 403, request id: ba058875-7fd9-4a46-bafb-83c56100e40d, host id: 
upload err AccessDenied: Access Denied.
        status code: 403, request id: ac76e142-3822-4801-bcfb-b4631ec59803, host id: 
upload err InternalError: object exists
        status code: 500, request id: 96044b3b-4beb-4d16-90c8-61b7ff2a9077, host id: 
upload err InternalError: object exists
        status code: 500, request id: 7991552e-9455-4de5-97d6-c687e4e54463, host id: 
upload err InternalError: object exists
        status code: 500, request id: c8d353b5-cde9-4ebb-b5f0-266c7f5ea11b, host id: 
upload err RequestError: send request failed
caused by: Put "http://localhost:8080/loadgen/Object-283": read tcp [::1]:38810->[::1]:8080: read: connection reset by peer
upload err InternalError: object exists
        status code: 500, request id: 69cd1e83-fdb3-4563-863a-6a368d1b7c81, host id: 
upload err InternalError: object exists
        status code: 500, request id: 78f5bd61-7c57-4ea9-8c05-58c83fd8b997, host id: 
upload err InternalError: object exists
        status code: 500, request id: 1648de57-3bbf-4062-b7ca-b502bc96902b, host id: 
upload err InternalError: object exists
        status code: 500, request id: 94063d21-63e8-4bc3-89d5-7b53ea5cedfd, host id: 

Gateway:

2021-06-30T23:15:03.898+0300    error   layer/container.go:52   could not fetch container       {"cid": "n6GtDnBkv5Qe6fWnsJXj1yZwNjibRXb4JLKoJG1zdeN", "request_id": "91136b79-5cf0-4765-b585-d40687914865", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: 4a31a4b40b41e539cecd00ffb757b661ad812d0f2079e30ca2ec3f3d69f8831 : ef4b5297222aeb3afbfa707d2a4187e1d90e5d96febdb1a608c1c4556eb63c3c: invalid signature"}
...
2021-06-30T23:15:04.306+0300    error   layer/container.go:52   could not fetch container       {"cid": "n6GtDnBkv5Qe6fWnsJXj1yZwNjibRXb4JLKoJG1zdeN", "request_id": "0fdc3781-4a02-4fa8-962a-b67edd8e6155", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: d787e41be7bcaabe7d9733affdc21f2fe5eb81488b3b5dc2ea88302e0aaada76 : 7e23aa87100447b0a9550c685ac82e04911c583ed2e4feab57aeeabff2ab7ff0: invalid signature"}
...
2021-06-30T23:15:04.724+0300    error   handler/put.go:57       could not upload object {"request_id": "c6fed2ec-b802-4602-a765-14e45b4220ee", "bucket_name": "loadgen", "object_name": "Object-247", "error": "reading the response failed: rpc error: code = Unknown desc = malformed request: invalid session token signature"}
...
2021-06-30T23:15:04.779+0300    error   api/user-auth.go:29     failed to pass authentication   {"error": "reading the response failed: rpc error: code = Unknown desc = malformed request: invalid session token signature"}
...
2021-06-30T23:15:04.831+0300    error   handler/put.go:57       could not upload object {"request_id": "77782c49-e2f4-4ac2-a78b-db3778f07a47", "bucket_name": "loadgen", "object_name": "Object-250", "error": "object exists"}
...
2021-06-30T23:15:05.694+0300    error   layer/container.go:52   could not fetch container       {"cid": "4yCZ5GyvvVSPkMMZT1Mju4UhsXy3XtUPkHair9KSBaxv", "request_id": "c243731f-028b-40db-9475-d5e0fa8f5e98", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: c4bca3349551ad966651f722ac4918a9c9f0ef1e30385642d387ac37073984c6 : 51836ee9a5f2e7d2c5ca1d48b8e9c51451f4d2dd0c9a9457c3382b7e8424f554: invalid signature"}
2021-06-30T23:15:05.694+0300    error   layer/container.go:105  could not fetch container info  {"request_id": "c243731f-028b-40db-9475-d5e0fa8f5e98", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: c4bca3349551ad966651f722ac4918a9c9f0ef1e30385642d387ac37073984c6 : 51836ee9a5f2e7d2c5ca1d48b8e9c51451f4d2dd0c9a9457c3382b7e8424f554: invalid signature"}
2021-06-30T23:15:05.694+0300    error   handler/put.go:57       could not upload object {"request_id": "c243731f-028b-40db-9475-d5e0fa8f5e98", "bucket_name": "loadgen", "object_name": "Object-275", "error": "rpc error: code = NotFound desc = bucket not found"}

Easily reproducible.

[roman-khimov]

Similar things happen during download phase:

2021-06-30T23:15:08.214+0300    error   layer/container.go:95   could not fetch container       {"request_id": "3fce87bd-1d7c-4ef0-ac4a-ab8c57507e55", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: 596bbf923e23e6ad671de3bea08acee2a76b82ead99c2b498a0fdad2071a70f2 : aae9ba2ff67603c252618f1839703223d887c70b6609fce9c3ba13d57b559d10: invalid signature"}
2021-06-30T23:15:08.214+0300    error   layer/layer.go:355      could not fetch bucket info     {"error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: 596bbf923e23e6ad671de3bea08acee2a76b82ead99c2b498a0fdad2071a70f2 : aae9ba2ff67603c252618f1839703223d887c70b6609fce9c3ba13d57b559d10: invalid signature"}
2021-06-30T23:15:08.214+0300    error   handler/get.go:157      could not find object   {"request_id": "3fce87bd-1d7c-4ef0-ac4a-ab8c57507e55", "bucket_name": "loadgen", "object_name": "Object-287", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: 596bbf923e23e6ad671de3bea08acee2a76b82ead99c2b498a0fdad2071a70f2 : aae9ba2ff67603c252618f1839703223d887c70b6609fce9c3ba13d57b559d10: invalid signature"}
...
2021-06-30T23:15:08.363+0300    error   layer/layer.go:367      could not fetch object head     {"error": "sending the request failed: rpc error: code = Unknown desc = malformed request: invalid session token signature"}
2021-06-30T23:15:08.363+0300    error   handler/get.go:157      could not find object   {"request_id": "99bfae89-d542-481b-848b-9757979384dc", "bucket_name": "loadgen", "object_name": "Object-87", "error": "sending the request failed: rpc error: code = Unknown desc = malformed request: invalid session token signature"}
...
2021-06-30T23:15:08.513+0300    error   layer/layer.go:358      could not find object id        {"error": "reading the response failed: rpc error: code = Unknown desc = malformed request: invalid session token signature"}
2021-06-30T23:15:08.513+0300    error   handler/get.go:157      could not find object   {"request_id": "4182fc16-f979-4a60-9c5e-5c66502c8613", "bucket_name": "loadgen", "object_name": "Object-213", "error": "reading the response failed: rpc error: code = Unknown desc = malformed request: invalid session token signature"}
...
2021-06-30T23:15:08.677+0300    error   layer/container.go:52   could not fetch container       {"cid": "4yCZ5GyvvVSPkMMZT1Mju4UhsXy3XtUPkHair9KSBaxv", "request_id": "92bca956-08bb-475b-b338-35638c3f38ba", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: 896b86b122874fef3a8b767da79bc09f8ef72050c61ff2b422bddddd860ec8a2 : d20d856e607c51771b84e9e6e3721d759612957c7c9599c8f66fc6d743e0c241: invalid signature"}
2021-06-30T23:15:08.677+0300    error   layer/container.go:105  could not fetch container info  {"request_id": "92bca956-08bb-475b-b338-35638c3f38ba", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: 896b86b122874fef3a8b767da79bc09f8ef72050c61ff2b422bddddd860ec8a2 : d20d856e607c51771b84e9e6e3721d759612957c7c9599c8f66fc6d743e0c241: invalid signature"}
2021-06-30T23:15:08.677+0300    error   layer/layer.go:355      could not fetch bucket info     {"error": "rpc error: code = NotFound desc = bucket not found"}
2021-06-30T23:15:08.677+0300    error   handler/get.go:157      could not find object   {"request_id": "92bca956-08bb-475b-b338-35638c3f38ba", "bucket_name": "loadgen", "object_name": "Object-273", "error": "rpc error: code = NotFound desc = bucket not found"}

[roman-khimov]

And delete:

2021-06-30T23:15:22.595+0300    error   layer/container.go:52   could not fetch container       {"cid": "4yCZ5GyvvVSPkMMZT1Mju4UhsXy3XtUPkHair9KSBaxv", "request_id": "5eef0736-5c2f-4eb5-aad1-2c73f3abb007", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: f06ccbdd9e03101471762baafdac0f9695f6f097ad65946aadd4235b2ae350b3 : 23bb5a6b298b54d32fb71bf133690204cad8bf12e7753dc68400e605f0907323: invalid signature"}
2021-06-30T23:15:22.595+0300    error   layer/container.go:105  could not fetch container info  {"request_id": "5eef0736-5c2f-4eb5-aad1-2c73f3abb007", "error": "transport error: rpc error: code = Unknown desc = could not verify request: could not verify meta header: f06ccbdd9e03101471762baafdac0f9695f6f097ad65946aadd4235b2ae350b3 : 23bb5a6b298b54d32fb71bf133690204cad8bf12e7753dc68400e605f0907323: invalid signature"}
2021-06-30T23:15:22.595+0300    error   handler/delete.go:54    could not delete object {"request_id": "5eef0736-5c2f-4eb5-aad1-2c73f3abb007", "bucket_name": "loadgen", "object_name": "Object-55", "error": "rpc error: code = NotFound desc = bucket not found (Object-55)"}
...
2021-06-30T23:15:22.738+0300    error   handler/delete.go:54    could not delete object {"request_id": "7cf124bf-e0ec-4f05-8939-d50437316c67", "bucket_name": "loadgen", "object_name": "Object-57", "error": "sending the request failed: rpc error: code = Unknown desc = could not handle request: could not handle request: (*putsvc.Streamer) could not close object target: could not close target: could not close next target: incomplete object put (Object-57)"}

[KirillovDenis]

Couldn't reproduce using api v1.28.3. However it is worth increasing NEOFS_OBJECT_PUT_POOL_SIZE on the storage node