Caching application/json responses

[InterferencePattern]

Hi, I'm seeing some confusing results when I cache larger JSON responses, and was wondering what I'm missing.

I'm using CacheResponsesPlugin to cache the results like so:

with proxy.Proxy(
    ["--cache-requests", "--cache-dir", "/cache/"],
    hostname=ipaddress.IPv4Address("127.0.0.1"),
    port=8890,
    plugins=[ModifyHeadersPlugin, CacheResponsesPlugin],
) as p:

1. The resulting .txt files contain some strange characters within the JSON dictionary:

GET /endpoint?limit=30&offset=0 HTTP/1.1
Host: redacted.com
Accept: */*
User-Agent: Python/3.11 aiohttp/3.11.12
Accept-Encoding: identity

HTTP/1.1 200 OK
Date: redacted
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Build-Id: redacted
Server-Timing: redacted
... other redacted things...

1e6f
{redacted dictionary stuff
bb6
the rest of the same dictionary}
0

Are these characters (1e6f, bb6, 0) due to the JSON being chunked? Am I missing a step to reassemble the dictionary?

Another question:
2. In some other instances, I'm seeing more than 1 response logged in a single file (particularly when I get paginated requests from a single endpoint. The responses from these two endpoints may end up in the same TXT starting with: GET /endpoint?limit=30&offset=0 HTTP/1.1 and GET /endpoint?limit=30&offset=30 HTTP/1.1. Is this intended?

[abhinavsingh]

Are these characters (1e6f, bb6, 0) due to the JSON being chunked? Am I missing a step to reassemble the dictionary?

@InterferencePattern These looks like chunked encoded response. Cache response plugin do not process the response by itself. They are dumped as is. You will need to use the chunked parser to make sense out of these.

Another question: 2. In some other instances, I'm seeing more than 1 response logged in a single file (particularly when I get paginated requests from a single endpoint. The responses from these two endpoints may end up in the same TXT starting with: GET /endpoint?limit=30&offset=0 HTTP/1.1 and GET /endpoint?limit=30&offset=30 HTTP/1.1. Is this intended?

You may end up seeing this if requests are being pipelined i.e. same connection used for multiple requests. This is normal and can be expected. As mentioned above, cache response plugin do not process the data, it simply keeps dumping it to files. But you can always post-process these files as you wish.

Let me know if this helps.