[FEATURE] Passing additional custom user-agent fields for tracing use

[snazzyfox]

Problem Statement

I'm building a tool that allow our users to request access to databricks resources. I've seen that databricks cli provides tracing information in its user agent that's exposed in the audit log table (see below) which is very useful in identifying which changes originated from which user action outside databricks. I would like to be able to do the same with my app - attach tracing information to each call to databricks via the user agent string, so I know which actions on my side resulted in each databricks api call.

The sdk currently computes the full user agent string at client construction time, and it's not possible to change that string (without accessing many private instance variables at least) after the client is initialized.

Given that using UAs for auditing is already done in databricks-cli, I think it would make sense for this sdk to also support doing the same for our custom built applications.

Proposed Solution

Allow a way to provide custom user agents. The exact way this can be implemented can vary a lot; I think the most likely (but specific) option is

# before
client.grants.update(...)
# after
client.with_user_agent({"trace-id": trace_id}).grants.update(...)

This can, of course, be made more general for other use cases that may require modifying the request at a lower level for individual calls, such as:

client.with_config(user_agent={}).grants.update(...)
client.with_hook(before_request=lambda req: ...).grants.update(...)

Where the with_* methods in the config simply returns a copy of the client with the relevant config options applied. This should not update the client in-place.

Adding additional parameter(s) to every single api call method, although makes sense from a user point of view, probably would introduce a maintenance nightmare (esp if more than one of these are added later) so that's probably not a good idea.

Additional Context

here is an example of an user agent string found in databricks audit log:

cli/0.238.0 databricks-sdk-go/0.54.0 go/1.23.4 os/linux cmd/bundle_validate cmd-exec-id/<a uuid here> auth/oauth-m2m cicd/gitlab

This user agent string not only identifies databricks-cli as the application the request was sent from, and the auth scheme, which databricks-sdk does; it also adds in additional useful information specific to that particular request, including the CLI command that triggered the api call (bundle validate), and a command ID (which can be used to correlate to CI logs)

I've considered extending the client classes in my own code to override this behavior. But because the user agent is hardcoded at the lowest-level _ApiClient class with the assumption that it's a static string provided at class construction time, with the do method force-setting it in headers assuming it's a string, there really is no place where this logic can be added per-request without essentially duplicating most client classes, including private ones. The amount of code I have to duplicate out of the sdk would make this very hard to maintain going forward.