AAPL extensions/context

[tucari]

Hi!

Thanks so much for ksmbd!

I wonder if there is any interest or ongoing work in implementing AAPL SMB2 extensions (as seen in vfs_fruit for Samba)?
Main reason (in my case) would be for the bulk readdirattr calls which -hugely- improve directory traversal speed on macOS.

I've managed to get the context handling and capability advertising working in ksmbd but not yet the readdirattr call.

Thanks
Jon

[namjaejeon]

Thanks for suggestion:)
I don't have Mac book and can't test it although readdirattr is implemented.
Is there any specification of AAPL SMB2 extensions ?

[tucari]

Hi Namjae,

Hope you're well.

Not found a solid spec but been working off the following:
https://www.snia.org/sites/default/files/SDC/2016/presentations/smb/RafalSzczesniak_Sailing_Unchartered_Waters_Apple_OSX_Extensions_SMB2_Wide.pdf

As well as the vfs_fruit module from Samba, specifically:
https://github.com/samba-team/samba/blob/master/source3/modules/vfs_fruit.c

static NTSTATUS check_aapl(vfs_handle_struct *handle,
			   struct smb_request *req,
			   const struct smb2_create_blobs *in_context_blobs,
			   struct smb2_create_blobs *out_context_blobs)
{
	struct fruit_config_data *config;
	NTSTATUS status;
	struct smb2_create_blob *aapl = NULL;
	uint32_t cmd;
	bool ok;
	uint8_t p[16];
	DATA_BLOB blob = data_blob_talloc(req, NULL, 0);
	uint64_t req_bitmap, client_caps;
	uint64_t server_caps = SMB2_CRTCTX_AAPL_UNIX_BASED;
	smb_ucs2_t *model;
	size_t modellen;

	SMB_VFS_HANDLE_GET_DATA(handle, config, struct fruit_config_data,
				return NT_STATUS_UNSUCCESSFUL);

	if (!config->use_aapl
	    || in_context_blobs == NULL
	    || out_context_blobs == NULL) {
		return NT_STATUS_OK;
	}

	aapl = smb2_create_blob_find(in_context_blobs,
				     SMB2_CREATE_TAG_AAPL);
	if (aapl == NULL) {
		return NT_STATUS_OK;
	}

	if (aapl->data.length != 24) {
		DEBUG(1, ("unexpected AAPL ctxt length: %ju\n",
			  (uintmax_t)aapl->data.length));
		return NT_STATUS_INVALID_PARAMETER;
	}

	cmd = IVAL(aapl->data.data, 0);
	if (cmd != SMB2_CRTCTX_AAPL_SERVER_QUERY) {
		DEBUG(1, ("unsupported AAPL cmd: %d\n", cmd));
		return NT_STATUS_INVALID_PARAMETER;
	}

	req_bitmap = BVAL(aapl->data.data, 8);
	client_caps = BVAL(aapl->data.data, 16);

	SIVAL(p, 0, SMB2_CRTCTX_AAPL_SERVER_QUERY);
	SIVAL(p, 4, 0);
	SBVAL(p, 8, req_bitmap);
	ok = data_blob_append(req, &blob, p, 16);
	if (!ok) {
		return NT_STATUS_UNSUCCESSFUL;
	}

	if (req_bitmap & SMB2_CRTCTX_AAPL_SERVER_CAPS) {
		if ((client_caps & SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR) &&
		    (handle->conn->fs_capabilities & FILE_NAMED_STREAMS)) {
			server_caps |= SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR;
			config->readdir_attr_enabled = true;
		}

		if (config->use_copyfile) {
			server_caps |= SMB2_CRTCTX_AAPL_SUPPORTS_OSX_COPYFILE;
			config->copyfile_enabled = true;
		}

		/*
		 * The client doesn't set the flag, so we can't check
		 * for it and just set it unconditionally
		 */
		if (config->unix_info_enabled) {
			server_caps |= SMB2_CRTCTX_AAPL_SUPPORTS_NFS_ACE;
		}

		SBVAL(p, 0, server_caps);
		ok = data_blob_append(req, &blob, p, 8);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}
	}

	if (req_bitmap & SMB2_CRTCTX_AAPL_VOLUME_CAPS) {
		int val = lp_case_sensitive(SNUM(handle->conn));
		uint64_t caps = 0;

		switch (val) {
		case Auto:
			break;

		case True:
			caps |= SMB2_CRTCTX_AAPL_CASE_SENSITIVE;
			break;

		default:
			break;
		}

		if (config->time_machine) {
			caps |= SMB2_CRTCTX_AAPL_FULL_SYNC;
		}

		SBVAL(p, 0, caps);

		ok = data_blob_append(req, &blob, p, 8);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}
	}

	if (req_bitmap & SMB2_CRTCTX_AAPL_MODEL_INFO) {
		ok = convert_string_talloc(req,
					   CH_UNIX, CH_UTF16LE,
					   config->model, strlen(config->model),
					   &model, &modellen);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}

		SIVAL(p, 0, 0);
		SIVAL(p + 4, 0, modellen);
		ok = data_blob_append(req, &blob, p, 8);
		if (!ok) {
			talloc_free(model);
			return NT_STATUS_UNSUCCESSFUL;
		}

		ok = data_blob_append(req, &blob, model, modellen);
		talloc_free(model);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}
	}

	status = smb2_create_blob_add(out_context_blobs,
				      out_context_blobs,
				      SMB2_CREATE_TAG_AAPL,
				      blob);
	if (NT_STATUS_IS_OK(status)) {
		global_fruit_config.nego_aapl = true;
	}

	return status;
}

Converted to ksmbd context, this then shows the share as type: SMB (OSX) in macOS and tries to use their readdirattr extension (i.e stuffing a whole folder into another structure inside the response). This stops macOS from reading the listing then statting each file for attributes and, testing with Apple's SMB server:

Traversing folder structure of 400 folder and 18000 files:
Without AAPL: 28 seconds
With AAPL: 2 seconds

I should have a working PR for this soon if you're interested.

(different issue, I'm currently working from Commit 888b780 rather than master, as master causing a hang and 100% cpu load of ksmbd-io on kernel 6.13, unsure if that's something specific to my config)

Thanks!
Jon

[namjaejeon]

I should have a working PR for this soon if you're interested.

Okay, I am very interested in this! Please send a PR to us:)

I'm currently working from Commit 888b780 rather than master, as master causing a hang and 100% cpu load of ksmbd-io on kernel 6.13, unsure if that's something specific to my config)

Sound great:) Let me know if you have question while checking this issue.

[mmakassikis]

(different issue, I'm currently working from Commit 888b780 rather than master, as master causing a hang and 100% cpu load of ksmbd-io on kernel 6.13, unsure if that's something specific to my config)

is that what the last commit on master is supposed to fix ("ksmbd: fix infinite loop when client send search pattern") ?

AAPL create context is also required for Time Machine backups on SMB shares. However, there are a few more things (durable handles v2 for example). I found this page which may be of interest:

https://developer.apple.com/library/archive/releasenotes/NetworkingInternetWeb/Time_Machine_SMB_Spec/

[namjaejeon]

@mmakassikis

is that what the last commit on master is supposed to fix ("ksmbd: fix infinite loop when client send search pattern") ?

Right. I think so. He should check it with the latest master version.

AAPL create context is also required for Time Machine backups on SMB shares. However, there are a few more things (durable handles v2 for example). I found this page which may be of interest:

Thanks for sharing it.

[tucari]

Hi Both,

Working from latest version now.

Work got a bit busy so still working on this, AAPL context is done as is AAPL Readdirattr (which stuffs unix perms and finderinfo into reserved field and easize). I think Time Machine requires the full sync flag to be considered for use, haven't yet looked at that in the samba vfs_fruit code.

Should have a PR soon for the context and basic ops.

Best
Jon

P.s thanks for the extra Time machine info!
Has been hard to find much info on AAPL extensions so I've mostly wiresharked it from a osx server.

[namjaejeon]

@tucari Sound great and thanks for your work.