[API Proposal]: Swift VisionKit APIs

[MicahKimel]

Background and motivation

Currently swift bindings for CryptoKit have been implemented, I would like to work on contributing the native swift VisionKit api. This could turn into a rather large project, but to start I would like to implement the DataScannerViewController for barcode scanning in C# MAUI.

Apple is continually adding swift native packages to iOS, if MAUI plans to stay relevant and up to date with new iOS features objective c headers for swift native packages need to be added to runtimes anyways.

Where are the native swift bindings going to be put? Should they go in https://github.com/dotnet/runtime/tree/ec118c7e798862fd69dc7fa6544c0d9849d32488/src/native/libs/System.Native?

API Proposal

[BaseType (typeof(NSObject))]
interface DataScannerViewController
{
	// -(UIViewController * _Nonnull)getViewController __attribute__((warn_unused_result("")));
	[Export ("getViewController")]
	UIViewController ViewController { get; }

	// -(void)setScanCallbackWithCallback:(void (^ _Nonnull)(NSArray<NSString *> * _Nonnull))callback;
	[Export ("setScanCallbackWithCallback:")]
	void SetScanCallbackWithCallback (Action<string []> callback);

	// -(void)setScanUpdatedCallbackWithCallback:(void (^ _Nonnull)(NSArray<NSString *> * _Nonnull))callback;
	[Export ("setScanUpdatedCallbackWithCallback:")]
	void SetScanUpdatedCallbackWithCallback (Action<string []> callback);

	// -(void)startScan;
	[Export ("startScan")]
	void StartScan ();

	// -(void)stopScan;
	[Export ("stopScan")]
	void StopScan ();
}

API Usage

using VisionKit;

ViewController barcodeScanner = DataScannerViewController();
barcodeScanner.StartScan();
barcodeScanner.SetScanCallbackWithCallback(result =>
{
    Console.WriteLine("Scanned items:");
    foreach (var item in result)
    {
        Console.WriteLine(item);
    }
});

Alternative Designs

No response

Risks

No response

[huoyaoyuan]

The runtime repo doesn't contain any GUI related component. System.Native only contains native binding for fundamental stuff that needs cross-platform support. It can't be consumed out of the repo.

Apple OS-specific binding lives in https://github.com/xamarin/xamarin-macios . The tooling for swift binding is currently in develop at https://github.com/dotnet/runtimelab/tree/feature/swift-bindings . Xamarin-macios should then consume the binding tooling when it becomes stable.

[jkotas]

cc @vitek-karas @kotlarmilos @jkurdek

[kotlarmilos]

@MicahKimel Thanks for reaching out.

We are currently working on a tool that generates C# bindings from an input Swift framework, such as VisionKit: https://github.com/dotnet/runtimelab/tree/feature/swift-bindings

It is still in the experimental phase and is not officially supported. You can definitely contribute in two ways:

• Provide end-user scenarios, like this one
• Contribute to the tool itself

API Usage

I assume this code snippet is a proposal for how it could be used from C#, but I didn't find the callback method. Could you provide the Swift code snippet that you want to implement?

Based on this, we can determine if this is something the tool already supports and decide on the next steps.

[MicahKimel]

Thanks @kotlarmilos!

I'll check out the repo! Should something like this be implemented in src/Swift.Bindings/tests/IntegrationTests/FunctionalTests? If I understand correctly I think the CMakeLists.txt one level up is creating .dylib from those swift files so that then C# can use the swift mangled names.

Here's some swift code to go along with the above C# api,

import Vision
import VisionKit

@objc(DataScannerViewController)
public class ScanControllerWrapper : NSObject, DataScannerViewControllerDelegate
{
    private var viewController: DataScannerViewController
    private var scannerCallback: ((_ codes: [String]) -> Void)? = nil
    private var scannerUpdateCallback: ((_ codes: [String]) -> Void)? = nil

    @objc
    @MainActor override init()
    {
        let supportedBarcodes = VNDetectBarcodesRequest.init().symbologies
        let viewController = DataScannerViewController.init(recognizedDataTypes:[.barcode(symbologies:supportedBarcodes)])
        self.viewController = viewController

        super.init()

        self.viewController.delegate = self
    }

    @objc
    public func getViewController() -> UIViewController
    {
        return self.viewController
    }

    private func handleBarcodeCallback(barcodes: [RecognizedItem], callback: ((_ codes: [String]) -> Void)?)
    {
        if(callback != nil)
        {
            let scannedBarcodes: [String] = barcodes.map
            { (item) -> String in
                switch(item)
                {
                case .barcode(let barcode):
                        return barcode.payloadStringValue ?? "<barcode not read>"
                    default:
                        return "<not a barcode>"
                }
            }
            callback!(scannedBarcodes)
        }
    }

    public func dataScanner(_ dataScanner: DataScannerViewController, didAdd addedItems: [RecognizedItem], allItems: [RecognizedItem])
    {
        handleBarcodeCallback(barcodes: addedItems, callback: self.scannerCallback)
    }

    public func dataScanner(_ dataScanner: DataScannerViewController, didUpdate updatedItems: [RecognizedItem], allItems: [RecognizedItem])
    {
        handleBarcodeCallback(barcodes: updatedItems, callback: self.scannerUpdateCallback)
    }

    @objc
    @MainActor public func setScanCallback(callback: @escaping ([String]) -> Void)
    {
        self.scannerCallback = callback
    }

    @objc
    @MainActor public func setScanUpdatedCallback(callback: @escaping ([String]) -> Void)
    {
        self.scannerUpdateCallback = callback
    }

    @objc
    @MainActor public func startScan()
    {
        try? // NOTE: hides errors
        self.viewController.startScanning()
    }

    @objc
    @MainActor public func stopScan()
    {
        self.viewController.stopScanning()
    }
}

[kotlarmilos]

Should something like this be implemented in src/Swift.Bindings/tests/IntegrationTests/FunctionalTests?

Correct. Ideally, we run the tooling to generate the bindings instead of adding a manual bindings.

If I understand correctly I think the CMakeLists.txt one level up is creating .dylib from those swift files so that then C# can use the swift mangled names.

The generated bindings call into the native library. The tooling consumes .ABI.json file which is a result of swiftc compilation.

Here's some swift code to go along with the above C# api,

Thanks, I will run the tooling and check what we can supported atm.