[Help]: Multiple MSD in advertisement is wrong on Android

[P0labrD]

Requirements

• I've looked at the README 'Common Problems' section

Have you checked this problem on the example app?

No

FlutterBluePlus Version

1.33.2

Flutter Version

3.24.3

What OS?

Android, iOS

OS Version

iOS 15.8.3, Android 14

Bluetooth Module

ESP32-WROOM-32E

What is your problem?

I'm looking at the advertisementData of my bluetooth device on both Android and iOS, and I can't figure why the format of
manufacturerData and msd properties are different: the manufacturerId key is accessed differently between both os

{
 "android": {
   "advName": "advName",
   "txPowerLevel": null,
   "appearance": null,
   "connectable": true,
   "manufacturerData": {
     "0": [5, 54 /* ... */],
     "12345": []
   },
   "msd": [
     [0, 0, 5, 54 /* ... */],
     [48, 57]
   ],
   "serviceData": {},
   "serviceUuids": []
 },

 "ios": {
   "advName": "advName",
   "txPowerLevel": null,
   "appearance": null,
   "connectable": true,
   "manufacturerData": {
     "12345": [0, 0, 5, 54 /* ... */]
   },
   "msd": [[48, 57, 0, 0, 5, 54 /* ... */]],
   "serviceData": {},
   "serviceUuids": []
 }
}

I can't find any documentation on this difference, so before introducing platform specific code in production, I would like to be sure it is developed as intented.

Thank you very much!

Logs

{
  "android": {
    "advName": "advName",
    "txPowerLevel": null,
    "appearance": null,
    "connectable": true,
    "manufacturerData": {
      "0": [5, 54 /* ... */],
      "12345": []
    },
    "msd": [
      [0, 0, 5, 54 /* ... */],
      [48, 57]
    ],
    "serviceData": {},
    "serviceUuids": []
  },

  "ios": {
    "advName": "advName",
    "txPowerLevel": null,
    "appearance": null,
    "connectable": true,
    "manufacturerData": {
      "12345": [0, 0, 5, 54 /* ... */]
    },
    "msd": [[48, 57, 0, 0, 5, 54 /* ... */]],
    "serviceData": {},
    "serviceUuids": []
  }
}

[chipweinberger]

they should not be different. please open a PR if there is a bug.

msd = list of full raw data, split by manufacturer
manufacturerData = map of <manufactureId, data without manufactureId>

[chipweinberger]

also, where is this json coming from?

please print out the advertisement data directly and report back

print(advertisementData)

  @override
  String toString() {
    return 'AdvertisementData{'
        'advName: $advName, '
        'txPowerLevel: $txPowerLevel, '
        'appearance: $appearance, '
        'connectable: $connectable, '
        'manufacturerData: $manufacturerData, '
        'serviceData: $serviceData, '
        'serviceUuids: $serviceUuids'
        '}';
  }

[P0labrD]

I wrote the json with what I saw in my debugger, anonymizing the data
Here's the real (still anonymized) print ouput for iOS:

flutter: AdvertisementData{advName: product-name, txPowerLevel: null, appearance: null, connectable: true, manufacturerData: {12345: [0, 0, 5, 54, 48, 48, 85, 48, 54, 57, 48, 57, 50, 49]}, serviceData: {}, serviceUuids: []}

And for Android:

I/flutter (19040): AdvertisementData{advName: product-name, txPowerLevel: null, appearance: null, connectable: true, manufacturerData: {0: [5, 54, 48, 48, 85, 48, 54, 57, 48, 57, 50, 49], 12345: []}, serviceData: {}, serviceUuids: []}

So if I understand correctly, my device is seen with one manufacturerId 12345 for iOS, but with two for android, 0 and 12345, but with the rawData being on the wrong side

I'm not a hardware developer myself, but I could ask the one who did develop on the chip what he implemented in order to give good reproduction info.

Here's the plugin method I call to start scanning:

FlutterBluePlus.startScan(
  timeout: timeout,
  withMsd: [MsdFilter(12345)],
  removeIfGone: const Duration(seconds: 3),
  continuousUpdates: true,
);

[chipweinberger]

the android code is here. you should debug it further :)

particularly, you should print the result of : byte[] bytes = adv.getBytes();

it probably has a bug. iOS is probably correct.

    Map<Integer, byte[]> getManufacturerSpecificData(ScanRecord adv) {
        byte[] bytes = adv.getBytes();
        Map<Integer, byte[]> manufacturerDataMap = new HashMap<>();
        int n = 0;
        while (n < bytes.length) {

            // layout:
            // n[0] = fieldlen
            // n[1] = datatype (MSD)
            // n[2] = manufacturerId (low)
            // n[3] = manufacturerId (high)
            // n[4] = data...
            int fieldLen = bytes[n] & 0xFF;

            // no more or malformed data
            if (fieldLen <= 0) {
                break;
            }

            // Ensuring we don't go past the bytes array
            if (n + fieldLen >= bytes.length) {
                break;
            }

            int dataType = bytes[n + 1] & 0xFF;

            // Manufacturer Specific Data magic number
            // At least 3 bytes: 2 for manufacturer ID & 1 for dataType
            if (dataType == 0xFF && fieldLen >= 3) {

                // Manufacturer Id
                int high = (bytes[n + 3] & 0xFF) << 8;
                int low = (bytes[n + 2] & 0xFF);
                int manufacturerId = high | low;

                // the length of the msd data,
                // excluding manufacturerId & dataType
                int msdLen = fieldLen - 3;

                // ptr to msd data
                // excluding manufacturerId & dataType
                int msdPtr = n + 4;

                // add to map
                if (manufacturerDataMap.containsKey(manufacturerId)) {
                    // If the manufacturer ID already exists, append the new data to the existing list
                    byte[] existingData = manufacturerDataMap.get(manufacturerId);
                    byte[] mergedData = new byte[existingData.length + msdLen];
                    // Merge arrays
                    System.arraycopy(existingData, 0, mergedData, 0, existingData.length);
                    System.arraycopy(bytes, msdPtr, mergedData, existingData.length, msdLen);
                    manufacturerDataMap.put(manufacturerId, mergedData);
                } else {
                    // Otherwise, put the new manufacturer ID and its data into the map
                    byte[] data = new byte[msdLen];
                    // Starting from n+4 because manufacturerId occupies n+2 and n+3
                    System.arraycopy(bytes, msdPtr, data, 0, data.length);
                    manufacturerDataMap.put(manufacturerId, data);
                }
            }

            n += fieldLen + 1;
        }

        return manufacturerDataMap;
    }
    ```

[chipweinberger]

@MrCsabaToth

[MrCsabaToth]

@MrCsabaToth

On Sunday I might have a little time to inspect the related code. I don't have an iOS device though, so it'll be Android and code inspection. As far as I know last time I modified it because of the FedEx tag, it meant to be the same. It had an off by one byte bug you corrected later.

[chipweinberger]

i inspected the code, looked fine to me.

[MrCsabaToth]

Hey @P0labrD, could you screenshot the raw advertisement data displayed in nRF Connect on Android (https://play.google.com/store/apps/details?id=no.nordicsemi.android.mcp) vs iOS (https://apps.apple.com/us/app/nrf-connect-for-mobile/id1054362403) so we can have even more munition? The raw data looks like this #785 (comment) and this #785 (comment)

[P0labrD]

Hello @MrCsabaToth, thank you for your time.

Here are the two raw advertisement data displayed:

Android	iOS

[P0labrD]

Update: my hardware developer apparently did not fully understand the Core Blutooth Specification Part A 1.4. Manufacturer Specific Data: as seen on the android screen above, he decided to put the manufacturerId on another "lane" as the specific data, whereas it should be but in the 2 first bytes of the 16 bytes of the manufacturerData.

What I don't understand at the moment is how iOS CoreBluetooth works, and how it "magically" did the matching itself.

[chipweinberger]

yes we should still try to fix them so they match

[chipweinberger]

So I think I have the answer.

According to ChatGPT, in BLE you should only have one MSD in an advertisement. Having multiple breaks spec.

So that's why iOS just concatenates all the data together. This is clear in the code. manufData just an array, not a map.

I don't know how we missed this. iOS code:

- (NSDictionary *)bmScanAdvertisement:(NSString*)remoteId
             advertisementData:(NSDictionary<NSString *, id> *)advertisementData
                          RSSI:(NSNumber *)RSSI
{
    NSString     *advName        = advertisementData[CBAdvertisementDataLocalNameKey];
    NSNumber     *connectable    = advertisementData[CBAdvertisementDataIsConnectable];
    NSNumber     *txPower        = advertisementData[CBAdvertisementDataTxPowerLevelKey];
    NSData       *manufData      = advertisementData[CBAdvertisementDataManufacturerDataKey];
    NSArray      *serviceUuids   = advertisementData[CBAdvertisementDataServiceUUIDsKey];
    NSDictionary *serviceData    = advertisementData[CBAdvertisementDataServiceDataKey];

    // Manufacturer Data
    NSDictionary* manufDataB = nil;
    if (manufData != nil && manufData.length >= 2) {
        
        // first 2 bytes are manufacturerId (little endian)
        uint8_t bytes[2];
        [manufData getBytes:bytes length:2];
        unsigned short manufId = (unsigned short) (bytes[0] | bytes[1] << 8);

        // trim off first 2 bytes
        NSData* trimmed = [manufData subdataWithRange:NSMakeRange(2, manufData.length - 2)];
        NSString* hex = [self convertDataToHex:trimmed];
        
        manufDataB = @{
            @(manufId): hex,
        };
    }

So we should fix this on Android, and remove all the "map" stuff. And fix/break the API.

@MrCsabaToth , can you do this?

The public facing API should be.

adv.msd.raw
adv.msd.payload
adv.msd.id

Also lets simplify the BmScanAdvertisement to just pass the raw concatenated msd data. We'll do the 2 byte parsing in Dart.