Barcode ML kit not working on specific android device

[cagrialta]

Describe your issue. If applicable, add screenshots to help explain your problem.

In some android devices barcode scan is not working! (For now samsung galaxy tab a7 Android 12 and tab s9+ Android 13) There is no specific error just return empty barcode list. But there is specific log on devices which not working.

I/tflite  (23965): Initialized TensorFlow Lite runtime.
E/libc    (23965): Access denied finding property "ro.mediatek.platform"
I/tflite  (23965): Created TensorFlow Lite XNNPACK delegate for CPU.

This logs are not showing on working devices.

Here my camera controller how initialized

 var cameraController = CameraController(
    ultraWideCamera ?? backCameras.firstOrNull ?? cameras.first,
    ResolutionPreset.veryHigh,
    enableAudio: false,
    imageFormatGroup: Platform.isIOS ? ImageFormatGroup.bgra8888 : ImageFormatGroup.nv21,
  );

Steps to reproduce.

Run on specific devices ( now we only know Tab A7 & Tab s9)

What is the expected result?

Normal barcode scanning behaviout

Did you try our example app?

Yes

Is it reproducible in the example app?

Yes

Reproducible in which OS?

Android

Flutter/Dart Version?

[✓] Flutter (Channel stable, 3.22.1, on macOS 14.4 23E214 darwin-arm64, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 15.4)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2023.3)
[✓] VS Code (version 1.89.1)

Plugin Version?

google_mlkit_barcode_scanning: ^0.12.0
camera: ^0.11.0

[cagrialta]

After I added this nv21 converting function it worked! So somehow in some android devices even you give NV21 image format to controller it is not returning nv21, or it is not proper nv21. But in somewhere we should get format exception.

Here my convert function

extension Nv21Converter on CameraImage {
  Uint8List getNv21Uint8List() {
    var width = this.width;
    var height = this.height;

    var yPlane = planes[0];
    var uPlane = planes[1];
    var vPlane = planes[2];

    var yBuffer = yPlane.bytes;
    var uBuffer = uPlane.bytes;
    var vBuffer = vPlane.bytes;

    var numPixels = (width * height * 1.5).toInt();
    var nv21 = List<int>.filled(numPixels, 0);

    // Full size Y channel and quarter size U+V channels.
    int idY = 0;
    int idUV = width * height;
    var uvWidth = width ~/ 2;
    var uvHeight = height ~/ 2;
    // Copy Y & UV channel.
    // NV21 format is expected to have YYYYVU packaging.
    // The U/V planes are guaranteed to have the same row stride and pixel stride.
    // getRowStride analogue??
    var uvRowStride = uPlane.bytesPerRow;
    // getPixelStride analogue
    var uvPixelStride = uPlane.bytesPerPixel ?? 0;
    var yRowStride = yPlane.bytesPerRow;
    var yPixelStride = yPlane.bytesPerPixel ?? 0;

    for (int y = 0; y < height; ++y) {
      var uvOffset = y * uvRowStride;
      var yOffset = y * yRowStride;

      for (int x = 0; x < width; ++x) {
        nv21[idY++] = yBuffer[yOffset + x * yPixelStride];

        if (y < uvHeight && x < uvWidth) {
          var bufferIndex = uvOffset + (x * uvPixelStride);
          //V channel
          nv21[idUV++] = vBuffer[bufferIndex];
          //V channel
          nv21[idUV++] = uBuffer[bufferIndex];
        }
      }
    }
    return Uint8List.fromList(nv21);
  }
}

[SuryaAbyss]

I can contribute on this issue , can you please assign it to me @flutter-ml

[fbernaly]

@SuryaAbyss: Assigned to you. Thanks.

[gauravRNDev]

where did you used it because i m still getting error

InputImage? _inputImageFromCameraImage(CameraImage image) {
   // get image rotation
   // it is used in android to convert the InputImage from Dart to Java
   // `rotation` is not used in iOS to convert the InputImage from Dart to Obj-C
   // in both platforms `rotation` and `camera.lensDirection` can be used to compensate `x` and `y` coordinates on a canvas
   final camera = widget.camera[0];
   final sensorOrientation = camera.sensorOrientation;
   InputImageRotation? rotation;
   if (Platform.isIOS) {
     rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
   } else if (Platform.isAndroid) {
     var rotationCompensation =
         _orientations[_controller!.value.deviceOrientation];
     if (rotationCompensation == null) return null;
     if (camera.lensDirection == CameraLensDirection.front) {
       // front-facing
       rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
     } else {
       // back-facing
       rotationCompensation =
           (sensorOrientation - rotationCompensation + 360) % 360;
     }
     rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
   }
   if (rotation == null) return null;

   // get image format
   final format = InputImageFormatValue.fromRawValue(image.format.raw);
   // validate format depending on platform
   // only supported formats:
   // * nv21 for Android
   // * bgra8888 for iOS
   if (format == null ||
       // (Platform.isAndroid && format != InputImageFormat.nv21) ||
       (Platform.isIOS && format != InputImageFormat.bgra8888)) return null;

   // since format is constraint to nv21 or bgra8888, both only have one plane
   // if (image.planes.length != 1) return null;
   final plane = image.planes.first;
   final nv21Image = image.getNv21Uint8List();

   // compose InputImage using bytes
   return InputImage.fromBytes(
     bytes: Platform.isAndroid ? nv21Image : plane.bytes,
     metadata: InputImageMetadata(
       size: Size(image.width.toDouble(), image.height.toDouble()),
       rotation: rotation, // used only in Android
       format: format, // used only in iOS
       bytesPerRow: plane.bytesPerRow, // used only in iOS
     ),
   );
 }

[cagrialta]

Before convert input image.

[gauravRNDev]

Ah, I see! The issue was with the format being InputImageFormat.yuv_420_888, which was passed to the metadata and caused the error.
Although it is reported for the ios I don't why is causing issue in the android also

[mayudevID]

where did you put the code for the nv21 conversion? @gauravRNDev

[JakubBatel]

Here is a more complete example using the code from @cagrialta in case anyone is still struggling with where to put it and how to make it work. The following code works on both Android and iOS.

extension Nv21Converter on CameraImage {
  // Code from @cagrialta 
}

const _orientations = {
  DeviceOrientation.portraitUp: 0,
  DeviceOrientation.landscapeLeft: 90,
  DeviceOrientation.portraitDown: 180,
  DeviceOrientation.landscapeRight: 270,
};

InputImage? cameraImageToInputImage(
    CameraImage image,
    CameraDescription camera,
    DeviceOrientation deviceOrientation,
  ) {
    final format = InputImageFormatValue.fromRawValue(image.format.raw);
    if (format == null) {
      return null;
    }
    final plane = image.planes.firstOrNull;
    if (plane == null) {
      return null;
    }

    final sensorOrientation = camera.sensorOrientation;

    final InputImageRotation? rotation;
    if (Platform.isIOS) {
      rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
    } else if (Platform.isAndroid) {
      var rotationCompensation = _orientations[deviceOrientation];
      if (rotationCompensation == null) {
        return null;
      }
      if (camera.lensDirection == CameraLensDirection.front) {
        // front-facing
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
      } else {
        // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
      }
      rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
    } else {
      rotation = null;
    }

    if (rotation == null) {
      return null;
    }

    final Uint8List bytes;
    if (Platform.isAndroid) {
      bytes = image.getNv21Uint8List();
    } else {
      final allBytes = WriteBuffer();
      for (final plane in image.planes) {
        allBytes.putUint8List(plane.bytes);
      }
      bytes = allBytes.done().buffer.asUint8List();
    }

    return InputImage.fromBytes(
      bytes: bytes,
      metadata: InputImageMetadata(
        size: Size(image.width.toDouble(), image.height.toDouble()),
        rotation: rotation, // used only in Android
        format: Platform.isAndroid ? InputImageFormat.nv21 : format, // Hardcode format for Android
        bytesPerRow: plane.bytesPerRow, // used only in iOS
      ),
    );
  }

Then simply call it wherever you have the input image like this

final inputImage = cameraImageToInputImage(image, camera, cameraController.value.deviceOrientation);
if (inputImage == null) {
  return;
}
final barcodes = await _barcodeScanner.processImage(inputImage);

Don't forget to hardcode the image format for Android otherwise it uses yuv420 (at least on my device) which doesn't work and causes the PlatformException even if you put bytes that are formatted as nv21.

[simeonangelov94]

Here is a more complete example using the code from @cagrialta in case anyone is still struggling with where to put it and how to make it work. The following code works on both Android and iOS.

extension Nv21Converter on CameraImage {
  // Code from @cagrialta 
}

const _orientations = {
  DeviceOrientation.portraitUp: 0,
  DeviceOrientation.landscapeLeft: 90,
  DeviceOrientation.portraitDown: 180,
  DeviceOrientation.landscapeRight: 270,
};

InputImage? cameraImageToInputImage(
    CameraImage image,
    CameraDescription camera,
    DeviceOrientation deviceOrientation,
  ) {
    final format = InputImageFormatValue.fromRawValue(image.format.raw);
    if (format == null) {
      return null;
    }
    final plane = image.planes.firstOrNull;
    if (plane == null) {
      return null;
    }

    final sensorOrientation = camera.sensorOrientation;

    final InputImageRotation? rotation;
    if (Platform.isIOS) {
      rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
    } else if (Platform.isAndroid) {
      var rotationCompensation = _orientations[deviceOrientation];
      if (rotationCompensation == null) {
        return null;
      }
      if (camera.lensDirection == CameraLensDirection.front) {
        // front-facing
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
      } else {
        // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
      }
      rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
    } else {
      rotation = null;
    }

    if (rotation == null) {
      return null;
    }

    final Uint8List bytes;
    if (Platform.isAndroid) {
      bytes = image.getNv21Uint8List();
    } else {
      final allBytes = WriteBuffer();
      for (final plane in image.planes) {
        allBytes.putUint8List(plane.bytes);
      }
      bytes = allBytes.done().buffer.asUint8List();
    }

    return InputImage.fromBytes(
      bytes: bytes,
      metadata: InputImageMetadata(
        size: Size(image.width.toDouble(), image.height.toDouble()),
        rotation: rotation, // used only in Android
        format: Platform.isAndroid ? InputImageFormat.nv21 : format, // Hardcode format for Android
        bytesPerRow: plane.bytesPerRow, // used only in iOS
      ),
    );
  }

Then simply call it wherever you have the input image like this

final inputImage = cameraImageToInputImage(image, camera, cameraController.value.deviceOrientation);
if (inputImage == null) {
  return;
}
final barcodes = await _barcodeScanner.processImage(inputImage);

Don't forget to hardcode the image format for Android otherwise it uses yuv420 (at least on my device) which doesn't work and causes the PlatformException even if you put bytes that are formatted as nv21.

This solution is quite good but still have issues when testing on Android - google_mlkit_object_detection: ^0.14.0
--> The solutions works with camera: ^0.11.0 but for some reason the rotation of the camera is switched by 90 degrees. I believe is an issue with the camera package. The solution was to downgrade to camera: ^0.10.6 but then trying your code on Android device is again returning an error below. I already tested with iPhone has no issue.

It looks like using Camera 0.10.6 - image.planes has only one item in the array instead of 3.

CameraImage? img;
doObjectDetectionOnFrame() async {
InputImage? frameImg;

frameImg = cameraImageToInputImage(
    img!, cameras[0], controller!.value.deviceOrientation);

// getInputImage();
if (frameImg != null) {
  List<DetectedObject> objects =
      await objectDetector.processImage(frameImg);
  // print("len= ${objects.length}");
  setState(() {
    _scanResults = objects;
  });

  // print('length results: ' + objects.length.toString());
}
//   else {
//   print('InputImage is null');
// }
isBusy = false;

}

E/flutter (17521): #0 _Array.[] (dart:core-patch/array.dart)
E/flutter (17521): #1 Nv21Converter.getNv21Uint8List
E/flutter (17521): #2 cameraImageToInputImage
E/flutter (17521): #3 _ObjectDetectionFeedState.doObjectDetectionOnFrame
E/flutter (17521): #4 _ObjectDetectionFeedState.initializeCamera.

I fixed it initializing the controller with yuv420 instead of nv21

controller = CameraController(
  cameras.first,
  ResolutionPreset.high,
  imageFormatGroup: Platform.isAndroid
      ? ImageFormatGroup.yuv420
      : ImageFormatGroup.bgra8888,
);

[JakubBatel]

@simeonangelov94 Thanks! Actually I run into the same issue right now when tested on different physical device. I applied your suggestions and it works great.