Support of converting YUV_240_888 to Bitmap

[siralam]

Hello, first of all thanks for your great library!

I am now trying to do something to each frame returned by Android Camera 2 API (Which I guess most user of this library do).

Note that I have already specified YUV_240_888 for the ImageReader:

        mFrameImageReader = ImageReader.newInstance(largest.getWidth() / 4, largest.getHeight() / 4,
                ImageFormat.YUV_420_888, 2);

And in onImageAvailable:

        @Override
        public void onImageAvailable(ImageReader reader) {
            Image image = reader.acquireNextImage();
            try {
                if (onFrameCallback != null) {
                    ByteBuffer buffer = image.getPlanes()[0].getBuffer();
                    byte[] bytes = new byte[buffer.capacity()];
                    buffer.get(bytes);
                    Bitmap frameBitmap = YuvToRgb.yuvToRgb(rs, bytes, width, height);
                    Log.i("onFrame", frameBitmap.getWidth() + ", " + frameBitmap.getHeight());
                }
            } finally {
                image.close();
            }
        }

When I attach a debugger, I can see the conversion line is executed but before the logging line, the app crashes without any stacktrace, instead giving this error message:

Fatal signal 7 (SIGBUS), code 2, fault addr 0xbe68f040 in tid 1699

Do you have any idea / any working example with Camera 2 API?

By the way, Camera 1 returns NV21 format and I have tried with Camera 1 API and NV21 works perfectly. But Camera 2 does not support NV21...

[silvaren]

Hi @siralam !

First, without further look at the actual running code, I would like to point out that errors like these most often comes from unexpected-sized input. The input for the NV21 byte array input should be (width * height) * 1.5 bytes (check this diagram, please note UV byte order is mistakingly reversed though). Looking at your code it looks like you are only taking the Y-plane of your YUV_420_888 image, which is only a (width * height)long byte array, thus causing the crash. You would need to also pass the right UV data in order for any conversion to work.

Then, there is the bigger issue of actual conversion support of the format you are trying to do on this library. As far as I know, Camera 2 API provides these YUV_420_888 images in a separate plane presentation. Note in the referred documentation that each plane has pixel and row stride, meaning several formats could be encoded in that Image object you are reading from, NV21 being just one possibility. So, in general, the library doesn't support conversion from these objects provided from Camera 2 API, unless you somehow convert those possible formats to a NV21 compliant one, or make sure the one you are receiving from the API is NV21 (and then you still need to do some byte array stitching).

In short, it looks like the big issue is actually lack of conversion support for Camera 2 API provided images. I think this is a legitimate request and a valuable one, and while I may not have the time to do it shortly, I'd be glad to mark it as "Help wanted" and welcome community contributions to it. So please if you'd like that support to be implemented please open that issue so that people could eventually contribute to it :)

[siralam]

Hi @silvaren ! Thanks for your response, actually I have managed to do it, but did not use any RenderScript.

What I did is convert the byte[] from Camera 2 API to NV21 format, and then use your library as usual to convert NV21 to Bitmap.

And below is how I convert that byte[] to NV21:

    public static byte[] YUV420toNV21(Image image) {
        Rect crop = image.getCropRect();
        int format = image.getFormat();
        int width = crop.width();
        int height = crop.height();
        Image.Plane[] planes = image.getPlanes();
        byte[] data = new byte[width * height * ImageFormat.getBitsPerPixel(format) / 8];
        byte[] rowData = new byte[planes[0].getRowStride()];

        int channelOffset = 0;
        int outputStride = 1;
        for (int i = 0; i < planes.length; i++) {
            switch (i) {
                case 0:
                    channelOffset = 0;
                    outputStride = 1;
                    break;
                case 1:
                    channelOffset = width * height + 1;
                    outputStride = 2;
                    break;
                case 2:
                    channelOffset = width * height;
                    outputStride = 2;
                    break;
            }

            ByteBuffer buffer = planes[i].getBuffer();
            int rowStride = planes[i].getRowStride();
            int pixelStride = planes[i].getPixelStride();

            int shift = (i == 0) ? 0 : 1;
            int w = width >> shift;
            int h = height >> shift;
            buffer.position(rowStride * (crop.top >> shift) + pixelStride * (crop.left >> shift));
            for (int row = 0; row < h; row++) {
                int length;
                if (pixelStride == 1 && outputStride == 1) {
                    length = w;
                    buffer.get(data, channelOffset, length);
                    channelOffset += length;
                } else {
                    length = (w - 1) * pixelStride + 1;
                    buffer.get(rowData, 0, length);
                    for (int col = 0; col < w; col++) {
                        data[channelOffset] = rowData[col * pixelStride];
                        channelOffset += outputStride;
                    }
                }
                if (row < h - 1) {
                    buffer.position(buffer.position() + rowStride - length);
                }
            }
        }
        return data;
    }

I believe the above method is a bit slower compared to using RenderScript. But since I don't know how to implement a RenderScript by myself, I have to use this method.

But I also found someone posted how to use RenderScript to convert YUV_240_888 to RGB, you may have a try to see if it works.
Since it directly converts YUV_240_888 to RGB, and use RenderScript instead of Java code, I think it should be much faster than the current method I am using.

Thanks!

[silvaren]

The renderscript version you pointed out looks great @siralam ! I might give it a try at some point to include it in the library :) Thanks!

[siralam]

Thanks @silvaren , looking forward to your good news :D

[in-skyadav]

I am not able to create instance of RenderScript . below is the code

byte[] bytes = YUV420toNV21(image);
RenderScript rs = new RenderScript.create(getApplicationContext());
Bitmap outputBitmap = Nv21Image.nv21ToBitmap(rs, bytes, image.getWidth(), image.getHeight());

Error : Cannot resolve symbol 'create'