Camera2 API on MWC 2015 devices: Galaxy S6, HTC One M9 and more Lollipop devices

Since Google announced Android 5 Lollipop at Google I/O 2014, mobile photography and video enthusiasts are hoping for a sizable upgrade of their smartphone’s camera capabilities.

History

The historic Android Camera programming interface was designed before the very first OS release Astro and replicates the logic of a point-and-shoot camera. Except that instead of human fingers pressing buttons, apps were doing so programmatically.

Android Camera API

This first design, while extensible in features via a liberal text key-value interface proved increasingly inefficient for the evolving needs of camera applications.
One revolution ongoing is the advent of computational photography which is often based on the capture of multiple frames. You likely heard and used one of the common applications like HDR (High Dynamic Range), superresolution, both joined, low-light mode, eraser, best face.
Android also started powering connected cameras with full-fledged variable aperture optical zooms (ex: Samsung Galaxy Camera) or large sensors seen before on DSLR-size bridges (ex: Panasonic CM1 with a 1″ sensor like Sony RX10).

While Android Camera  up to KitKat was too limited, we’ve seen manufacturers extending capabilities on each generation extensively but solely via undocumented, proprietary or even obfuscated APIs, allowing access to advanced features only to their own Camera application.

Google’s new approach

Camera2 pipeline

For Lollipop, Google decided to redesign from scratch a new Camera API suitable for current needs, and named that Camera2.
The logic is entirely different: the camera system becomes a pipeline you feed with requests and give you in output individual frames in one or several buffers of your choice.

Two videos from Google I/O 2014 and a DevBytes explain this API.

Some of the main benefits are:

  • Each frame parameters can be controlled separately
  • With no distinction between photo and video, the same controls are available in both, for the benefits of users and application developers.
  • Multiple simultaneous output buffers allow JPEG+RAW output.
  • Full sensor resolution capture at 30 FPS is suitable for computational photography

With Camera2 API in Android 5.0 Lollipop release, Google also declared the earlier Camera API deprecated, a sign that both are not meant to cohabit: the new one replace the old one now on its way out.

Since Lollipop announcement, Google released Android 5.0 updates for multiple Nexus mobile devices: Nexus 5, Nexus 7 2012 & 2013, Nexus 4, Nexus 10. Nexus 9 and Nexus 6 also came out with Lollipop out of the box.
The Nexus 5 served as reference device for Camera2 full implementation, and Nexus 7 2013 as legacy.

Full, Limited and Legacy are three class of Camera2 supported devices. Because Camera2 is such a departure from anything available before and the new API implementation is mandatory, Google had to compromise and add a lot of flexibility in how much capabilities manufacturers actually expose.

If Google required every manufacturer to implement a complete set of manual exposure controls as well as custom image processing capabilities, it is likely that Lollipop deployment would have become the slowest in Android history.

This is the reason Camera2 has three support levels (from Android developer reference)

Camera2 Supported Hardware Levels

FULL: all that’s required but not everything

  • 30fps operation at sensor resolution preferred, 20fps required for at least uncompressed YUV output: useful for burst capture and computational photography
  • Per frame control
  • Manual sensor control: Like exposure time, ISO gain
  • Manual post-processing control: color rendition, contrast/gamma curve, noise reduction
  • Arbitrary cropping, in size and coordinates
  • At least 3 processed non-stalling output streams formats: like YUV 4:2:0 used for preview.
  • The ability to list and auto-configure output formats according to a list of supported streams

Note that FULL does not require RAW capture nor valid metadata and calibration profile for the DNG writer.

LIMITED: partial support, your mileage may vary

Cameras describing themselves as limited have some of the Camera2 capabilities but not all of them, and not all the ones required for the FULL supported hardware level either.
The reason would be either because the camera module or its Image Signal Processor lacks some of the required capabilities, or because the manufacturer didn’t implement them so they become available in Camera2 API.

LEGACY: a wrapper around the old Camera API

Legacy Camera2 means only really the historic Camera API is implemented. But for good measure, Lollipop adds a layer over it so modern camera applications do not have to implement two separate ways to control the imaging systems.

Each camera (usually back and front) has its own capability set. For instance, Nexus 5 hardware support level is FULL for the back and LIMITED for the front.

Enter Mobile World Congress 2015

A number of new Android devices have been introduced and demonstrated during Barcelona’s MWC 2015, making it the perfect opportunity to evaluate popular smartphone manufacturer’s approach of Google’s updated platform offering.

For the occasion, we wrote a specialized tool built to dump for analysis nearly everything that makes Camera2 API in Android 5.0.1 API level 21.

We have some good and some bad news for you:

The good

Major new Lollipop devices offer manual controls.

Highlighted by Samsung and HTC during their presentations, this year’s most anticipated flagships offer manual control for exposure compensation, ISO sensitivity, white balance, and scene mode.

Samsung Galaxy S6 and S6 Edge offers manual focus and adds adjustments for color preset or custom profiles on Color temperature, Tint, Contrast, Saturation, Highlight and Shadows.
You can save these settings in two custom presets, which is a nice touch to offer hardware grading option.

HTC One M9 offers shutter speed and adds adjustments for contrast, saturation and sharpness.

The bad

None export critical advanced camera capabilities in Camera2 API.

Devices supporting FULL

None identified.

Devices supporting LIMITED

Samsung Galaxy S6 & S6 Edge:

Galaxy S6 and S6 Edge Camera app Pro mode, demonstrated by Erica Griffin
Galaxy S6 and S6 Edge Camera app Pro mode, demonstrated by Erica Griffin. Manual controls and advanced color processing capabilities are unavailable via Camera2 API

Running Samsung’s own Exynos silicon, the new Galaxy line impressed during its presentation event showcasing strong imaging capabilities.

Real-time HDR, F/1.9 optics, OIS on the back and uses of computational photography promise to bring the manufacturer’s hardware and applicative solution to higher image quality than the competition, while using the same back sensor as its Galaxy Note 4 older brother.

The manufacturer’s camera application offers a limited (but efficient on a smartphone) set of features in a Pro mode as described above. Notably, RAW DNG capture and shutter speed are lacking.
These omissions could be remedied fully by third-party applications using Camera2 API.

A fair comment is that the Camera2 API implementation on the devices demonstrated, running an early 5.0.2 firmware (G920F and G925F code XXUA1AMW5) raises as many questions as it answers.
Back and Front cameras do not implement FULL but only a small subset hence their LIMITED qualification.

  • Tonemapping capability allows a custom contrast curve of 128 points, which should allow to customize the flatness of images processed by the ISP before their conversion in 8-bit color.
  • ISP color rendering cannot be controlled further: no manual white balance (via RGB gains), no custom color space conversion matrix.
  • Manual exposure parameters are absent. Only exposure compensation is provided: +-2 EV in +-4 steps.
  • Metering is fixed and evaluated through a single region that can be changed.
  • Manual ISO is absent despite being exposed in Samsung’s camera Pro mode.
  • Shutter speed control is missing just like it is in Samsung’s camera app.
  • OIS and software video stabilization function cannot be changed.
  • Available are a limited set of effects and modes, not matching with Samsung’s camera app offering.

About manual capabilities and RAW support, you might have noticed that some publications went ahead and published early highly optimistic information, which was brought to our attention on social media (from this source).
While we can’t explain fully why GD Software’s Manual Camera Compatibility application reported at that time that all of Manual Focus, Manual White Balance, Manual ISO, Manual Shutter Speed, RAW support are available, inconsistencies in the Galaxy S6 Camera2 API output might be a cause.

It appears to have lead into error Android Headlines and probably others, rejoicing too early about Samsung’s latest flagship compatibility.

An area of inconsistency is RAW support.
It appears Samsung engineers started some work here, but could not finish for the MWC 2015 firmware.
All color profile for each illuminant are filled with a 3×3 identity matrix instead of real color space transformation values; both illuminants being set to daylight.
Both back and front camera sensors black levels values are 1000, white level to 4000 which is likely arbitrary numbers.

One thing we can be certain is that in case RAW capture was working, the DNG files written via Android’s new DngCreator would be, in the current state, unusable without some serious calibration work and hacking.
We worked on an additional RAW DNG analysis capture analysis tool during the MWC to attempt evaluating the Galaxy S6 and S6 Edge, but it only succeeded to crash, leaving the smartphone and its Camera driver in an unstable state for a few seconds.

An early conclusion on the most anticipated device announced in Barcelona is that more of Samsung’s camera application features could be made available in Camera2 API, which might or might not happen.
RAW capture might become a possibility but is not functioning at the moment – before commercialization or even sample units are sent to reviewers.

Devices supporting LEGACY

HTC One M9

Sam Pullen M9 camera image adjustments settings demo
Sam Pullen demonstrates in video the HTC One M9 camera features and settings. Image adjustments are missing entirely in M9 Camera2 API.

Huge disappointment for the Taiwanese latest flagship that doesn’t export any of its advanced imaging capabilities either for the back or front camera shooters.
Full resolution burst capture is not possible on the back camera as non-stalling output formats are limited to 1920×1088 resolution.
At 20 Mpix (5376×3752) and in JPEG, the maximum framerate expected is only 0.61 fps.

The front camera is comparable and will output up to 2.04 fps at its full sensor resolution, 4MPix (2688×1520) in JPEG.

Both offer a typical choice of scene mode, effects, white balance presets or auto, up to 10 auto exposure regions, exposure compensation of +-2 EV in +-12 steps, auto exposure and auto white balance lock.

Legacy also means no RAW DNG capture. At least not via Camera2 API.

Additional features missing through LEGACY Camera2 as a result are 4K 30fps, 1080p60 or slow motion video capture.

ASUS Zenfone 2

Comparably to the One M9, the other Taiwanese flagship running on Intel platform instead doesn’t offer any advanced capability via Camera2 API but only a subset of its real capabilities.

Maximum burst rate for the back camera is 0.91 fps at full 13 Mpix (4096×3072) resolution JPEG, and on the front: 1.82 fps at full 5 Mpix (2560×1920) resolution JPEG.

Available: a selection of standard modes, effects, white balance presets, +-2 EV exposure compensation in +-6 steps, 1 auto exposure region, 9 auto focus regions with auto exposure and auto white balance lock.

Lenovo VIBE Shot

The Chinese maker original device attracted the attention of MWC’s audience thanks to its fully featured camera with optically stabilized zoom optics, as demonstrated by PhoneArena.

Unfortunately, despite the device runs Lollipop 5.0.2 and its photo-centric nature none of the manual controls are made available in Camera2 API.

Maximum burst rate for the back camera is 0.75 fps at full 16 Mpix (5328×2997) resolution JPEG, and on the front: 1.30 fps at full 8 Mpix (3264×2448) resolution JPEG.

Available: a selection of standard modes, effects, white balance presets, +-2 EV exposure compensation in +-12 steps, up to 5 auto exposure regions with auto exposure and auto white balance lock.

Sony Xperia M4 Aqua

As Sony’s first smartphone running Lollipop out of the box, offers a preview of the Japanese maker implementation of Camera2 API. Let’s note that Sony representative heavily insisted on the fact that their Lollipop 5.0 build was not final and not representative of the final product.

Hopefully it means that Sony is still working on the imaging programming interface, because the one dumped does not offer more functionality than the legacy Camera API.

Maximum burst rate for the back camera is 0.90 fps at full 13 Mpix JPEG(4128×3096) resolution JPEG, and on the front: 1.79 fps at full 5 Mpix (2592×1944) resolution JPEG.

Available: a selection of standard modes, effects, white balance presets, +-2 EV exposure compensation in +-12 steps, up to 5 auto exposure regions with auto exposure and auto white balance lock.

GiONEE Elife S7

This impressively thin smartphone running on MediaTek’s Quad Cortex A53 64bit cores gives us some elements to speculate on the Taiwanese SoC giant ISP drivers.

Unfortunately this is also a Legacy Camera2 implementation that we can find here, devoid of advanced manual controls.

Maximum burst rate for the back camera is 0.87 fps at full 13 Mpix (4224×3168) resolution JPEG, and on the front: 1.30 fps at full 5 Mpix (2592×1944) JPEG.

Available: a selection of standard modes, effects, white balance presets, +-3 EV exposure compensation in +-3 steps, up to 9 auto exposure regions with auto exposure and auto white balance lock.

Conclusions

If Google made an effort to upgrade the camera capabilities of their Nexus line smartphones, it seems that this interest couldn’t reach yet other manufacturers, which latest and greatest mostly ignore Android revamped API key aspects.

Today, we can find some elements explaining the current state of things like this comment from Steve Kondik of Cyanogen found on Google+:steve-kondik-comment-camera2

  • Potential lack of reference Camera2 API hardware abstraction layer provided by Qualcomm, MediaTek and Intel.
    Manufacturers could make their own based on Google’s code and specification but so far have not or only partially.
  • Likely lack of interest to share advanced features with third-party camera apps instead of keeping them device-specific as competition advantage.
  • Camera2 FULL adds complexity to implement in HAL and ISP levels while Manufacturer’s camera app use a different proprietary interface.
  • Lack of time.

Time only will tell if Camera2 API will ever gain traction outside of Google’s reference devices.
Hopefully, silicon solution providers will find advantages or be lobbied and offer FULL Camera2 implementations to their partners.

Meanwhile, make sure to read this full article keeping in mind that we based our analysis shared here on data extracted exceptionally early in the release cycle of each device discussed.

Published by

François Simond

SpectraStudy founder, system engineer, software developer, application author and analyst

23 thoughts on “Camera2 API on MWC 2015 devices: Galaxy S6, HTC One M9 and more Lollipop devices”

  1. Awesome article ,Thank you. got to know so much about the present situation of camera2 api. i wish my one plus one gets the full camera2 api support soon.

  2. Fantastic article. Absolutely everything we needed to know (but not everything we wanted to be true..). For once it shows that it’s not Googles/Androids fault, when some features are badly or not implemented. The camera2 API is great. On paper definitely. Manufacturers don’t care as they may have hardware which directly could compete with other product segments. I can understand that having manual shutter/aperture etc on a smartphone, would kill potential clients for the smartcameras. So it leaves us clients with the knowledge that “it’s possible, but they don’t want to give it to us, even though we would have paid a premium for it..”. A Galaxy S6 should cost 400 USD max with all the crippled features

  3. Brilliant article. You’ve helped me to confirm that neither Samsung or HTC will be getting my money this year. I had a good play around with the S6.The maximum ISO in pro mode is a pitiful 800!That’s a joke frankly. I did wonder about the HTC M9,in spite of bad reports, I thought I could sort things with a 3rd party app, but you’ve informed me that it isn’t possible 100%.

    1. 800 ISO is quite conservative as maximum manual sensitivity indeed.
      The F/1.9 lens aperture will help a bit here, but probably not enough for all kind of low-light situations. OIS doesn’t freeze subject’s movements.

      1. Francois, thanks again for this informative article. You’ve actually saved me wasting my money on an S6 Edge. I don’t suppose you know how that XDA dev is getting on with the G3, because I think there could release a lot of potential of the G3’s hardware.
        Otherwise, do you know of any other devices running Camera2, other than the Nexus 5 & 6, on Custom ROMS? I think I’m just gonna sit tight and hold onto my money for now.

        1. The situation could evolve quickly once SoC and ISP vendors like Qualcomm will distribute Camera2 API capable drivers, the next 6 months are hard to predict.
          On Samsung / Exynos products it depends only on them however.

          In case it doesn’t happen quickly enough, we can hope that small manufacturers will put to shame the major ones using full Camera2 API support as a key product differentiator and marketing avantage.

    2. I’ve seen a screenshot of Manual Camera Compatibility of the S6 which states that all features are available. I would like to see somebody installing Manual Camera, L Camera or Camera FV-5 on the S6 to actually see what’s possible – ISO, Aperture (won’t be possible i think), Shutter Speed etc. I don’t care about scenes or filters.

      1. Yes, the picture you mention is linked in the article.

        I also tested the existing applications you mentioned and none provided manual controls, as you would expected in light of the findings of this analysis.

        This is where it’s useful to be a developer able to write your own tools instead of relying on existing ones 😉

        Note: Manual camera preview was green-colored, indicating an additional bug in the color format conversion routines.

        1. Ah yes i see. Thanks for pointing it out again. Well, i have to say, quite disappointed in the camera department of Samsung. Why claiming it being the most advanced offering if it’s just another auto mode with some tweaks. I guess there’s still a long way to go and thus I’ll keep my hands off an S6. Btw, what tools are you talking about that your able to develop on your own?

          1. For this analysis, I developed two Android applications:

            • one that dumps in its entirety the Camera2 API capabilities
            • a camera app based off Camera2 able to shoot RAW DNG to try out the S6 unfinished RAW capture advertised in the API (based on something I started before)

            We can observe that Samsung focused on the S6 camera performance and tuning before finishing the Camera2 API implementation for third party apps.
            It makes sense, and time only will tell.

  4. Hey François Simond,
    Thanks for making things clear with even examples. but still google have not updated its doc about how to implement full features of the api.
    well its quite complicated without proper document.

    Thanks man.

  5. Hello Francois
    The low FPS (eg. .91 FPS on Zenfone 2), is that figure the same across apps like Lcamera and FV5?
    Also, do you know if the Nexus 6 has full 30fps readout?
    Thanks for your work

    1. Hello Jeremy.
      This is only with using Camera2. FV5 is also capable of using the older Camera API so it might be able to access a different capability set.
      As for the Nexus 6, yes it does 30 FPS at 4160×3120 (in YUV_420_888), 2.50 FPS in JPEG, and at 1920×1080 it does 60 FPS in YUV_420_888 as well.

      1. Sorry to mention again, but have you thought about releasing your app that dumps the device’s capabilities?
        The new LG G4 could be very powerful with Full camera2 api support..
        Regards,
        Jeremy

  6. Hi again,
    Just wondering if you might release your app that dumps the device’s camera2 api capabilities? It could be very useful for testing upcoming phones like the LG G4
    Regards

  7. Very informative. Thank you.

    Currently I am more than a little annoyed with Samsung for not including the Camera2 API with the Lollipop update for my Note 4. Hopefully, they will remove their heads from their nether regions and make this happen.

  8. tested the L camera with nexus 5 (android 5.1). Capturing raw dng images is really open many doors for app developers. No ISP or auto image editor is perfect. Even not for serious photographer, we just hope to make the image captured the way we like it to be….

    for image sensor engineers, they can really do apple to apple comparison for the performance of sensors. regardless of what ISP, image editor or display the phone is using.

    thanks google.
    and hopefully FULL support phone will be available soon.

Leave a Reply