Latest / CXOTalk / Ex Apple Engineers: The AI Future of Photography | CXOTalk #872
Transcript
- 0:00Are AI powered smartphones truly ready to replace professional
- 0:05cameras or is that just Silicon Valley height?
- 0:09Today on CXO Talk episode 872, we're going behind the scenes
- 0:16with two former Apple engineers who created the iconic Portrait
- 0:21mode on your iPhone. Now, as founders of glass
- 0:26imaging, Ziv Atar and Tom Bishop are using neural networks to
- 0:31extract stunning detail from smartphone cameras.
- 0:35Ziv Tell us about glass imaging. We want to revolutionize digital
- 0:40photography, specifically using AI.
- 0:43And just to clarify what I mean by that, it doesn't mean taking
- 0:48images from cameras and making them better.
- 0:51That's something that you know, a lot of people are doing in
- 0:54various industries, including smartphones and and drones and
- 0:57and photography in general. Our goal is in the longer term
- 1:01is actually to drive changes in hardware that are only enabled
- 1:05by by AI that we're developing. So so for example, you can make
- 1:10smaller cameras, cheaper cameras, but reach higher
- 1:13quality. You can you, you can make
- 1:15thinner cameras if we're talking about smartphones that fit into
- 1:18it, this form factor which is very limited in in its size and
- 1:25also to enable new types of of optics and new types of camera
- 1:29module architectures that give you some type of significant
- 1:33benefit. Tom, we hear terms like
- 1:37computational photography take us behind the scenes.
- 1:40What? What are we talking about there?
- 1:42In traditional photography, you have a camera and you get an
- 1:46image out at the end and maybe you can adjust it in an editing
- 1:50program like Photoshop. With computational photography,
- 1:54there's a lot of power that is put into creating that image,
- 1:58typically on the camera or in algorithms that's embedded into
- 2:03the the digital processing on the camera.
- 2:07What that enables is collecting a lot more information and
- 2:13extracting more information from the scene that's there to give
- 2:15you a much better quality image. What it's actually doing is
- 2:19typically combining various captures together, pushing the
- 2:26limits on the hardware capabilities, or even designing
- 2:30new kinds of hardware lenses and sensors and so on that can
- 2:34benefit from using algorithms to get a better finished image.
- 2:39So it's rethinking the whole camera from a computational
- 2:42perspective. So when you talk about something
- 2:46like portrait mode that that you were you were both involved
- 2:51with, where does computational photography fit in?
- 2:56Or more generally, maybe you can just give us a brief primer on
- 3:02how something like portrait mode works, just as an example.
- 3:06Portrait mode is an example of computational photography
- 3:09because it's not something that existed before we had enough
- 3:12compute power. What portrait mode does is
- 3:15basically it takes an image, normal image, color image like
- 3:19like existed before portrait mode and it uses multiple
- 3:24cameras to create this. It's it's called stereo, but
- 3:26basically to create a depth map. So now when you have a color
- 3:29image, you have another dimension to that image, which
- 3:33is the depth, the distance of of each pixel.
- 3:37Obviously, it's, it's not perfectly accurate, but it's
- 3:40good enough to mimic this out of focus blur, which you have on
- 3:44big camera. So that was intention with
- 3:46portrait mode basically to, to mimic what happens on a big
- 3:49camera on, on big cameras, it happens optically.
- 3:52So optics when something is not focused, it gets very blurry,
- 3:56but but it's also a nice thing. It kind of helps separate the
- 3:59background and foreground. And then it what sets
- 4:01smartphones apart from professional big cameras.
- 4:06So portrait mode was an attempt to bridge that gap and basically
- 4:10give smartphone users the ability to blur the background.
- 4:13Now back to your question, Michael, you're asking kind of
- 4:16this about portrait mode, but also more general about
- 4:19computational photography. So computational photography has
- 4:22been, has been around, I, I would say pretty much or a
- 4:25little bit after smartphone photography started.
- 4:30And, and I, you could say that digital cameras are all
- 4:33computational because they have a, what was called a Bayer
- 4:36pattern on the sensor and you have to interpolate some data
- 4:38that's not there and, and that's compute, right.
- 4:42But, but I, I would actually call computational photography
- 4:45when you start to do tricks like HDR and you start to do like
- 4:49complicated denoising in, in a multi frame image fusion, which
- 4:54is happening on on any smartphone today or any some
- 4:57other projects as well. It's it's just means you're
- 5:02using heavy compute to do things.
- 5:04Now this is not new. This has been going on for I, I,
- 5:08I would say, let's say 10 years or or so.
- 5:10And of course every year the chips on phones or computers,
- 5:13laptops I think is getting much stronger.
- 5:16So you can use more advanced computational photography.
- 5:19But what we're seeing and that also comes in in kind of two
- 5:22phases, the use of, of AI, more specifically neural networks in,
- 5:26in this space. So we started seeing use of AI
- 5:29even on smartphones quite a few good years ago.
- 5:33For example, if your things like face detection, segmentation,
- 5:37this is a sky, this is a skin, we want to have a smoother sky.
- 5:41We, we can be more aggressive with denoising, for example.
- 5:44So, so all these semantics things, but they happen at low
- 5:47resolution. So if you have an image from
- 5:49from a phone and you want to say, hey, there's a face there
- 5:52and then we want to do something with the face.
- 5:54You can run a face detector on a thumbnail image.
- 5:57You don't need to have the four res.
- 5:58So those things that happened at low res AI that happened at low
- 6:02res that started a few good years ago.
- 6:04We're now in the last year or two, we're at a point where AI
- 6:08or the AI engines on phones are able to actually handle full res
- 6:12compute, AI compute. And, and that opens up
- 6:16possibilities to, to for doing a lot of very innovative and very
- 6:22impactful things like what we're doing class.
- 6:24We'll talk about it a little bit more in detail in a minute.
- 6:26The term computational photography I think came out
- 6:30with regard to smartphones maybe 5-8 years ago when they first
- 6:36started fusing together multiple images.
- 6:39So they capture a burst of images and then to reduce noise
- 6:44and to increase detail and to maintain sharpness, they can try
- 6:48to fuse together, incorporate information from that sequence
- 6:51of images to get one still photo.
- 6:55And that for a while to many people was computational
- 6:57imaging. But things I've mentioned quite
- 6:59a lot of other use cases there and things we're now doing with
- 7:03computational photography where it's really a new paradigm of
- 7:07how do you design the whole system?
- 7:09How do you process with AI? How do you manipulate the
- 7:13hardware even to give you a better image, knowing that you
- 7:17have this computational power available to you?
- 7:20You've both mentioned hardware and software.
- 7:25Can you tell us, drill into that a little bit and then Tom, maybe
- 7:31you can show us some example images.
- 7:35But first, I just want to tell everybody that you can ask
- 7:40questions. If you're watching on Twitter,
- 7:42pop your questions into the onto Twitter, onto X using the
- 7:49hashtag CXO talk. If you're watching on LinkedIn,
- 7:52just pop your questions into the chat.
- 7:55So, so, so Tom, just drill into this a little bit more.
- 7:59So when we're talking about use of AI to improve image quality,
- 8:05that's actually something in the case of a phone that's running
- 8:09typically on the phone itself, you know, most of the processing
- 8:14happens to on the phone to produce an image.
- 8:17There are some cases coming to the forefront today with, you
- 8:21know, generators of AI where there are services available to
- 8:24process images after the fact. But when you capture an image,
- 8:28the sensor captures raw data, which is really just the the
- 8:33light that hits the sensor and the this electronic signal that
- 8:36comes from that. There's a lot of processing
- 8:39steps that typically occur to give you the finished image.
- 8:42You can view on a screen those possibly done by software
- 8:48engineers who create algorithms, many different blocks that go
- 8:51together, try and increase sharpness, reduce noise, get the
- 8:56right colour and so on. And then there's many novels
- 8:59they have to adjust in order to give a good quality image.
- 9:03What we're doing at Glass Imaging actually is trying to
- 9:06create a neural network that's an end to end AI solution that
- 9:12replaces all of those algorithms with one, one process that can
- 9:19extract the most quality from a given image and it's adapted to
- 9:25a particular cameras as well. So in the traditional image
- 9:29processing sense, all that demosaicing, denoising,
- 9:33sharpening and detail extraction, correcting for lens
- 9:37issues, those are all things that we can do now with AI and
- 9:41we run on a camera on the smartphone to extract the best
- 9:46possible image. Check out cxotalk.com Subscribe
- 9:50to our newsletter so that you can join our community.
- 9:54We have really awesome shows coming up.
- 9:57What about the quality of traditional photography using,
- 10:05using lenses, of course, versus the quality that you're able to
- 10:09get through AI? And would it be would it be
- 10:13correct to say that this is an artificial process as opposed to
- 10:18traditional photography using lenses?
- 10:21I wouldn't say it's a artificial.
- 10:23So it's there is interpolation going on and there has been
- 10:28basically for any colored camera that ever existed, like I
- 10:32mentioned earlier, there is a color pattern on the sensor even
- 10:36on big, you know, traditional Canon Nikon cameras in which
- 10:41means that in every pixel location you have either red or
- 10:44green or blue information. We also means that if you're now
- 10:48in a green pixel, it means you don't know what the red value
- 10:51there and what the blue value is and, and you have to guess it.
- 10:54And the process traditionally is, is known to be called demos
- 10:58Iking, which means guessing, interpolating information.
- 11:02So you, you, the question is like, do we call this
- 11:04hallucination or guessing? It's, it's, it's on a small
- 11:08scale, on a pixel level, it is making up information.
- 11:11I, I think this question, Michael, you bring up, it
- 11:14becomes more interesting nowadays with generative AI
- 11:18because now you're not only able to guess what a value of pixel
- 11:22is, you can guess what the mouth of a person looks like or, or
- 11:25the, the, the eye or a house or a bird or a flower.
- 11:28And you can make up the whole flower.
- 11:30You don't just guess one pixel. And I, I think if, if you're, if
- 11:35you're talking about multiple pixels of replacing multiple
- 11:37picks, like a chunk of 100 by 100 pixel, let's say there's a
- 11:40flower in there that, that, that's purely generative.
- 11:43If you're replacing a few pixels here and there and moving some
- 11:46noise, then I, I wouldn't go out and call it generative.
- 11:51Although again, it's a, it's totally a Gray area.
- 11:53It's not black and white like like I tried to explain.
- 11:56There really is this question of reality versus what's made-up.
- 12:03And it, as you said, at one extreme, you have generative AI
- 12:08that is concocting something that never existed.
- 12:13At the other extreme, you have traditional photography.
- 12:17And so how do you make that distinction?
- 12:21And how far do you, at glass imaging, push the the envelope
- 12:27in terms of adding or changing or adapting the original
- 12:32photograph in order to make it quote UN quote better, whatever
- 12:37better happens to mean? Regarding the better there there
- 12:39is if you're taking a a picture of something and you're trying
- 12:42to test some algorithms, AI, anything, let's say that it,
- 12:47it's, I don't know, some business card you put on the
- 12:49wall. You can actually walk up to that
- 12:51business card or just take a picture with a very high
- 12:54resolution camera. So you have you have a ground
- 12:55truth. You can judge what you're doing
- 12:58in terms of how good or bad did did it interpolate or
- 13:02hallucinate the missing information.
- 13:04We specifically at glass are, are actually focused on the
- 13:08trying to be as less generative as possible in, in, in the sense
- 13:12of not making up information that's completely not there.
- 13:17Some, you know, some other companies or other pieces of
- 13:22software offering that, you know, you have Dolly and then
- 13:25Sora and, and all these things. We, we see it as a post
- 13:29processing. So it, it kind of complements
- 13:31what we're doing 'cause even if you have an image and you want
- 13:34to do some generative things on it, like remove skin
- 13:38imperfections or, or something like that, you're always better
- 13:42off with having the input image with the highest possible
- 13:46resolution and as true as possible to, to, to reality.
- 13:49And then you can go ahead and apply some, let's say, fake
- 13:52effects on it, which is fine. I'm not criticizing it.
- 13:56I think a lot of people love it. But I think it's also important
- 14:00to start with something that as true as possible to, to reality.
- 14:04And we're very proud that, that we're able to achieve, let's
- 14:08say, a high confidence level or, or high comparability with the
- 14:11actual, with reality, with the ground truth.
- 14:14Maybe I'll say a little bit about how we're doing it because
- 14:18we, we talked about what we're doing, but I, we, we haven't
- 14:20explained how. So when, when you look at a
- 14:23camera and it doesn't matter if it's a phone or a drone or a
- 14:25professional camera or a medical device, a camera consists of a
- 14:29lens and a, and a sensor than a bunch of algorithms, right?
- 14:33The more constraints you have on the camera in terms of size and
- 14:36cost and, and, and and and weight and materials that you
- 14:39can use usually means you'll have more aberration.
- 14:43The aberration means imperfections in the lens and it
- 14:46means the lens is going to be a little bit blurry at me.
- 14:49If, if you're cramped with space like phones or or electronics in
- 14:52general, also means you're likely using a very small
- 14:55sensor, which typically means a lot of noise in the image.
- 14:59So you have, if you look at the image that just comes out of the
- 15:02sensor and, and all these small devices, the images is bad.
- 15:06It's really bad. We, we look at it, you know,
- 15:08internally, but most people never see an image, a raw image
- 15:10from a phone. It's super noisy.
- 15:13It has color, noise, color speckles all over the place.
- 15:15It's very not sharp. It has lots of optical
- 15:19aberrations and most visible ones are usually chromatical
- 15:22aberrations. So if you have black and white
- 15:24text suddenly on the raw image, when, if when you look at it, it
- 15:26has purple and blue and green, but you look at the, the, the
- 15:30real paper and says it's black and white.
- 15:32So these, these are chromatical aberrations and noise and color
- 15:35noise, it all add, it all adds up to being, to creating a very
- 15:39bad image in the process of correcting all these things.
- 15:42It is, let's say algorithms, computational algorithms,
- 15:46additional photography. Now back to what glass is doing,
- 15:50which is also different than than what others have been doing
- 15:53in the last few years. We have, we build special labs
- 15:57that we can take any camera. So if, if for example, we'll
- 16:01take like a iPhone, we're showing some, it's from iPhone.
- 16:03So we took the, the physical iPhone, we put it in the lab.
- 16:07We have labs that we built dedicated for this, what the lab
- 16:10does and it's all I, I can't describe what, what, what like
- 16:13to, to the very details, but the lab learns how to characterize
- 16:18the optics and the sensor extremely, an, an extremely
- 16:21detailed manner. So for every pixel, for every
- 16:24angle, for every type of light, for every amount of light, it
- 16:28characterizes the optics and the sensor and the interactions
- 16:31between the two in it. And then all this data is
- 16:34collected, terabytes of data. It takes us a few days to
- 16:37collect that all that is fit into a, a software for training
- 16:40in neural network that trains in neural network specifically, if
- 16:43we talk, if we're talking about smartphones, something that can
- 16:45actually run on a smartphone. Curiously, we're offering our
- 16:48solutions on Qualcomm chips, but we also have demos on, on iPhone
- 16:51and and other devices. Once we do it, we have a network
- 16:55that basically learns to take these bad images and create good
- 16:59images, basically remove all the optical imperfections, all the
- 17:03as, as much as possible, the sensor noise and, and sensor
- 17:06imperfections. Now everything I explained now I
- 17:09was, I was given an example of an iPhone.
- 17:11The camera on the iPhone is they're trying to make it as
- 17:13good as possible in terms of sharpness, lens operation
- 17:16sensor. But because we can correct these
- 17:19things, that kind of opens up the, the, the question, OK, if
- 17:22you can correct all these things, why does Apple or
- 17:25Samsung or any other phone maker, why do they need to make
- 17:27it perfect, right. And the answer is they don't.
- 17:31We, we, we can make the lens much worse than they are today
- 17:35or the camera in general and still get a good image of it.
- 17:39And, and then the next question, you know, that I'm asking
- 17:41myself, OK, why make it bad if you can make it good?
- 17:44And, and that, that the answers to this are what makes things
- 17:47interesting. So you, you make it bad if you
- 17:49can correct it only to gain something, right?
- 17:52Because what we're not university, we're not trying to
- 17:55prove something. We're trying to, to create
- 17:56something tangible that that is actually useful for products.
- 17:59So, so the making something bad to gain something you can gain
- 18:03cost, you can make lens cheaper, can make them smaller, you can
- 18:07make them support a bigger sensor without going crazy with
- 18:10the number of elements that you have inside the lens.
- 18:13And, and, and that just opens up lots of the design space you
- 18:16have now is, is way bigger than what you had when you needed to
- 18:19make a good lens and a good sensor.
- 18:21Very interesting. So the source of data, correct
- 18:25me if I'm wrong, If I understood this correctly, the source of
- 18:27data is the profiling that you do of the camera of of the of
- 18:35the hardware and that data then gets fed into your neural
- 18:43network, which then has can essentially construct a
- 18:49simulation, a digital twin. Would that be a correct way of
- 18:52saying it? Yeah, essentially this, this
- 18:55kind of three steps, I think 1, you characterize the camera.
- 18:59So we take any camera, smartphone or other type of
- 19:02camera, we've put it in our lab and we do many measurements
- 19:06that's completely profile how the lens and the sensor behave
- 19:10in different lighting conditions, different kinds of
- 19:14brightnesses and scenes and so on.
- 19:17The second step is we take that data and we use that to train a
- 19:20neural network with. So in a way it's a self
- 19:25supervised process. We don't have any data
- 19:27annotation or labelling going on from AAI point of view.
- 19:32So we train a neural network that's dedicated to that device.
- 19:36And the third step is we have to port it to run efficiently on
- 19:38the edge. So most AI today is running in
- 19:42the cloud, you know, big data centres, lots of GPU's.
- 19:47What's the power on these mobile devices?
- 19:50You have actually some pretty capable chips today.
- 19:54So Carcom is on the majority of Android phones today.
- 19:59Pixel, Google Pixel and Apple iPhone have their own processors
- 20:04that also have some kind of neural processing unit.
- 20:07So you have the CPUGPU and an MPU, three different systems of
- 20:11which they all have different merits.
- 20:14But for any of these kind of neural networks, the MPU neural
- 20:17processing unit is super capable, very power efficient.
- 20:22So we can put that neural network running on that MPU.
- 20:26And that requires, you know, some careful software
- 20:29engineering to make it run efficiently.
- 20:31We want to process like the 12 or 50 or even 200 megapixels
- 20:35coming out of some of these sensors today in close to real
- 20:38time. Meaning you take the picture and
- 20:41you don't notice that there's some background processing going
- 20:44on and you've rendered that image on the screen and you save
- 20:48it to disk using that neural network, which is is taking
- 20:52those bursts of raw images that come in from the sensor and
- 20:57creating one final image file that has all the the sharpness
- 21:00and detail and low noise and clarity that you expect.
- 21:04O Your neural network then is closely bound to the specific
- 21:09hardware of the phone, to that particular model, with the
- 21:12characteristics of that particular lens and processor
- 21:16and sensor and so forth. I'd say that's one of the
- 21:19advantages because there are solutions like today available
- 21:24for enhancing images. You'll see some of them in
- 21:26Photoshop, for example, Lightroom, you'll see, you know,
- 21:29upsampling websites, You can upload an image and produce an
- 21:34AI generated upsampling. Those are not specific to the
- 21:38type of camera that's used to take the picture.
- 21:41So they can improve somewhat the image quality.
- 21:44And as Steve was explaining before, there's this continuum
- 21:47of how much generated versus how much is recovered from the
- 21:52image. What we're focused on is really
- 21:54trying to recover detail that's encoded in the signal from a
- 22:00sort of information theory or signal processing point of view
- 22:03that there is there's image content.
- 22:07The image has information that is scrambled up in a way.
- 22:11It's mixed up, it's blurred, it's noisy, it has some
- 22:14uncertainty about it. There is a true underlying
- 22:18signal, which is the the image that comes from the scene.
- 22:21And we're trying to reconstruct that as faithfully as possible
- 22:24using the the information that's there.
- 22:27But it's just a scrambled up. So if you take one of those
- 22:31other approaches, they don't actually to do that unscrambling
- 22:35in a way. They just try to generate or
- 22:38create or invent plausible detail might be there.
- 22:41So we're actually compensating for issues of the lens like
- 22:45softness, aberrations, noise, in the sense they're using a
- 22:49profile that's measured in our lab.
- 22:52You're tightly bound to the hardware and you're compensating
- 22:57for deficiencies that may exist in in the lens or in the
- 23:03software platform, whatever, whatever it might be.
- 23:05That's part of that overall mobile system, that phone.
- 23:09Any lens, any sensor has deficiencies.
- 23:12It's a measurement of the the physical worlds.
- 23:15When you have a sensor, you're collecting photons.
- 23:19When you have a lens that's only, you're trying to focus
- 23:21rays onto small points on the center.
- 23:23And we can get into the the physics of it, but it's, it's,
- 23:26it has limits physically and we try to overcome this as best as
- 23:32possible using software. You asked or you mentioned that,
- 23:36OK, this neural network that we just described would be tailor
- 23:40made to that specific camera. It has pros and cons.
- 23:44So the and we solve the con. I'll explain how.
- 23:47So the big Pro is that it's tailor made to a specific
- 23:50camera. So it's, it's able to correct
- 23:51the specific aberrations of that camera in, in the, the specific
- 23:55noise of that sensor. It that what, what is allowing
- 23:59us to achieve the maximum potential quality of, of that
- 24:02camera, basically maximize them well that you can from a given
- 24:05system. The downside of of this approach
- 24:08is that you have to train for each specific camera.
- 24:11And basically most of the time we spent last few years is
- 24:14actually building a system which is a physical lab and a bunch of
- 24:19software for, for for training and software for controlling the
- 24:22lab. There's some robotics inside.
- 24:24What we what we concluded a while ago that OK, the way to
- 24:27get the maximum quality is to train for, for every different
- 24:30camera. We need to make it fast.
- 24:33So we spend a lot of efforts on and resources on making these
- 24:37labs that can take any device with a camera on it or multiple
- 24:40cameras like phones and within, you know, it can be as fast as a
- 24:44few hours and go up to one or two days.
- 24:48We, we can capture all the data needed to train the network for
- 24:51a specific device and then the training takes anything between
- 24:54a few hours and, and, and, and maybe a day or two.
- 24:57So the whole process can start on Monday and finish by
- 25:01Thursday, right. And just to kind of put things
- 25:04where this stands in terms of what without glass, how would
- 25:08you do it, right? So all the phone makers, they
- 25:11design a new hardware for their new generation phone.
- 25:14So you know, like Apple is not working on iPhone 17, I assume.
- 25:18So it's at some point several months before the launch.
- 25:21These these companies, they get the first hardware prototypes
- 25:24like the phone with the new lens and the new sensor, new
- 25:27everything, and then they need to tune.
- 25:30It's called tuning tuned image quality.
- 25:33Typically there's depend on the company size, but there's
- 25:36anything between a few hundreds to a few thousands of people
- 25:39working for a few good months, maybe even six months on tuning
- 25:43these things, right. So, so you know, when, when we
- 25:47say four days, it's very fast. And also worth mentioning those
- 25:50four days that I mentioned, Monday to Thursday, there's
- 25:53almost 0 human hours involved. So there's maybe 20-30 minutes
- 25:58setting up the device in the lab.
- 26:00Once it's set up, we close the door and we, we let it go.
- 26:03And then the training, obviously, you know, it's NVIDIA
- 26:06GPU is crunching numbers. It's not people doing actual
- 26:10work. So, so we put what, what I want
- 26:12to say, we put tons of efforts on automating the process.
- 26:16This way we can support, you know, thousands of, of different
- 26:19project at the same time and, and provide the, the output
- 26:23which is in your network that's designed to run on a specific
- 26:27platform, whether it's a Qualcomm chip or, or some PC or
- 26:30NVIDIA chip within a few days. So we could support very high
- 26:35number of customers every week. So we have a question.
- 26:38Well, let's see from LinkedIn, Kamaru Kamaru Dean Lawal says.
- 26:44Is glass imaging hiring? Good question.
- 26:47Yeah, thanks for bringing up. Yeah, we, we have a few open
- 26:50roles on our website and we are going to add several more open
- 26:56roles. It's some software roles, some
- 27:00business roles and and some machine learning AI roles.
- 27:05How maybe you want to add something?
- 27:06Yeah. Optics.
- 27:07Computational optics also. Yeah.
- 27:11Kamaro Dean, look at their website.
- 27:13Gloria Putnam says, where do I get one of these glass AI track
- 27:17suits? They look sharp.
- 27:20And here we have a more technical question from Ayush
- 27:25Jamdar. And he says, do the trained
- 27:30neural networks learn from the specific camera's response
- 27:34function, CRF mapping from sensor irradiance to pixel
- 27:39color, therefore replacing entirely the camera pipeline
- 27:43processing at the phone's ISP. So basically what he's wondering
- 27:49is where do you guys fit in relation to the camera, if I
- 27:54understand the question? We can essentially replace the
- 27:58whole ISP image signal processing pipeline that exists
- 28:02today. We can also replace part of it,
- 28:05but we sit at the, as I said, the the front end where we take
- 28:10in those raw images directly from the sensor all the way up
- 28:13to producing a finished RGB image.
- 28:17Typically we say it's a linear RGB image, meaning it's hasn't
- 28:21had any tone response or manipulation of colour on top of
- 28:26it, which is more of an aesthetic treatment that every
- 28:30camera or phone or software does today.
- 28:34So we leave that up to vendors to apply their own or we can
- 28:37provide a solution that gives, you know, optimal tone and
- 28:40colour. It can all be done end to end.
- 28:42So it's it's flexible. So would it be correct to say
- 28:46then that you are performing your magic, improving the
- 28:52sharpness, reducing the noise, you're leaving the colors as
- 28:55they are so that the phone platform can then apply their
- 29:02own processing like like Samsung does certain things or other
- 29:06manufacturers do other things within their phone software?
- 29:12Is that their end user software as well?
- 29:14That's very correct, Michael. And the reason behind it is not
- 29:17necessarily what we can do or what we, we want to do.
- 29:21It's, you know, if you look at smartphones, some of them have
- 29:26their own, I'd say color tendency, warmer looks, this
- 29:30type of white balance, that type of white balance.
- 29:32If, if we were selling the 10 smartphone companies the same
- 29:36post proceed pipeline, they will all end up with the exact same
- 29:38colors. And it's, it's one of their ways
- 29:40to differentiate, you know, between, between phones.
- 29:44And also it's worth mentioning that these color preferences are
- 29:48very geographically varied. So if we're looking for phones
- 29:52that are designed in China and sold in China versus phones that
- 29:56are designed for the Korean market or Japanese market versus
- 30:00the US or European market, you'll see very different trends
- 30:04in terms of white balance, exposure, color vibrancy, things
- 30:08like that. It's also worth missing.
- 30:10These trends are not fixed. So we've seen in the past, let's
- 30:14say five years ago, we've seen that Europe and US devices
- 30:18mostly like Apple and, and, and, and Samsung were trying to stick
- 30:22to more realistic looking colors and tones, while the, the, the,
- 30:28the leading phone makers in China were kind of promoting
- 30:31very saturated colors. And now we actually see or not
- 30:35now like the last 2-3, as we kind of see some inverse of
- 30:38that. So I'm just saying that this
- 30:40trend is not constant with time, it changes.
- 30:42It has a very slow cycle, maybe maybe every few three years.
- 30:46I, I think this is just my personal opinion.
- 30:50I think the, the reason to go more true to life is kind of
- 30:55coupled with the quality, the sharpness of damages.
- 30:58I, I think that the more you're getting closer to let's say SLR
- 31:01quality or professional quality on images, I think that will
- 31:04actually drive demand for more realistic looking colors and not
- 31:08very, you know, overly vibrant saturated colors.
- 31:11A good time to look at some images if important.
- 31:14Not Michael. OK, so we're looking at an
- 31:16image. We're looking at 2 images side
- 31:18by side. On the left, these are both
- 31:20images from the recent iPhone. On the left, it's actually the
- 31:25processing out straight out of the Apple Camera app, and on the
- 31:28right is using our Glass AI neural network processing
- 31:32running on the device. So what you can see immediately
- 31:36and we can pan and zoom around here, but you can see that you
- 31:40have better detail, sharpness, definition, color.
- 31:47Like if you zoom in on this roof, you can see the red color
- 31:51is much better represented. You can see the trees are clear
- 31:54up. The definition of the lines in
- 31:57these buildings is is much better.
- 31:59The saturation is better. Many, many things to point out.
- 32:06Let's look at another image here.
- 32:08This is one where actually said you captured this one.
- 32:13It's I guess it's zoom in. Yeah, it's the one where we were
- 32:17talking about earlier, Michael, you're talking about
- 32:19hallucination and we talked about how it's Gray and not a
- 32:23binary. So I think this is a good
- 32:25example of how we differ from other, let's say, AI processing.
- 32:30So the image on the left is captured by the default iPhone
- 32:34camera app. This is a lot of zooming in
- 32:37going on here. So this this object is pretty
- 32:39far from the camera. So when you zoom in, zoom in a
- 32:42lot, what you see on the processing done by the iPhone
- 32:45camera is that the text starts to break at some point and it
- 32:50starts to look very weird. This this weirdness is actually
- 32:55hallucination. Obviously the photons that came
- 32:57from the scene through the lens on the sensor did not look like
- 33:01that. The photons look correct, but
- 33:03it's very hard because there's so much noise in it and the, the
- 33:07optics is so blurry. It's very hard to to separate
- 33:11information from from noise and and blur because we're training
- 33:15neural network specifically for this phone.
- 33:17In this case, it's the iPhone 16 Pro Max Telecamera.
- 33:20It's the 5X Telecamera. What we're able to, to, to
- 33:23achieve better reliability in terms of how, how this matches
- 33:29to the, the reality or the real text on this box or whatever
- 33:34this thing is. And, and you know, both, you
- 33:36could say both left and right are hallucination to some extent
- 33:40on, on the pixel level. But I think this is a good
- 33:43example to show that the hallucinations on the left
- 33:46extend to maybe 45610 pixels, whereas on the right we're
- 33:52hallucinating only or interpolating only maybe one or
- 33:56two pixels in size. So even though the text on the
- 34:00right does I, I wouldn't say it looks good.
- 34:02It's a little bit blurry and, and smudgy, but you don't see
- 34:05things that should not be there. I mean, we, we as humans look at
- 34:09this text and we know it's English letters and we know what
- 34:11it should look like, right? So, so we're not achieving
- 34:15perfect sharpness, but we're not inventing lines that were not
- 34:19there, which you know, just this was to explain the hallucination
- 34:23or lack of hallucination objective.
- 34:26You might ask how do we do this? These actually come from, you
- 34:29know, the, the same raw files that are processed differently
- 34:34in each case. So we're actually, like I said,
- 34:38replacing all those blocks in this traditional image signal
- 34:41processing pipeline that are kind of lost seat.
- 34:44So every every time you apply a sharpen denoise tone curve, you
- 34:51know, HDR, various different steps, they all lose some
- 34:56information in the signal, whereas you do it end to end
- 35:00with a neural network, you can actually preserve those things.
- 35:03So there's demosaicing, which is you have this beta pattern on
- 35:07the sensor where you only see every other green pixel and
- 35:11every one in four blue and red pixels.
- 35:13So you're having to make those up.
- 35:15But the sensor is also noisy, so you're trying to guess the
- 35:18missing pixels in the face of actually extreme noise in these
- 35:22tiny cell phone sensors. And that's what leads to these
- 35:26kind of artefacts when it's done with a handcrafted algorithm, as
- 35:30has been done for the the past. Well, since digital cameras have
- 35:34been around, you tend to those noise artefacts get amplified
- 35:40through those different steps. Whereas on the right, the neural
- 35:43network is actually able to make its its best interpretation
- 35:48using the the characterization that we did in our lab to
- 35:51untangle the the effects of the the noise that the mosaicing and
- 35:54the sensor and so on. So when you talk about the
- 35:56characterization, now that we've seen some examples and it's
- 36:01obviously very, very impressive, what what is the
- 36:05characterization? Can you be more specific?
- 36:08You're talking about aberrations in the lens, limitations on the
- 36:12sensor and, and how do you know? How do you know?
- 36:16I mean, if you know where those aberrations exist, you know, is
- 36:21it relative to absolute perfection?
- 36:23Is it relative to nothing? Can you give us some calibration
- 36:26sense here? We can give some more
- 36:29information but not explain exactly how we do it because
- 36:31that's, you know, part of our secret sauce.
- 36:34So when the lens goes in the lab, I'll explain methods used
- 36:41by let's say traditional. Let let's say that forget about
- 36:44the labs again. Let's say that you have a new
- 36:46camera or a phone with a camera on it.
- 36:48And you want to determine what is the what, what does the
- 36:51optics look like? How is the blur looking like?
- 36:54So you, you, you could do a few things, but maybe the most
- 36:56obvious 1 you, you would take some lab with nice lighting and,
- 37:00and, and and charts. And you would pick a chart that
- 37:04has, there's something called SFR chart was basically just a
- 37:07bunch of black squares on white background.
- 37:10And then on these squares, you can go to the edge where it's
- 37:13supposed to go from black to white.
- 37:15You know that it's infinitely sharp.
- 37:17You so you know the ground truth because you printed that chart
- 37:19and created that chart or, or, or you, you buy it, but you know
- 37:22how it looks like. And then you take a picture of
- 37:25that chart with your, let's say, in the case of a phone with a
- 37:28phone, but you just take a raw picture.
- 37:31You don't, you don't do any processing on the image.
- 37:33So now you look at this raw image and you look at the, let's
- 37:36say the black, black and white edge and you see that, OK, it's
- 37:40not actually transitioning straight from black to white,
- 37:43it's actually blurry. So the the transition takes
- 37:46maybe three or four or five pixel.
- 37:48So you can learn what the blur of the lens from that.
- 37:51And then you would also see that this edge is supposed to be
- 37:56either black or white or Gray. You see that it has for example,
- 37:59some greenish hue on the right side and some magenta color on
- 38:04the left side. Then you know that you have a
- 38:06color aberration and you can actually even use like some
- 38:10simple calculations to calculate how much is the color aberration
- 38:14in pixel. You can manually characterize
- 38:17these things that are AI is doing, but it would it would
- 38:19take you years to characterize 11 camera because you would have
- 38:23to do this thing that I just mentioned every location in the
- 38:28sensor. So and from going from corner to
- 38:30corner to the center, you have to do it for all types of light
- 38:33level or light levels and light types because the noise plays a
- 38:37game here. If if you have low light, you
- 38:39have the defective noise is much more dominant and you have the
- 38:42temperature of the light, which also has an effect.
- 38:45Now, just to make things more explained, the reality is even
- 38:50more complicated because we have these lenses are not static,
- 38:53they have Oas on them, so they move around.
- 38:56So everything I described now is also dependent on the position
- 38:59of the lens during stabilization, during capture.
- 39:02And you also have an autofocus system on these cameras which
- 39:06tries to focus on on your subject, but that's never
- 39:10perfectly accurate. So you have some variance there,
- 39:12some slight autofocus like mismatch and you also have
- 39:15object distance. So everything I describe now you
- 39:18need to repeat the whole thing from a few centimeters all the
- 39:20way to, to very, you know, several 10s of feet.
- 39:24So, so this, this can be done mentally, but it will just take
- 39:27you, my guess is a year of, of full time employee in the lab
- 39:32to, to just do all these measurements.
- 39:34And then there's also a question, what do you do with
- 39:36all this data, right? You'll end up with just tons of
- 39:39data and and what do you do with it?
- 39:41Let's not look at another example.
- 39:43Maybe Michael, can you see the the screen here?
- 39:46Yep, KPMG. Oh, look at that.
- 39:48Yes. This is a a building taken, you
- 39:50know, from reasonably far away. But on the left, this is another
- 39:53phone. It's an Android phone, but
- 39:56Archie has some pretty extreme, extreme lens aberrations going
- 40:00on. So exactly what Zenith was
- 40:02talking about some of these and zoom in some more.
- 40:04Did you see the edge transition here is kind of blurry.
- 40:08It's it's sort of sharp, but sort of not.
- 40:10So it's not, it's not like just. Soft, but you you kind of have a
- 40:14double edge this is and you have some colour fringing.
- 40:18You can see also on these wires here in the roof and on these
- 40:22pipes, you kind of got red and green on either side of these
- 40:25whites and black transitions that it kind of looks like an
- 40:30old camera actually. And you know the reason why old
- 40:32cameras look like that because they didn't have very well
- 40:35controlled lenses. Nowadays you have much better
- 40:37manufacturing tolerances and producing lenses, but still when
- 40:41you try this is actually in an extreme case, you're trying to
- 40:44zoom in maybe 10X or 20X on the camera.
- 40:48I'm trying to fit that in a cell phone's really difficult.
- 40:51So when you combine it with very tiny pixels, you start to see
- 40:54these lens separations where the light is diverging in different
- 40:58ways on the sensor of different wavelengths.
- 41:00So the red, the green focus differently.
- 41:03You get chromatic aberrations and other other things like
- 41:05that. So this is what you would get
- 41:07with typical processing, but after applying Glass AI, it
- 41:12learns to calibrate and correct for those things with that
- 41:15neural network under all those very conditions, as if
- 41:18mentioned. So different focus settings,
- 41:20different distances and so on, uh, different parts of the field
- 41:24of view, they all behave differently.
- 41:26So it's a huge space of parameters the network has to
- 41:29learn, but the result is you, you can start to see those very
- 41:32fine details. Umm, just one more example, you
- 41:37know, let's zoom in on this balcony.
- 41:39Umm, you can see these very small shiny objects here.
- 41:43Actually well resolved the edge of this towel hanging up here.
- 41:48It's it's much better. But but Tom, in this image, with
- 41:53all due respect, there's there's also a plasticky kind of feel to
- 42:01the to the towel that's hanging. Sure.
- 42:04This is, this is an extreme zoom case, like really or something
- 42:09pushing beyond the limits of the actual sensor.
- 42:12So there is actually no physical texture detail here.
- 42:15What we're able to do is to correct those edges in this case
- 42:18and give something that's, you know, when you zoom out a little
- 42:22bit it it, I think you'll agree it's it's crisper, it's clearer.
- 42:27It's not just adding contrast, it's it's that clarity that
- 42:30that's for sure the mid frequency details become a lot
- 42:35crisper and and more natural when you zoom it, when when you
- 42:38see it like that, that was 100 X image zoomed in the pixel level.
- 42:43So when you view it on a normal screen it just looks a lot
- 42:46better. Yes, it's very clear.
- 42:48We have another question and this is from Steph Bishop.
- 42:52What is the relative importance of correction for lens defects
- 42:56compared to coping with noise? If you're in sunny day outside,
- 43:01the importance of correcting the lens is more dominant than
- 43:05correcting the sensor noise. If you're in I, I'm saying low
- 43:09light, but just put things in perspective.
- 43:12Anything indoor for phones is low light, It's going to be
- 43:14noisy unless you have some very fancy studio lights in your
- 43:17house. But even in any normal house
- 43:19with lights on is already low light.
- 43:22And in that case, I would say that both are as important
- 43:25correcting sensor noise and the the lens effect and, and just
- 43:28emphasize again, I know we said a few times, we're not
- 43:31correcting them sequentially. They're being corrected at at
- 43:34one pass with one network, which is very important.
- 43:37And I think when you go to extreme low light, like let's
- 43:40say 0.1 lux or something like that, then the noise becomes
- 43:43more dominant. Because if if you have a 12
- 43:46megapixel camera like you have on most, most phones and you're
- 43:49now in extreme low light, you would be very thankful to get a
- 43:53good quality one megapixel image, right?
- 43:55So, So what it means is that actually correcting like one
- 43:58pixel size operation is not super important.
- 44:01It's more about correcting noise.
- 44:02But again, back to my statement, if you correct them separately,
- 44:06you're never going to achieve optimal results when you correct
- 44:08them together. Even this extreme low light,
- 44:11it's very important. And maybe to kind of help
- 44:16explain it, if you have a lot of noise, right, and you have a
- 44:18fancy traditional denoising algorithm, what do you want to
- 44:22do? You basically want to average
- 44:24pixels out, right? But you don't want to average
- 44:26pixels out when you reach an edge, you have an edge.
- 44:28You want to preserve the edge, right?
- 44:31Now, the definition of an edge actually depends on the lens,
- 44:33not just as you can find an edge as a big transition, but if
- 44:37you're not taking into account what your lens can do and not do
- 44:40or the blurve the lens, you're not going to denoise perfectly.
- 44:44You'll denoise but not as good as you can.
- 44:46So take taking these things together as as as one and using
- 44:50one piece of algorithm, a neural network in our case, to do these
- 44:54things together gives you optimum.
- 44:55Results. It depends on the character of
- 44:58the lens and the sensor as well the pixel size versus the amount
- 45:02of aberration you have on the lens.
- 45:04And also not just the, the, the size of the blurb, but the, the
- 45:08nature. So some, some lenses have that
- 45:11are easy to correct, some have ones which are harder to
- 45:14correct. And so that the noise sort of
- 45:17operates at different spatial frequencies.
- 45:19So your local frequencies, your course details are easier to
- 45:22correct and the high frequencies, the very fine
- 45:24details are very hard to correct.
- 45:26Typically it's where the noise tends to dominate.
- 45:29One thing that we haven't talked about that I think is
- 45:31interesting to bring up is this idea of deep optics, which is
- 45:35something else we're working on, which is more in the realm of
- 45:39joint hardware and software design.
- 45:42So given that we have this powerful neural ISP, this glass
- 45:47AI that we've developed the the software that can correct these
- 45:50lens operations. Another thought is like, how do
- 45:54you produce better hardware taking into account of that
- 45:58software capability? And that's where you can
- 46:00actually engineer and design that blur that that point spread
- 46:04function of the lens such that the the neural network can
- 46:09extract the maximum detail and it can overcome that, that trade
- 46:13off with noise and, and deep blurring or or sharpening.
- 46:17I understand now that the the model that you're building is so
- 46:26very thorough. Very basically you have this
- 46:28this very deep model of the various characteristics of the
- 46:33entire system and that's why you're able to make the changes
- 46:37that you make when you process the image.
- 46:39Ultimately, we have another question now more on the
- 46:44business side from Twitter. And the question is this, if you
- 46:49were, if using your system let's a smart a smartphone
- 46:54manufacturer use lower quality, cheaper camera parts, is the
- 47:00combined cost of the cost the the cost reduction plus your
- 47:07system less than the need to buy a more expensive phone or a more
- 47:12expensive imaging system? Smartphone cameras started 1015
- 47:17years ago, expensive with around $5 and the goal is to bring it
- 47:23down to $1.00. What actually happened is that
- 47:25the price went up like crazy because there's more cameras now
- 47:29and all of them have Oas and all of them have high resolution
- 47:32sensor and most phones have at least 4 cameras, 5 cameras
- 47:35today, or at least the, let's say flagship phones.
- 47:39The the bill of material of these cameras can easily reach
- 47:42$50 a $100 on some, some, some there's, there's one device even
- 47:46speculated to a few years ago that went up to well above 100.
- 47:51But, but I would say most of them fall within the several of
- 47:5310s of dollars. And what that means and, and it,
- 47:58it's a very good question. You, you could ask it the other
- 48:01way. Let's say that you don't have
- 48:02constraints on size. If you're a phone maker and you
- 48:05have some quality today, some you reach some quality.
- 48:08Let's say you have a score of 70 and you want to reach a score of
- 48:1390 or or 100, whatever the 100 is, you want to improve a lot.
- 48:17You have two options, right? One is to use our software.
- 48:21I'm I'm just using as an example.
- 48:23The other option is to pay more for hardware, use bigger
- 48:25sensors, bigger, better lenses, better sensors.
- 48:29The amount of money that you would have to pay to create the
- 48:31same gap in quality would be in the order of several 10s of
- 48:35dollars. And, and I'm putting aside the
- 48:36fact that it's not going to fit in a phone, which, which is a
- 48:39big deal too, but it's going to be several 10s of dollars.
- 48:42Where, where is, you know, not because I'm not, I'm not
- 48:44discussing, you know, our, our pricing models, but it, it's,
- 48:48it's, it's much lower than that, like an order of magnitude
- 48:50lower. So.
- 48:51So the bottom line is, if you're building a camera and you're
- 48:54debating between putting more expensive hardware or using the
- 48:57software coupled with your camera, you'll save a lot of
- 49:01money by using the software and not the hardware.
- 49:04Aside from money, there's an interesting thing to observe is
- 49:06the trend in in smartphone thicknesses over the years.
- 49:09Such, you know, Apple tried to keep them really thin for a long
- 49:12time and then other companies added these bumps that started
- 49:16growing thicker and thicker. Apple eventually followed that
- 49:19trend and now we've maybe reached a limit where these
- 49:22things barely the the flagship Android phones anywhere.
- 49:25They're maybe fourteen, 1516mm thick.
- 49:28They're barely fit in your pocket.
- 49:29And I think there's a desire to go back the other way.
- 49:31So they've added better and better hardware over the last
- 49:34five, 6-7 years. Now they're looking like, well,
- 49:38what do we do? So adding a, a software solution
- 49:41like ours is actually a way they can make it go thin again.
- 49:44And I think these things go in waves.
- 49:47That might be the next trend in in smartphones.
- 49:50So where is all of this going? What is the future of smartphone
- 49:55photography? And oh, by the way, why doesn't
- 50:00Apple just build this in? And you know, they have as much
- 50:06money as any company, so why haven't they just invented this
- 50:10and do this? It's.
- 50:11Difficult. I've, I've actually been working
- 50:13in this field for probably 20 years now.
- 50:17You know, obviously many, many smart people work on these kind
- 50:20of topics, but back when I started doing this, it was
- 50:24taking like a supercomputer to be able to to run deconvolution
- 50:30algorithms on, you know, they're taking weeks, but now we have it
- 50:33running in less than a second on a mobile processor.
- 50:37Doing that amount of engineering to actually make things run
- 50:40efficiently is super difficult. There's a lot of getting
- 50:44everything just right. As we said, we have to capture a
- 50:48lot of data under very many different conditions.
- 50:51If you make something wrong, then you can screw up the image
- 50:55and obviously you don't want to do that.
- 50:57But besides that, I think these big companies have, I don't
- 51:02know, a traditional thinking when it comes to how these image
- 51:06signal processing pipelines integrate in tunes.
- 51:10They have a lot of invested like thousands of people and many
- 51:12resources and momentum going that direction.
- 51:15So it's a little bit classical innovators dilemma that it takes
- 51:19a new mindset to be able to deploy this.
- 51:21And I think now is the, the time we have the, the processing
- 51:24power available and we, we've been actually working for
- 51:27several years to get this working well on these edge
- 51:32devices. But I think it's, it's, you
- 51:34know, the right moment for this technology to be deployed.
- 51:38And where is this all going? There's a short term where it's
- 51:41going and there's long term where it's going.
- 51:42Short term, I think we'll see more AI used in, in all phones,
- 51:47not just by class, by by any all the phone makers, by, by the
- 51:51chip makers, by the whole industry, drone security,
- 51:54cameras, everything. And it's happening already.
- 51:55So we'll see more AI and we'll also specifically see more AI,
- 51:59pixel level AI, which is what we're doing basically.
- 52:02And obviously it will get better over the years and there'll be
- 52:04competition between companies and one will do a little better
- 52:07than the other. So I, I think that's the short
- 52:10term. I think the more interesting
- 52:11answer is where is this going in the long term?
- 52:13In long term, in three years and above, it might, might be sooner
- 52:18is actually the change. You can drive a change to the
- 52:20hardware industry based on this AI.
- 52:23And I think that's super interesting.
- 52:25You know, there's, there's lots of new optics technologies,
- 52:28there's metal lenses, diffractive elements, various
- 52:34type of autofocus materials and, and, and mechanisms that things
- 52:41that work but are not enabled, that you can't use them without
- 52:46this AI or, or the non existing this stuff that we're developing
- 52:50with the deep optics that Tom mentioned.
- 52:53I'll give an example. There was an attempt to use
- 52:56under display cameras for smartphones, for the front
- 53:00facing camera. Some, some phone makers actually
- 53:03launched it and the the cams were just horrible, right.
- 53:07The reason for that is because putting a display of a camera
- 53:09serves kind of as a defractive grading and it creates a very
- 53:13messy ESF, which is point spread function.
- 53:15The the optics just becomes very, very aberrated, but also
- 53:19in a very non predictable way for a traditional algorithm to
- 53:22fix. That's something that for us,
- 53:24for example, is, is, is is is not an issue at all.
- 53:27We would just take that phone with under display camera and
- 53:31put in the lab and we, we get a perfect image out of it.
- 53:34So just just giving a small example of something that's
- 53:37enabled. It's a piece of hardware that's
- 53:39enabled by this, this technology that we developed.
- 53:44And I think you can extend it to cheaper lenses, thinner lenses
- 53:47also. So we, we will see in the coming
- 53:49years phones with the same sensor size, but much smaller
- 53:53thickness of the lens. So all these, you know, bumps on
- 53:55the phones will be able to get much, much smaller because, you
- 53:59know, one or two millimeter smaller then you don't need a
- 54:01bump at some point. It's also worth mentioning that
- 54:04the foldable phones are trending up now.
- 54:07We we just came back from a Mobile World Congress last week
- 54:10and Barcelona, me and and Tom, and we saw tons of foldable
- 54:13devices, you know, triple, triple folding, dual folding
- 54:17screens on all side. These foldable phones are very
- 54:20cool and useful because they give you a big screen, but they
- 54:22have a big problem. Any given section of the phone
- 54:26is actually very thin. So it means that when you design
- 54:29a camera to go in there, you now have half the space or maybe 1/3
- 54:33of the space that you have in a normal phone.
- 54:35And that just puts more constraints on the camera.
- 54:37So they're just using worse cameras today.
- 54:40But if we can give those cameras a big boost in quality enough
- 54:44that customers will say, OK, I'm going to buy this foldable
- 54:48device, even though it has small camera, I'm still getting the
- 54:50good image quality out of it. So that that's like a kind of
- 54:542nd order drive of technology. But I I just Long story short,
- 54:57being able to to use this AI to drive changes and hardware will
- 55:02lead to very interesting things. If you could show this image
- 55:05quickly, this is one we took from a drone as well.
- 55:08So we applied our technology on not just smartphones, but other
- 55:12platforms that have cameras. So zooming in here.
- 55:15These these are some friends from a well known photography
- 55:18website. They came to visit our office
- 55:21and you can see again we compared with the one on the
- 55:23left that comes straight out of the drone that we can resolve a
- 55:26lot more detail. We're able to apply our
- 55:30technology to anywhere that has a camera needs processing that
- 55:34runs on the edge like like also wearables like smart glasses and
- 55:37things like that, which are big up and coming trend we believe.
- 55:41That is a very significant difference.
- 55:45It's pretty amazing. Yeah, it's amazing.
- 55:48The difference between these two images very quickly.
- 55:52Large camera manufacturers have also been playing with software
- 55:59based corrections. For example, I believe that
- 56:03Fujifilm and some of their cameras have been doing that,
- 56:05others as well. Any thoughts on this kind of
- 56:10technology showing up in large cameras?
- 56:13But very quickly, please. It's coming.
- 56:15I mean, I think that behind in terms of the processes they have
- 56:18on their smartphones typically have better processes today
- 56:22because of higher shipping volumes.
- 56:24But as they start to integrate the same kind of Qualcomm chips,
- 56:27for example, then they will be able to run our software and we
- 56:30can develop versions that run on those as well.
- 56:33Do we lose anything by applying AI to photographs as opposed to
- 56:43just light going through glass on a sensor?
- 56:47You don't lose anything besides processing time in a little bit
- 56:50of battery power. But actually that even is not
- 56:54completely true because computational photography, non
- 56:57AI computer photography actually consumes more power than AI.
- 57:00Just because you know, Qualcomm and, and that the chip makers,
- 57:04they, they spend a lot of effort to making their chips very, very
- 57:07power efficient. And we can run insane amount of
- 57:11calculations on on that chip for a whole 2nd and they will
- 57:14consume less power than traditional algorithms running
- 57:17on CPUGP for the same amount of time.
- 57:19So. So did I think the answer is no.
- 57:22From your perspective, looking inside these smartphones, who's
- 57:27got the best camera? The short answer is some of the
- 57:31Chinese Android phones have a lot better hardware.
- 57:34There's flagship phones that are actually more expensive than the
- 57:37iPhone or any any Pixel or anything like that.
- 57:41But the software. Wait till we're shipping, then
- 57:45we'll have the best. But the image is a result of the
- 57:49combination of the hardware and the software.
- 57:52So would it be accurate to say that merely buying better
- 57:57hardware because you want a better image is an incomplete
- 58:01solution because you'd need to look at both?
- 58:05Yeah, you need to look at the total and there's, you know,
- 58:07there's a website called Dxo which they test cameras and they
- 58:11rank them. And if you go to the top of the
- 58:13chart, you will see some phones with more expensive hardware on
- 58:16it. Not surprising.
- 58:18But some of the phones up there actually are not that they have
- 58:21much cheaper hardware, just with better software.
- 58:23So there is kind of a like, like you said, it's a combination of
- 58:26the two and and if you have the best hardware and the best
- 58:30software, you'll get the best image quality obviously.
- 58:33And I know that DXO has rated your results very highly.
- 58:37So congratulations on that. Thank you very.
- 58:39Much And with that, we're out of time.
- 58:42A huge thank you to Ziv Atar and Tom Bishus Bishop.
- 58:47Thank you both so much for having.
- 58:49Us Michael, it was great nice talking to you and thanks to the
- 58:52audience for sending interesting questions also.
- 58:54And thank you to everybody who watched, especially you folks
- 58:57who asked such great questions. Now, before you go, check out
- 59:01cxotalk.com, subscribe to our newsletter so that you can join
- 59:06our community. We have really awesome shows
- 59:10coming up. This has been more of a
- 59:12technical deep dive than we usually do, but I thought it's
- 59:16such an important and interesting topic that thought,
- 59:19well, let's let's go for it and let me know in the comments or
- 59:23send a message, smoke signals, whatever you want.
- 59:27What kind of shows from CXO Talk do you want to see?
- 59:30So let me know because we are very responsive as far as that
- 59:34goes. Thanks so much everybody.
- 59:36Have a great day and we'll see you again next time.