One of my tasks here at Abel is to evaluate new technologies and products so that we can better understand them and better advise our clients. This is the first entry in a series meant to give some insight into how I go about these evaluations. I’ll be concentrating on the process, not the results. Also, I am constantly improving and expanding on my testing techniques, so these will be snapshots into evolving processes. If you will be at NAB this year, feel free to stop by our booth to discuss any of this in person, or if not, make an appointment to see me some time in the office.
The Big Picture
If you make your living in the film or television production business, you probably have a pretty good intuitive sense of what kinds of pictures you like. You probably have also developed a vocabulary to describe images and the qualities that you’re looking for. They may be warm or cool, punchy or flat. You may describe pictures that have something wrong with them as being thin, noisy or plasticky. Sometimes you may describe certain pictures as being filmic or video-like.
Generally, there are a number of factors that create a particular look — the way colors are reproduced, the sharpness of edges, noise or grain, etc. One of the main goals of the testing that I do is to identify those specific qualities in an image that evoke a certain feeling.
There are many tests that produce easily quantifiable results: more resolution is generally better than less, a faster download is always better than a slower one, etc. However, many of these factors tend to counteract each other. For instance, all things being equal, higher resolution tends to imply lower sensitivity. When evaluating a product, it’s important to take into consideration the specific applications for it and find the best compromise for your use.
The Fine Details
Each further entry in this series will concentrate on an individual realm of testing. Some topics will include resolution and sharpness, compression, dynamic range and noise. With all of these subjects we strive to follow certain principles:
Specificity: Because there are so many factors that go into making an image, it can be difficult to isolate which individual factors are affecting the image in which ways. Therefore, we design tests that isolate these factors as much as possible.
Reproducibility: In order for tests to be meaningful, it is necessary for them to be reproducible. We need to make sure that the same tests are performed exactly on different equipment. If we perform the same test on the equipment, then we expect to get similar results or we know something is wrong with the test.
That Certain Something
With all of the concentration on objective tests, it’s easy to lose sight of the real purpose of the equipment we use – to present our artistic vision and provoke an emotional response. It’s important when we’re evaluating a piece of gear, that we keep our eyes open for the intangible qualities, whether it’s in the images created by a camera, or the way a piece of gear feels to use.
As an example, consider these two photographs of the same subject, taken at about the same time.
The first picture was taken with a digital camera, and by almost any measurement it is technically superior to the second. The second picture was taken with a cheap plastic toy camera, yet it is much more successful as a photograph. After all the testing is done and the numbers are in, we still need to look at the gear we review with an artistic eye. There are some qualities that just can’t be measured by any scientific method. So our final step in any evaluation is to determine if the piece of gear has a certain quality that appeals to our aesthetic values.