How We Test Televisions
See How We Test Other Products:
Reviewed.com Televisions tests TVs with a rigorous set of scientific methods that employ the same tools and techniques as the manufacturers. Rather than just looking at an ad hoc set of images and videos on the screen, we perform an in-depth quantitative analysis using advanced instrumentation and professional tools that look at the performance of the display, determining how it produces on-screen images in extreme detail.
In addition to testing these TVs from a scientific perspective, we also look at qualitative questions that are aimed more at consumer usage, such as menu ease of use and what kind of smart apps are available. For more information about our objective TV testing and our subjective TV assessments, read on!
Instrumentation and Data Analysis
One of the ways we ensure the reproducibility of our measurements is using the Quantum Data QD780 signal generator. When connected (via HDMI cable) to the TV, this device sends out standard patterns, such as a checkerboard or a series of colored bars, which we can alter to meet our needs. Measuring color and dynamic range values for the same pattern ensures that our data are comparable across multiple TV sets.
To analyze the chromaticity of a TV display, we use a Konica Minolta CS-200 Chroma Meter, which provides extremely accurate color measurements for all display technologies. It has a narrow one-degree acceptance angle, which is very important for accurately measuring the luminance and color of TV panels. The CS-200 can measure colored light sources in the range of 0.01 to 20,000,000 cd/m2, with an absolute accuracy of +/- 0.02 cd/m2. The CS-200 connects to a PC via a USB port, and every data sample is logged.
When color data is not needed, we measure the luminance values of a TV using a Konica Minolta LS-100 Luminance Meter, an instrument built specifically for the measurement of dynamic light values. The LS-100 is capable of measuring light sources ranging from 0.001 to 299,900 cd/m2, and as a handheld spot instrument, is capable of reporting both absolute and variable readings very quickly with a ±2% degree of error.
Additional measurements and calibration are done with the SpectraCal CalMan Ultimate software package, which can remotely control both the Konica Minolta CS-200 Chroma Meter and the Quantum Data QD780 signal generator to great effect.
For roughly 24 hours prior to testing the TV, we plug it in and let it warm up. We also try out the different viewing modes with multiple types of content to see which mode is most like the experience you would have in a movie theatre; that is the viewing mode that will be used for testing purposes.
Our objective tests take place prior to the actual calibration process because very few consumers will end up having their TVs professionally calibrated. The test data, therefore, is based almost entirely on the TV’s straight out-of-the-box experience, which is how most people will actually watch their content.
We do not score TVs by altering controls that are hidden or require special access codes or equipment to access (such as those designed for professional installers or for use in calibrating the TV at the factory); if the control is not easily accessible to an everyday user, we don’t use it to score the display. We do this because we want to get the same experience that a user would get if they bought the display and then set it up, and most users will not be able to access the service menus.
If the light/color meter registers any ambient light or any reflections from the screen itself, it can lead to artificially high luminance values that are not representative of the screen’s actual performance. To prevent this, the walls of our TV lab are covered in Duvatyne, a special back fabric that both blocks sources of light, as well as absorbing light, instead of reflecting it. Any light that reaches the measuring device comes directly from the target on the screen, not from the room itself.
Part of a TV’s final score is its features and capabilities. Is it an LED-LCD TV, or an OLED TV? Is the TV HDR capable? What type of audio wattage is built in? How many HDMI and USB ports are available? We ask these questions and many more, with our questions targeted towards features that are the most important and most useful to the people actually buying these TVs.
We put each television through a battery of objective tests, and use the resulting nit (=cd/m2) measurements to assess how well the television performs. In addition to testing the TV in standard definition, however, we are staying on the cutting edge of television tech by also testing TVs in their HDR-enabled viewing modes, if HDR is available.
Black Level, Peak White, and Contrast Ratio
To note the black level of the display, we measure the luminance of the black squares on an industry standard ANSI checkerboard screen in nits. We measure the black level several times during testing, reporting on any variance we see with these multiple measurements and discussing any dynamic backlight or local dimming functions as we go. However, the main figure that we quote is for the black level. Our score is based on how dark the black is: the lower the luminance, the higher the score.
To measure the brightest white the TV can achieve, we measure the luminance of the white squares on the same ANSI checkerboard screen in nits. Our score is based on how bright the white is: the brighter the white, the higher the score.
To calculate the contrast ratio of a TV panel, we divide the aforementioned peak white luminance by the deepest black luminance. So, if a display has a deepest black of 0.4 cd/m2 and a peak white of 400 cd/m2, the contrast ratio is 1000:1. Our score here is based on how high the ratio is, where a higher contrast ratio means a higher score.
Tunnel Contrast and White Falloff
Tunnel contrast refers to the increased brightening of a black rectangle on a white background as the black rectangle gets smaller. White falloff refers to the opposite, where the luminance of a white rectangle on a black background generally decreases with the decreasing size of the white rectangle.
While black level, peak white, and contrast ratio give us information about the performance of the screen showing just pure whites and pure blacks, tunnel contrast and white falloff give us valuable insight into the screen performance in the more realistic situation of mixed white and blacks shades. Most displays have problems with this mixing: smaller objects in darker shades can appear washed out when surrounded by much brighter shades, and brighter shades are less bright when surrounded by much darker shades and shapes. High quality panels will be able to maintain consistently high white levels with darker backgrounds (low white falloff) and consistently low black levels in brighter backgrounds (low tunnel contrast).
Our viewing angle test examines the contrast ratio of the display at different viewing angles. We measure the contrast ratio at 10° increments from 0° (straight on) to ±80°. Our scoring for this test is based on the point at which the contrast ratio has fallen by 50 percent from the maximum we measured at 0°. This means that our ranges of satisfactory viewing angles are very different from the ones the manufacturers publish, which are generally based on the angle at which the contrast ratio falls to 10:1. We feel that this is far too low, since most displays have a face-on contrast ratio of over 1000:1, making a 10:1 contrast ratio unwatchable.
“Grayscale Gamma” refers to the way TVs translate an input signal (a voltage or a binary signal) to the amount of light that is produced on the screen. Humans have a tough time distinguishing the many light shades of a linear grayscale (where the signal input directly correlates to the output light signal), so the TV uses an exponential transfer function (“gamma”) to convert the input signal into a grayscale containing more dark shades that the human eye can actually see.
We measure the luminance of screens with varying signal intensities of gray from 0 to 255. There are different ideal transfer functions for standard definition and HDR sets, but the more closely the measured gray shades match the ideal luminance levels for that data type, the more a viewer will be able to discern the subtle shading of an image on the screen, and the higher the grayscale gamma score will be.
To visualize the range of possible colors that a TV can display, the possible color gamut is often shown on a color space, or a plot where individual colors have specific x- and y-coordinates. Using the CalMan software, we quantify the exactly color values for the main display colors (white, red, blue, green, yellow, magenta, and cyan) in the CIE 1976 Lu’v’ color space, and compare them to one of two possible color spaces: Rec.709 (for 8-bit color encoding) or DCI-P3 (for 10-bit color encoding).
The scoring for this test is based on the distance between the measured and standard values; the greater the distance, the lower the score. Note that a color value that falls outside the highlighted triangles is also undesirable; this will produce incorrect colors that are too saturated and not as the content producer intended.
The color of white that a TV produces can vary significantly with factory settings and picture modes. The correlated color temperature (CCT) specifies the exact color of white for a given TV panel.
Our CS-200 can measure the CCT very accurately. We use this to determine the coordinates of the D65 point, which is the ideal while value shown in the above color space. D65 approximates the color of daylight at noon on an overcast day and includes components of the both blue sky and direct sunlight.
While we actually score TVs based on their pre-calibration settings, we do ultimately calibrate the panels we test so that anyone who buys that TV can use our settings to calibrate their panel to get the best possible picture quality and performance. Right out of the box, almost all TVs default to a picture mode called “Vivid” or “Dynamic.” These modes are for big box stores, engineered so that the TV looks best in a brightly lit retail showroom. As a result, the TVs are set for maximum brightness, contrast, color, and bright room gamma, rather than correct picture settings per international standards. It’s common knowledge that even professional graphics monitors can shift in calibration during shipping, and often need a second look. Calibration is simply a part of a display’s life cycle.
After a period of break-in time, we follow the ISF standard calibration process for TVs rendering content in standard dynamic range (SDR) to tweak each TV to optimal settings. This involves setting up the TV in the lab with the Konica Minolta CS-200 and QuantumData 780a feeding data via USB to our testing computer. The QuantumData signal generator is controlled by the CalMan software, and outputs a pure HDMI signal to the TV in sync with the measurements collected by the CS-200.
The first part of calibration involves setting the TV to its most accurate manufacturer preset, usually called “Cinema” or “Movie” mode. We then take “as is” measurements to determine the starting points for the display’s primary and secondary color points, gamma correction curve, maximum luminance, and grayscale (white/gray point) sub-pixel balance.
Following the ISF standard calibration workflow, we proceed to set the display’s basic controls to standard legal limits. A maximum luminance of about 140 cd/m2 is desirable for the smoothest transfer function between color and grayscale steps. We set reference black and reference white to preserve black detail above step 16 through step 235, and use the TV’s gamma control (when available) to achieve the ideal gamma sum.
When the basics are done, we adjust the TV’s RGB sub-pixel balance to achieve the most even possible emphasis on each color filter, beginning with 2-point white balance controls and proceeding to 10-point or 20-point controls if necessary (and if available). This involves adjusting the “high” and “low” color emphasis between 10-50 and 60-100 IRE to eliminate grayscale error.
If a CMS (Color Management System) is available within the TV’s menu, we adjust the hue, saturation, and luminance of the TV’s color points, focusing on proper luminosity for primary colors (RGB) and proper hue/saturation for secondary colors (CMY). Proper gamma, grayscale, and color points are checked both at an 18% raster and full field to ensure uniformity.
When calibration is complete, we post “before” and “after” results, as well as the final calibration settings for all TVs on the Science Page. All televisions are currently scored based on their “before” results rather than on the results of what would be an involved calibration process, to show consumers both what to expect out of the box and what the display is ultimately capable of.
While the above-mentioned tests give us hard data that tells us whether a TV is technically a good product (or not), we also want to be able to comment on the more intangible aspects of owning and using a given TV. After using the TV for testing and more informally, we answer questions such as: How easy is it to use the TV interface? Are the smart features actually worth the money? If it has HDR, how good is that HDR? If a TV is technically perfect but not very user friendly, then a product’s overall score will suffer.
To see how we’ve rated the latest TVs, head over to our library of TV reviews. If you want to read about our favorite TVs, check out the Best Right Now articles, where we talk about everything from the Best Big Screen TVs under $1000, to the Best Gaming TVs to the Best HDR TVs.
SpectraCal, located in the Seattle area is the world’s leading provider of video display calibration software for both professional and consumer needs. SpectraCal constantly drives innovations in the field, providing the tools and training necessary to achieve accurate digital images and assisting customers with the step-by-step process of screen optimization.