Washington Tree Fruit Research Commission

Research Reports

Hyperspectral Imaging for Assessing the Quality and Condition of Apples (2004)

FINAL PROJECT REPORT
WTFRC Project #PH-01-135
YEAR 0/0
Organization Project #58-3635-2-435
Title:Hyperspectral Imaging for Assessing the Quality and Condition of Apples
PI:Renfu Lu
Organization:USDA ARS, Michigan State University, East Lansing, MI
 PDF version of report

Co-PIs

Randolph M. Beaudry, Horticulture

Daniel E. Guyer, Agricultural Engineering

Michigan State University, East Lansing, MI

 

Objectives

The overall objective of the project was to develop a novel light-based sensing technique to assess internal quality (firmness and sugar content or soluble solids content) of apples.  Specific objectives were to:
1.       Investigate a new sensing method, using hyperspectral and multispectral imaging, for quantifying the absorption and scattering properties of apple fruit;
2.       Develop mathematical/statistical approaches and computer algorithms to relate the absorption and scattering properties of apples to fruit firmness and soluble solids content (SSC); and
3.   Develop a prototype for sensing the firmness and SSC of apple fruit.

Significant findings

·         Assembled and tested a hyperspectral imaging (HSI) system for acquiring light absorption and scattering from apple fruit over the visible and short wave near infrared region.  The hyperspectral imaging system gave good predictions of fruit firmness and SSC, which demonstrated that light scattering is useful for measuring fruit firmness whereas light absorption is more closely related to chemical constituents of fruit such as SSC.
·         Assembled and tested a multispectral imaging (MSI) system from the initial version that could only acquire one scattering image each time to an improved version for acquiring multiple scattering images simultaneously. Firmness predictions by the MSI are comparable to those from HSI.
·         Designed, assembled, and performed preliminary tests on, a mini-size laboratory prototype which is capable of running at a speed up to two fruit/s.  Preliminary evaluation showed that the prototype is promising for sorting and grading fruit, but we also need to have an improved light source and better synchronization of the imaging system with the conveyor to improve firmness and SSC predictions.
·         Assembled and tested a compact multispectral imaging unit with a low cost CCD (charge-couple device) camera for measuring fruit firmness and SSC. Results from the 2003-2004 experiments are encouraging, demonstrating that a low cost, portable device can be developed for measuring fruit firmness and SSC.
·         Spectral scattering profiles are described by the proposed mathematical model (i.e., Lorentzian function) accurately and model parameters are related to fruit firmness and SSC.
·         Integration of light absorption and scattering resulted in better firmness predictions than did conventional near-infrared spectroscopy (NIRS).   Our firmness prediction results are better than or comparable to those with mechanical methods (such as sonic and impact).  The technique will provide a viable solution for sorting and grading apple fruit.  It is fast and easier to implement for online applications and, more importantly, has potential for measuring multiple quality attributes simultaneously.

 

Methods

Conventional NIRS provides approximate measure of light absorption in a sample.  The technique is useful for measuring fruit quality attributes that are related to chemical constituents such as sugar and acid.  Fruit firmness is related to the structure and mechanical strength of fruit tissue, which is difficult to measure with NIRS.  In this research, we proposed a new concept of measuring light scattering and absorption by using hyperspectral and multispectral imaging techniques in order to predict apple fruit firmness and SSC.

During the first year, we assembled a hyperspectral imaging (HSI) system (Fig. 1a), which mainly consisted of a high performance CCD camera, an imaging spectrograph, and a computer controlled DC quartz tungsten halogen (QTH) light source with specially designed focusing optics for generating a sharp, focused light beam.  The HSI was tested with Golden Delicious, Jonagold, and Red Delicious apples.  We also assembled and tested a multispectral imaging (MSI) system which employed a filter-wheel device for selecting specific wavelengths.  The MSI acquired five spectral scattering images in sequence from each fruit in the visible and near-infrared region, which would take more than 10 seconds to complete.  The MSI was tested with Red Delicious apples.  After completing images acquisition, fruit firmness was measured with the standard Magness-Taylor (MT) firmness testing instrument and SSC was determined from the juice released during MT measurement with a digital refractometer.  These destructive measurements were used as a reference, against which HSI and MSI were compared. Mathematical methods were studied for extracting essential information from the scattering profiles for apple fruit.  Calibration models were developed by using both neural networks and statistical methods for predicting fruit firmness and SSC.  System performance was validated with independent apple samples.

Based upon results from the first year study, our research in the second year was focused on improving the HSI for faster and more reliable acquisition of hyperspectral scattering images from apples.  The HSI (Fig. 1a) was modified and the light source was refined so that the system could acquire hyperspectral scattering images from individual fruit within 0.25 s.  We also made significant improvements for the MSI by using a common aperture multispectral imaging spectrograph.  The improved MSI acquired spectral scattering images for four wavelengths simultaneously, making it possible for real time detection of apple fruit quality.  We also tested and improved light source design for more efficient and effective acquisition of light scattering from apple fruit.  Extensive effects were made on optimizing the neural network for improved predictions of fruit firmness and SSC.

For the third year, a compact multispectral imaging unit, which uses a low-cost CCD camera (Fig. 1b), was assembled.  Experiments were performed on Golden Delicious and Red Delicious with this unit as well as a visible/NIR spectrometer to compare the two techniques for measuring fruit firmness and SSC.  We designed and assembled a mini-size laboratory packing line to test the MSI for real time detection of apple fruit firmness and SSC (Fig. 1c).  A specially designed light source with four laser diodes was used with the MSI prototype.  In-house computer programs were developed and integrated into the prototype so that it can acquire and process scattering images and predict fruit firmness and SSC in real time as apples pass through the imaging station for varying conveyor speeds up to two fruit/s.  The prototype was tested and evaluated with Golden Delicious and Red Delicious apples at a speed of 0.5 fruit/s.  In addition, we evaluated different mathematical methods for quantifying hyperspectral and multispectral scattering profiles.  New mathematical models were proposed for describing spectral scattering profiles and relating model parameters to fruit firmness and SSC.

Results and discussion

1.       Hyperspectral Imaging System (HSI)

Firmness and SSC predictions of Golden Delicious and Red Delicious apples for the first two years are shown in Fig. 2.   The HSI was able to predict fruit firmness with r=0.74 and 0.69 for Golden Delicious and Red Delicious apples in the first year (2001-2002).  These prediction results are considered relatively good given the fact that MT firmness measurements are highly variable (see more discussion on MT firmness later).  SSC predictions for Red Delicious apples were obtained with r=0.81 and the standard error of validation (SEV) of 0.57 Brix. 

For the second year (2002-2003), the improved HSI gave better firmness predictions with r=0.85 and 0.72 for Golden Delicious and Red Delicious, respectively (Fig. 2).  The SEVs (less than 7 N or 1.6 lbs.) were also lower compared to the first year study.  Good SSC predictions were obtained for Golden Delicious (r=0.89 and SEV=0.72 Brix) and Red Delicious (r=0.78 and SEV=0.89 Brix).  Overall, the results from the second year are better than those obtained in the first year as a result of improved hardware and algorithms.

Hyperspectral imaging acquires a large amount of spectral and spatial information from apple fruit in a short time. The technique is thus well suited for measuring light scattering in apple fruit.  Our first two year study indicated that HSI could achieve improved predictions of fruit firmness, even though it did not seem to be advantageous over NIRS for SSC measurements.  This latter finding was expected since HSI was primarily designed for quantifying light scattering.  We also found that our current system, though fast in acquiring scattering images, had one major drawback in that it line scans the fruit.  This means that hyperspectral scattering images could be influenced by factors such as fruit size and the shape or surface geometry of the fruit.  These factors certainly warrant our further investigation.  This drawback could be overcome by using multiple scans to improve the signal repeatability, but in doing so it would be accompanied with increased imaging time, which is not desirable for real time applications.  Hence, we adopted multispectral imaging for developing the prototype to quantify spectral scattering in apple fruit, which provides more reliable and consistent scattering measurement over HSI for selected wavelengths.

2.       Multispectral Imaging System (MSI)

Over the past three years, we have tested different versions of MSI for measuring fruit firmness and SSC.  The first version of MSI employed a motor-driven filter wheel for selecting wavelengths.  In the first year, we obtained good firmness predictions for Red Delicious with r=0.87 and SEV=5.8 N or1.3 lbs. (Fig. 3).  For SSC predictions, we had r=0.77 and SEV=0.78, which were not particularly impressive compared with HSI. 

In the second year, we adopted the common aperture design for our MSI system.  While the improved MSI enabled us to acquire spectral scattering images for four wavelengths simultaneously, it also entailed some expense on system performance.  The improved MSI was less efficient in acquiring light scattering and had a lower spatial resolution because different optical configurations were used.  We had firmness predictions with r=0.76 and 0.73 for Golden Delicious and Red Delicious, respectively (Fig. 3).  These results are decent but not as good as those for the first year. 

For the third year, we compared our compact MSI unit with a visible/NIR spectrometer for measuring fruit firmness of Golden Delicious and Red Delicious.  Results showed that the compact MSI gave good firmness predictions with r=0.84 and 0.82 for Golden Delicious and Red Delicious, respectively, which are considerably better than those from NIRS (r=0.60 and 0.52, respectively) (Fig. 4).  This compact MSI took scattering images in sequence and was similar in design to the first version of MSI with the filter wheel device.  That may explain why MSI results for the first and three years were comparable.  We used a new mathematical model for better description of scattering profiles, which should also lead to improved firmness predictions.  For NIRS measurements, we used interactance mode, which was considered best among three sensing modes (i.e., diffuse reflectance, transmission, and interactance) studied by others for measuring fruit internal quality.  The NIRS spectrometer covered the spectral region between 500 and 1100 nm.  These results suggest that MSI is superior to NIRS for predicting fruit firmness because light scattering is associated with the density and structural characteristics of apple fruit.  

Our preliminary evaluation of the prototype (running at a speed of 0.5 fruit/s) showed relatively poor firmness predictions with r=0.52 and 0.60 for Golden Delicious and Red Delicious, respectively (Fig. 5).  These results are considerably worse than those obtained with the compact MSI, although both systems were based on the same principle of measuring light scattering.  Efforts were made to find possible causes for the poor performance of the prototype.  As discussed earlier, MSI with the common aperture multispectral imaging spectrograph may have caused some comprise in system performance.  We identified three factors that may have also affected the prototype performance.  First, the specially designed laser light source was much less stable than we had expected.  A stable light source is critical for the MSI.  Second, synchronization of the imaging system with the conveyor is important.  The system should take images when a fruit is at the right position.  The photocell detector used in the prototype was triggered when a fruit reached the pre-selected position, and it did not consider size difference.  This could have some effect on the timing of the MSI to acquire images from each fruit. And third, some distortion in the images taken from the moving fruit was noticed.  Our current program did not have the capability of correcting the image distortion.  We need to add this capability in our program for better extraction of scattering profiles from each image.

The MT firmness test is well known for its variability and inconsistency because it only measures the maximum force (point measurement) required for the penetrometer to penetrate into fruit flesh for a pre-specified distance.  To improve MT repeatability and reliability, we tried other firmness parameters extracted from the force-deformation curves recorded during MT measurements.  Our preliminary analysis (results not shown here) indicated that the energy required for penetrating the MT probe into the fruit may be a better measure of fruit firmness; this parameter is highly correlated with MT force and more reliable.  We will continue to investigate this issue so that nondestructive sensing techniques are evaluated and judged against a better, more consistent MT firmness measure.

In summary, we investigated a new method of measuring light scattering for predicting fruit firmness and soluble solids content (SSC).  Hyperspectral and multispectral imaging systems were developed for acquiring scattering images from apples.  Mathematical methods and computer programs were created for describing light scattering profiles and relating them to fruit firmness and SSC.  Results showed that the proposed method can provide a viable solution to sorting and grading apple fruit for internal quality, especially firmness.  We have assembled and performed preliminary tests on a compact multispectral imaging unit and a prototype for real time detection of fruit quality.  Good firmness predictions were obtained with the compact unit but the results from the prototype were not satisfactory.  We have identified areas of improvement for the prototype.  Our next step is improve and refine the prototype, correct the problems identified, and develop algorithms to classify fruit into two or three firmness grades.  We will also continue our research on developing a low-cost, portable sensing unit for measuring fruit firmness and SSC. 
 

 

 

                 

 

  (a) Hyperspectral imaging system.                                     (b) A compact multispectral imaging unit

 

 

 


 

(c) A prototype multispectral imaging system capable of inspecting apples at a speed up to 2 fruit/s.

 

 

Figure 1.  Hyperspectral and multispectral imaging systems developed for measuring internal quality of apple fruit.

 

      r = 0.74

SEV = 8.3 N
Red Delicious
Golden Delicious

 

Red Delicious

Year 2001-2002

 

Red Delicious

Golden Delicious
 

Red Delicious

Golden Delicious
        Year 2002-2003
Figure 2.  Hyperspectral imaging for predicting firmness and soluble solids content (SSC) of apples for years 2001-2002 and 2002-2003.

 

 

Red Delicious


Red Delicious

 

 

Year 2001-2002

 

 

Red Delicious

 

Year 2002-2003

 

 

Figure 3.  Firmness and soluble solids content (SSC) predictions by the multispectral imaging system. 

 

Red

Delicious
Golden Delicious
(a)

Golden Delicious

Red
Delicious

(b)

 

Figure 4.  Comparison of firmness prediction results obtained from the compact multispectral imaging unit (a) and a visible/short-wave near-infrared spectrometer (b) in year 2003-2004. Apples tested in (a) were from a subgroup of the samples in used (b).

 

 

Red

Delicious
Golden Delicious

Figure 5. Firmness predictions obtained from the multispectral imaging prototype running at a speed of 0.5 fruit/s in year 2003-2004.

 

WA Tree Fruit Research Commission , 1719 Springwater Ave, Wenatchee WA 98801, 509-665-8271, Contact Us

The WTFRC Research Reports database is hosted on the servers of WSU-Tree Fruit Research & Extension Center