1 /*#******************************************************************************
2 ** IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.
4 ** By downloading, copying, installing or using the software you agree to this license.
5 ** If you do not agree to this license, do not download, install,
6 ** copy or use the software.
9 ** HVStools : interfaces allowing OpenCV users to integrate Human Vision System models. Presented models originate from Jeanny Herault's original research and have been reused and adapted by the author&collaborators for computed vision applications since his thesis with Alice Caplier at Gipsa-Lab.
10 ** Use: extract still images & image sequences features, from contours details to motion spatio-temporal features, etc. for high level visual scene analysis. Also contribute to image enhancement/compression such as tone mapping.
12 ** Maintainers : Listic lab (code author current affiliation & applications) and Gipsa Lab (original research origins & applications)
14 ** Creation - enhancement process 2007-2011
15 ** Author: Alexandre Benoit (benoit.alexandre.vision@gmail.com), LISTIC lab, Annecy le vieux, France
17 ** Theses algorithm have been developped by Alexandre BENOIT since his thesis with Alice Caplier at Gipsa-Lab (www.gipsa-lab.inpg.fr) and the research he pursues at LISTIC Lab (www.listic.univ-savoie.fr).
18 ** Refer to the following research paper for more information:
19 ** Benoit A., Caplier A., Durette B., Herault, J., "USING HUMAN VISUAL SYSTEM MODELING FOR BIO-INSPIRED LOW LEVEL IMAGE PROCESSING", Elsevier, Computer Vision and Image Understanding 114 (2010), pp. 758-773, DOI: http://dx.doi.org/10.1016/j.cviu.2010.01.011
20 ** This work have been carried out thanks to Jeanny Herault who's research and great discussions are the basis of all this work, please take a look at his book:
21 ** Vision: Images, Signals and Neural Networks: Models of Neural Processing in Visual Perception (Progress in Neural Processing),By: Jeanny Herault, ISBN: 9814273686. WAPI (Tower ID): 113266891.
23 ** The retina filter includes the research contributions of phd/research collegues from which code has been redrawn by the author :
24 ** _take a look at the retinacolor.hpp module to discover Brice Chaix de Lavarene color mosaicing/demosaicing and the reference paper:
25 ** ====> B. Chaix de Lavarene, D. Alleysson, B. Durette, J. Herault (2007). "Efficient demosaicing through recursive filtering", IEEE International Conference on Image Processing ICIP 2007
26 ** _take a look at imagelogpolprojection.hpp to discover retina spatial log sampling which originates from Barthelemy Durette phd with Jeanny Herault. A Retina / V1 cortex projection is also proposed and originates from Jeanny's discussions.
27 ** ====> more informations in the above cited Jeanny Heraults's book.
30 ** For Open Source Computer Vision Library
32 ** Copyright (C) 2000-2008, Intel Corporation, all rights reserved.
33 ** Copyright (C) 2008-2011, Willow Garage Inc., all rights reserved.
35 ** For Human Visual System tools (hvstools)
36 ** Copyright (C) 2007-2011, LISTIC Lab, Annecy le Vieux and GIPSA Lab, Grenoble, France, all rights reserved.
38 ** Third party copyrights are property of their respective owners.
40 ** Redistribution and use in source and binary forms, with or without modification,
41 ** are permitted provided that the following conditions are met:
43 ** * Redistributions of source code must retain the above copyright notice,
44 ** this list of conditions and the following disclaimer.
46 ** * Redistributions in binary form must reproduce the above copyright notice,
47 ** this list of conditions and the following disclaimer in the documentation
48 ** and/or other materials provided with the distribution.
50 ** * The name of the copyright holders may not be used to endorse or promote products
51 ** derived from this software without specific prior written permission.
53 ** This software is provided by the copyright holders and contributors "as is" and
54 ** any express or implied warranties, including, but not limited to, the implied
55 ** warranties of merchantability and fitness for a particular purpose are disclaimed.
56 ** In no event shall the Intel Corporation or contributors be liable for any direct,
57 ** indirect, incidental, special, exemplary, or consequential damages
58 ** (including, but not limited to, procurement of substitute goods or services;
59 ** loss of use, data, or profits; or business interruption) however caused
60 ** and on any theory of liability, whether in contract, strict liability,
61 ** or tort (including negligence or otherwise) arising in any way out of
62 ** the use of this software, even if advised of the possibility of such damage.
63 *******************************************************************************/
65 #ifndef BASICRETINAELEMENT_HPP_
66 #define BASICRETINAELEMENT_HPP_
72 * @class BasicRetinaFilter
73 * @brief Brief overview, this class provides tools for low level image processing:
74 * --> this class is able to perform:
75 * -> first order Low pass optimized filtering
76 * -> local luminance adaptation (able to correct back light problems and contrast enhancement)
77 * -> progressive low pass filter filtering (higher filtering on the borders than on the center)
78 * -> image data between 0 and 255 resampling with different options, linear rescaling, sigmoide)
80 * NOTE : initially the retina model was based on double format scalar values but
81 * a good memory/precision compromise is float...
82 * also the double format precision does not make so much sense from a biological point of view (neurons value coding is not so precise)
86 * // create object at a specified picture size
87 * BasicRetinaFilter *_photoreceptorsPrefilter;
88 * _photoreceptorsPrefilter =new BasicRetinaFilter(sizeRows, sizeWindows);
90 * // init gain, spatial and temporal parameters:
91 * _photoreceptorsPrefilter->setCoefficientsTable(gain,temporalConstant, spatialConstant);
93 * // during program execution, call the filter for local luminance correction or low pass filtering for an input picture called "FrameBuffer":
94 * _photoreceptorsPrefilter->runFilter_LocalAdapdation(FrameBuffer);
95 * // or (Low pass first order filter)
96 * _photoreceptorsPrefilter->runFilter_LPfilter(FrameBuffer);
97 * // get output frame and its size:
98 * const unsigned int output_nbRows=_photoreceptorsPrefilter->getNBrows();
99 * const unsigned int output_nbColumns=_photoreceptorsPrefilter->getNBcolumns();
100 * const double *outputFrame=_photoreceptorsPrefilter->getOutput();
102 * // at the end of the program, destroy object:
103 * delete _photoreceptorsPrefilter;
105 * @author Alexandre BENOIT, benoit.alexandre.vision@gmail.com, LISTIC : www.listic.univ-savoie.fr, Gipsa-Lab, France: www.gipsa-lab.inpg.fr/
107 * synthesis of the work described in Alexandre BENOIT thesis: "Le systeme visuel humain au secours de la vision par ordinateur"
111 #include "templatebuffer.hpp"
113 //#define __BASIC_RETINA_ELEMENT_DEBUG
115 //using namespace std;
118 class BasicRetinaFilter
123 * constructor of the base bio-inspired toolbox, parameters are only linked to imae input size and number of filtering capabilities of the object
124 * @param NBrows: number of rows of the input image
125 * @param NBcolumns: number of columns of the input image
126 * @param parametersListSize: specifies the number of parameters set (each parameters set represents a specific low pass spatio-temporal filter)
127 * @param useProgressiveFilter: specifies if the filter has irreguar (progressive) filtering capabilities (this can be activated later using setProgressiveFilterConstants_xxx methods)
129 BasicRetinaFilter(const unsigned int NBrows, const unsigned int NBcolumns, const unsigned int parametersListSize=1, const bool useProgressiveFilter=false);
132 * standrad destructore
134 ~BasicRetinaFilter();
137 * function which clears the output buffer of the object
139 inline void clearOutputBuffer(){_filterOutput=0;};
142 * function which clears the secondary buffer of the object
144 inline void clearSecondaryBuffer(){_localBuffer=0;};
147 * function which clears the output and the secondary buffer of the object
149 inline void clearAllBuffers(){clearOutputBuffer();clearSecondaryBuffer();};
152 * resize basic retina filter object (resize all allocated buffers
153 * @param NBrows: the new height size
154 * @param NBcolumns: the new width size
156 void resize(const unsigned int NBrows, const unsigned int NBcolumns);
159 * forbiden method inherited from parent std::valarray
160 * prefer not to use this method since the filter matrix become vectors
162 void resize(const unsigned int){std::cerr<<"error, not accessible method"<<std::endl;};
165 * low pass filter call and run (models the homogeneous cells network at the retina level, for example horizontal cells or photoreceptors)
166 * @param inputFrame: the input image to be processed
167 * @param filterIndex: the offset which specifies the parameter set that should be used for the filtering
168 * @return the processed image, the output is reachable later by using function getOutput()
170 const std::valarray<float> &runFilter_LPfilter(const std::valarray<float> &inputFrame, const unsigned int filterIndex=0); // run the LP filter for a new frame input and save result in _filterOutput
173 * low pass filter call and run (models the homogeneous cells network at the retina level, for example horizontal cells or photoreceptors)
174 * @param inputFrame: the input image to be processed
175 * @param outputFrame: the output buffer in which the result is writed
176 * @param filterIndex: the offset which specifies the parameter set that should be used for the filtering
178 void runFilter_LPfilter(const std::valarray<float> &inputFrame, std::valarray<float> &outputFrame, const unsigned int filterIndex=0); // run LP filter on a specific output adress
181 * low pass filter call and run (models the homogeneous cells network at the retina level, for example horizontal cells or photoreceptors)
182 * @param inputOutputFrame: the input image to be processed on which the result is rewrited
183 * @param filterIndex: the offset which specifies the parameter set that should be used for the filtering
185 void runFilter_LPfilter_Autonomous(std::valarray<float> &inputOutputFrame, const unsigned int filterIndex=0);// run LP filter on the input data and rewrite it
188 * local luminance adaptation call and run (contrast enhancement property of the photoreceptors)
189 * @param inputOutputFrame: the input image to be processed
190 * @param localLuminance: an image which represents the local luminance of the inputFrame parameter, in general, it is its low pass spatial filtering
191 * @return the processed image, the output is reachable later by using function getOutput()
193 const std::valarray<float> &runFilter_LocalAdapdation(const std::valarray<float> &inputOutputFrame, const std::valarray<float> &localLuminance);// run local adaptation filter and save result in _filterOutput
196 * local luminance adaptation call and run (contrast enhancement property of the photoreceptors)
197 * @param inputFrame: the input image to be processed
198 * @param localLuminance: an image which represents the local luminance of the inputFrame parameter, in general, it is its low pass spatial filtering
199 * @param outputFrame: the output buffer in which the result is writed
201 void runFilter_LocalAdapdation(const std::valarray<float> &inputFrame, const std::valarray<float> &localLuminance, std::valarray<float> &outputFrame); // run local adaptation filter on a specific output adress
204 * local luminance adaptation call and run (contrast enhancement property of the photoreceptors)
205 * @param inputFrame: the input image to be processed
206 * @return the processed image, the output is reachable later by using function getOutput()
208 const std::valarray<float> &runFilter_LocalAdapdation_autonomous(const std::valarray<float> &inputFrame);// run local adaptation filter and save result in _filterOutput
211 * local luminance adaptation call and run (contrast enhancement property of the photoreceptors)
212 * @param inputFrame: the input image to be processed
213 * @param outputFrame: the output buffer in which the result is writen
215 void runFilter_LocalAdapdation_autonomous(const std::valarray<float> &inputFrame, std::valarray<float> &outputFrame); // run local adaptation filter on a specific output adress
218 * run low pass filtering with progressive parameters (models the retina log sampling of the photoreceptors and its low pass filtering effect consequence: more powerfull low pass filtering effect on the corners)
219 * @param inputFrame: the input image to be processed
220 * @param filterIndex: the index which specifies the parameter set that should be used for the filtering
221 * @return the processed image, the output is reachable later by using function getOutput() if outputFrame is NULL
223 inline void runProgressiveFilter(std::valarray<float> &inputFrame, const unsigned int filterIndex=0){_spatiotemporalLPfilter_Irregular(&inputFrame[0], filterIndex);};
226 * run low pass filtering with progressive parameters (models the retina log sampling of the photoreceptors and its low pass filtering effect consequence: more powerfull low pass filtering effect on the corners)
227 * @param inputFrame: the input image to be processed
228 * @param outputFrame: the output buffer in which the result is writen
229 * @param filterIndex: the index which specifies the parameter set that should be used for the filtering
231 inline void runProgressiveFilter(const std::valarray<float> &inputFrame,
232 std::valarray<float> &outputFrame,
233 const unsigned int filterIndex=0)
234 {_spatiotemporalLPfilter_Irregular(get_data(inputFrame), &outputFrame[0], filterIndex);};
237 * first order spatio-temporal low pass filter setup function
238 * @param beta: gain of the filter (generally set to zero)
239 * @param tau: time constant of the filter (unit is frame for video processing)
240 * @param k: spatial constant of the filter (unit is pixels)
241 * @param filterIndex: the index which specifies the parameter set that should be used for the filtering
243 void setLPfilterParameters(const float beta, const float tau, const float k, const unsigned int filterIndex=0); // change the parameters of the filter
246 * first order spatio-temporal low pass filter setup function
247 * @param beta: gain of the filter (generally set to zero)
248 * @param tau: time constant of the filter (unit is frame for video processing)
249 * @param alpha0: spatial constant of the filter (unit is pixels) on the border of the image
250 * @param filterIndex: the index which specifies the parameter set that should be used for the filtering
252 void setProgressiveFilterConstants_CentredAccuracy(const float beta, const float tau, const float alpha0, const unsigned int filterIndex=0);
255 * first order spatio-temporal low pass filter setup function
256 * @param beta: gain of the filter (generally set to zero)
257 * @param tau: time constant of the filter (unit is frame for video processing)
258 * @param alpha0: spatial constant of the filter (unit is pixels) on the border of the image
259 * @param accuracyMap an image (float format) which values range is between 0 and 1, where 0 means, apply no filtering and 1 means apply the filtering as specified in the parameters set, intermediate values allow to smooth variations of the filtering strenght
260 * @param filterIndex: the index which specifies the parameter set that should be used for the filtering
262 void setProgressiveFilterConstants_CustomAccuracy(const float beta, const float tau, const float alpha0, const std::valarray<float> &accuracyMap, const unsigned int filterIndex=0);
265 * local luminance adaptation setup, this function should be applied for normal local adaptation (not for tone mapping operation)
266 * @param v0: compression effect for the local luminance adaptation processing, set a value between 0.6 and 0.9 for best results, a high value yields to a high compression effect
267 * @param maxInputValue: the maximum amplitude value measured after local adaptation processing (c.f. function runFilter_LocalAdapdation & runFilter_LocalAdapdation_autonomous)
268 * @param meanLuminance: the a priori meann luminance of the input data (should be 128 for 8bits images but can vary greatly in case of High Dynamic Range Images (HDRI)
270 void setV0CompressionParameter(const float v0, const float maxInputValue, const float){ _v0=v0*maxInputValue; _localLuminanceFactor=v0; _localLuminanceAddon=maxInputValue*(1.0f-v0); _maxInputValue=maxInputValue;};
273 * update local luminance adaptation setup, initial maxInputValue is kept. This function should be applied for normal local adaptation (not for tone mapping operation)
274 * @param v0: compression effect for the local luminance adaptation processing, set a value between 0.6 and 0.9 for best results, a high value yields to a high compression effect
275 * @param meanLuminance: the a priori meann luminance of the input data (should be 128 for 8bits images but can vary greatly in case of High Dynamic Range Images (HDRI)
277 void setV0CompressionParameter(const float v0, const float meanLuminance){ this->setV0CompressionParameter(v0, _maxInputValue, meanLuminance);};
280 * local luminance adaptation setup, this function should be applied for normal local adaptation (not for tone mapping operation)
281 * @param v0: compression effect for the local luminance adaptation processing, set a value between 0.6 and 0.9 for best results, a high value yields to a high compression effect
283 void setV0CompressionParameter(const float v0){ _v0=v0*_maxInputValue; _localLuminanceFactor=v0; _localLuminanceAddon=_maxInputValue*(1.0f-v0);};
286 * local luminance adaptation setup, this function should be applied for local adaptation applied to tone mapping operation
287 * @param v0: compression effect for the local luminance adaptation processing, set a value between 0.6 and 0.9 for best results, a high value yields to a high compression effect
288 * @param maxInputValue: the maximum amplitude value measured after local adaptation processing (c.f. function runFilter_LocalAdapdation & runFilter_LocalAdapdation_autonomous)
289 * @param meanLuminance: the a priori meann luminance of the input data (should be 128 for 8bits images but can vary greatly in case of High Dynamic Range Images (HDRI)
291 void setV0CompressionParameterToneMapping(const float v0, const float maxInputValue, const float meanLuminance=128.0f){ _v0=v0*maxInputValue; _localLuminanceFactor=1.0f; _localLuminanceAddon=meanLuminance*_v0; _maxInputValue=maxInputValue;};
294 * update compression parameters while keeping v0 parameter value
295 * @param meanLuminance the input frame mean luminance
297 inline void updateCompressionParameter(const float meanLuminance){_localLuminanceFactor=1; _localLuminanceAddon=meanLuminance*_v0;};
300 * @return the v0 compression parameter used to compute the local adaptation
302 float getV0CompressionParameter(){ return _v0/_maxInputValue;};
305 * @return the output result of the object
307 inline const std::valarray<float> &getOutput() const {return _filterOutput;};
310 * @return number of rows of the filter
312 inline unsigned int getNBrows(){return _filterOutput.getNBrows();};
315 * @return number of columns of the filter
317 inline unsigned int getNBcolumns(){return _filterOutput.getNBcolumns();};
320 * @return number of pixels of the filter
322 inline unsigned int getNBpixels(){return _filterOutput.getNBpixels();};
325 * force filter output to be normalized between 0 and maxValue
326 * @param maxValue: the maximum output value that is required
328 inline void normalizeGrayOutput_0_maxOutputValue(const float maxValue){_filterOutput.normalizeGrayOutput_0_maxOutputValue(maxValue);};
331 * force filter output to be normalized around 0 and rescaled with a sigmoide effect (extrem values saturation)
332 * @param maxValue: the maximum output value that is required
334 inline void normalizeGrayOutputCentredSigmoide(){_filterOutput.normalizeGrayOutputCentredSigmoide();};
337 * force filter output to be normalized : data centering and std normalisation
338 * @param maxValue: the maximum output value that is required
340 inline void centerReductImageLuminance(){_filterOutput.centerReductImageLuminance();};
343 * @return the maximum input buffer value
345 inline float getMaxInputValue(){return this->_maxInputValue;};
348 * @return the maximum input buffer value
350 inline void setMaxInputValue(const float newMaxInputValue){this->_maxInputValue=newMaxInputValue;};
354 /////////////////////////
356 TemplateBuffer<float> _filterOutput; // primary buffer (contains processing outputs)
357 std::valarray<float> _localBuffer; // local secondary buffer
358 /////////////////////////
360 unsigned int _halfNBrows;
361 unsigned int _halfNBcolumns;
363 // parameters buffers
364 std::valarray <float>_filteringCoeficientsTable;
365 std::valarray <float>_progressiveSpatialConstant;// pointer to a local table containing local spatial constant (allocated with the object)
366 std::valarray <float>_progressiveGain;// pointer to a local table containing local spatial constant (allocated with the object)
368 // local adaptation filtering parameters
369 float _v0; //value used for local luminance adaptation function
370 float _maxInputValue;
371 float _meanInputValue;
372 float _localLuminanceFactor;
373 float _localLuminanceAddon;
375 // protected data related to standard low pass filters parameters
380 /////////////////////////
383 // Basic low pass spation temporal low pass filter used by each retina filters
384 void _spatiotemporalLPfilter(const float *inputFrame, float *LPfilterOutput, const unsigned int coefTableOffset=0);
385 float _squaringSpatiotemporalLPfilter(const float *inputFrame, float *outputFrame, const unsigned int filterIndex=0);
387 // LP filter with an irregular spatial filtering
389 // -> rewrites the input buffer
390 void _spatiotemporalLPfilter_Irregular(float *inputOutputFrame, const unsigned int filterIndex=0);
391 // writes the output on another buffer
392 void _spatiotemporalLPfilter_Irregular(const float *inputFrame, float *outputFrame, const unsigned int filterIndex=0);
393 // LP filter that squares the input and computes the output ONLY on the areas where the integrationAreas map are TRUE
394 void _localSquaringSpatioTemporalLPfilter(const float *inputFrame, float *LPfilterOutput, const unsigned int *integrationAreas, const unsigned int filterIndex=0);
396 // local luminance adaptation of the input in regard of localLuminance buffer
397 void _localLuminanceAdaptation(const float *inputFrame, const float *localLuminance, float *outputFrame);
398 // local luminance adaptation of the input in regard of localLuminance buffer, the input is rewrited and becomes the output
399 void _localLuminanceAdaptation(float *inputOutputFrame, const float *localLuminance);
400 // local adaptation applied on a range of values which can be positive and negative
401 void _localLuminanceAdaptationPosNegValues(const float *inputFrame, const float *localLuminance, float *outputFrame);
404 //////////////////////////////////////////////////////////////
405 // 1D directional filters used for the 2D low pass filtering
407 // 1D filters with image input
408 void _horizontalCausalFilter_addInput(const float *inputFrame, float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd);
409 // 1D filters with image input that is squared in the function
410 void _squaringHorizontalCausalFilter(const float *inputFrame, float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd);
411 // vertical anticausal filter that returns the mean value of its result
412 float _verticalAnticausalFilter_returnMeanValue(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd);
414 // most simple functions: only perform 1D filtering with output=input (no add on)
415 void _horizontalCausalFilter(float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd);
416 void _horizontalAnticausalFilter(float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd);
417 void _verticalCausalFilter(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd);
418 void _verticalAnticausalFilter(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd);
420 // perform 1D filtering with output with varrying spatial coefficient
421 void _horizontalCausalFilter_Irregular(float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd);
422 void _horizontalCausalFilter_Irregular_addInput(const float *inputFrame, float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd);
423 void _horizontalAnticausalFilter_Irregular(float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd);
424 void _verticalCausalFilter_Irregular(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd);
425 void _verticalAnticausalFilter_Irregular_multGain(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd);
428 // 1D filters in which the output is multiplied by _gain
429 void _verticalAnticausalFilter_multGain(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd); // this functions affects _gain at the output
430 void _horizontalAnticausalFilter_multGain(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd); // this functions affects _gain at the output
432 // LP filter on specific parts of the picture instead of all the image
433 // same functions (some of them) but take a binary flag to allow integration, false flag means, 0 at the output...
434 void _local_squaringHorizontalCausalFilter(const float *inputFrame, float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd, const unsigned int *integrationAreas);
435 void _local_horizontalAnticausalFilter(float *outputFrame, unsigned int IDrowStart, unsigned int IDrowEnd, const unsigned int *integrationAreas);
436 void _local_verticalCausalFilter(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd, const unsigned int *integrationAreas);
437 void _local_verticalAnticausalFilter_multGain(float *outputFrame, unsigned int IDcolumnStart, unsigned int IDcolumnEnd, const unsigned int *integrationAreas); // this functions affects _gain at the output