An Efﬁcient Satellite Image Super Resolution Technique for Shift-Variant Images using Improved New Edge Directed Interpolation

Super resolution is a method that reconstructs a higher resolution image from single captured image or set of captured low resolution images. Super resolution imaging is used for several image processing applications like medical imaging, earth observation systems and surveillance systems. Image interpolation is one of conventional methods used to enhance resolution of image. Basic linear interpolation methods like bilinear, bicubic give blurred image as a result. Non-linear interpolation methods like New Edge Directed Interpolation (NEDI), Curvature based interpolation, neural network based interpolation enhance image but has limitations like several artifacts. In this paper, a novel innovative approach is proposed in which using dual tree complex wavelet transform (DT-CWT), low and high frequency sub bands are generated. High frequency sub band images are interpolated using improved NEDI which is NEDI with circular window and dynamic window. Improved NEDI (INEDI) algorithm proposed in the paper gives better results on high frequency components which lead to high resolution image without artifacts. Inverse DT-CWT is applied on interpolated sub bands to reconstruct high resolution image. Registration is applied on both images and shift adaptable bilinear interpolation is applied which reconstructs image into 4 interpolation factor. The proposed approach is veriﬁed for different interpolation factors and for different satellite images. The accuracy of proposed approach is veriﬁed by several contrast features. The algorithm proposed in this paper outperforms in comparison to state of the art algorithms.


Introduction
In various fields like medical imaging, remote sensing, recognition and computer graphics, high resolution image is desirable. There is high requirement of a high resolution image that provides better visualization and additional information details. In remote sensing, a higher resolution image is advantageous to attain better classification of areas [1,2,3,45,5]. High resolution image is helpful in medical imaging to make correct diagnosis and treatment [6,7,8,9,10]. In video observation system, high resolution videos are useful in identifying object or person of concern [11,12,13,14].
One major way for acquiring high resolution image is to ameliorate image acquisition device by increasing pixel density of the sensor. There is a constraint in decreasing pixel size of sensor. The captured image quality will be naturally reduced as the pixel size of sensor becomes too small [15]. As pixel size of sensor reduces, the power of signal also decreases. This fact affects the quality of image. Moreover,a higher value is needed to improve chip size. On base of above stated, the Super Resolution imaging turns out to be good option. Super Resolution imaging research has emerged really fast after initially described by Tsai and Huang [16] in 1984.
Image processing involves many images with lost detail due to a situation like improper adjustment of cameras, lighting defect or camera with low sensor cell. This type of lost detail of image can be retrieved using super resolution method. Image Super Resolution pertains to image processing algorithms that generate high quality images from group of low quality input images or from a single input image. A framework of super resolution imaging is shown in Figure 1.
To understand the super resolution imaging, some basic concepts are needed to be clarified. First, resolution of an image is different from its physical size. Second, term resolution is used in context of spatial resolution, not the temporal resolution of image. In different applications like medical imaging, remote sensing, object tracking [17] a small error can generate major problems. The motivation is to get high resolution image to overcome the situation. Our method allows the use of two low resolution image with small shift for generating improved resolution image. The ambition of super resolution is to generate image with more details from group of low resolution images. In other words, the main aim of super resolution techniques is to generate image with more clear content from low resolution image instead of obtaining a bigger size of image.
The paper is arranged as follows: Section 2 contains brief overview of different Super Resolution methods. Section 3 describes basic concepts which are used in proposed approach. Section 4 describes proposed approach. Section 5 contains results and section 6 is conclusion on this study followed by references.

Frequency Domain based Super Resolution
Frequency domain based super resolution method was proposed by Tsai and Huang [16], where their main focus was super resolution computation for noise free low resolution images. Initially, the input images are transformed into Discrete Fourier Transform and then integrated as specified by relationship between DFT coefficients of mentioned low resolution images and coefficients of high resolution image [16,18]. High resolution image is received by transforming integrated data into spatial domain. Rhee and Kang [19] used Discrete Cosine Transform to execute rapid image deconvolution for super resolution image computation.
The frequency domain techniques have a number of advantages. First, we can easily enhance details of the image by generalizing high frequency details available in low resolution image. Second, frequency domain techniques have low computational complexity.
Ji and Fermuller [20,21] proposed strong wavelet based super resolution technique to handle error caused in registration computation and blur recognition computation. Ei-Khamy et al. [22,23] proposed that different low resolution images are first registered in wavelet domain, then registered low resolution wavelet coefficients are fused to get single image and then interpolation is performed to obtain high resolution image. Chappalli and Bose [24] developed a super resolution technique with noise filtering by merging denoising stage into conventional wavelet approach.

Interpolation based Super Resolution
The Interpolation based super resolution approach generates high resolution image by mapping every obtained low resolution image to single image, then combine each and every information available from each image and finally deblurs image. Super Resolution problem cannot be handled well by single image interpolation because it cannot create the high frequency components which were missed throughout image acquisition procedure. The standard of interpolated image produced by executing single input image interpolation algorithm is assigned with the intensity value of nearest pixel value. The Interpolation based super resolution techniques generally consist of three steps: 1) registration of low resolution input images, 2) interpolation of low resolution images for producing high resolution image and 3) deblur reconstructed high resolution image generated in step (2) [25,26,27]. The interpolation performs important role in this approach. The traditional interpolation methods are nearest neighbor interpolation, bilinear interpolation and bicubic interpolation. In nearest neighbor algorithm, unknown pixel value is allocated with intensity value of nearest pixel. In bilinear interpolation, unknown pixel value is estimated by the weighted average of the neighboring four pixels.
Patti and Tekalp [27] developed an algorithm that produces high resolution image which is reconcilable with details arising from captured low resolution images and previous image prototype. Zhang and Wu [28] proposed a method where kriging interpolation is used based on intensity distance and geometry of pixel. High resolution image is estimated using intensity distance and local window surrounding every unknown pixel. Shi and Shan [29] proposed new interpolation approach that generates high resolution image by interpolating pixel position according to directional variations of images.

Reconstruction based Super Resolution
Super resolution reconstruction is an ill-posed problem, many reconstruction based algorithms are proposed for addressing this issue. Reconstruction based super resolution techniques are classified into:1) Iterative back projection [30,33,34] and 2) regularization method [35,36]. Irani and Peleg [30] proposed super resolution reconstruction approach using iterative back-projection, where high resolution image is estimated by iteratively protruding error between feigned low resolution image via imaging blur and observed low resolution image. The equation for iterative back projection method to evaluate high resolution image is: Where X n k denotes estimated high resolution image of k th image after n iteration, Y n k represents the simulated degraded low resolution image of after n iteration, ? BP is the back-projecting operator.
Zhang [31] developed an algorithm which combines multiple shifted holographic images to increase resolution of digital hologram. Many regularization based methods have been invented to solve issue of super resolution. Regularization strategy is used to incorporate some anterior knowledge about desired high resolution image in regularization methods [35,36]. Applying regularization methods, desired high resolution image can be estimated using minimization function expressed as: Where ∑ k ∥ykobs − OkDBM kz∥2 is data fidelity term, r(Z) indicates regularization term and λ 2 represents regularization parameter.

Example based Super Resolution
In Example based single image super resolution, by employing dictionary of patch correspondences, high resolution image is estimated. The connection between high resolution image patch and its low resolution patch is specified by dictionary. Internal similarities or the set of external training images are used to build patch. Such algorithms made up of two steps: a training step and Super Resolution step.
In training step, low resolution image is segmented within the coinciding patches. Then high resolution image is evaluated using patch correspondences between low resolution patches and high resolution patches. In super resolution step, final high resolution image is built by reassembling all estimated high resolution patches. Example based methods are further classified into Learning based methods [37,38] and regression based methods [39,40]. In learning based methods, high frequency data of low resolution image is estimated by acquiring high frequency data from specified training samples depend on internal characteristics of low resolution image. Hertzmann [37] put forward a method to generate high frequency information for captured low resolution image from a training image database, as shown in 4. In regression based methods, regression function is used to learn connection between low resolution patches and high resolution patches. The training patch pairs for regression are chosen based on similarity between low resolution patch and test patch. High resolution image is constructed by integrating all estimated high resolution patches.

DTCWT
The Dual Tree Complex Wavelet Transform (DT-CWT) is an extension of discrete wavelet transform which has two main additional properties: 1) shift invariant and 2) good directional selectivity in multidimension. It produces complex coefficients by applying dual tree of wavelet filters to acquire real and imaginary parts. The DT-CWT gives better directional selectivity as it provides six directional sub bands at each scale oriented at 15o, 45o and 75o. The ordinary Discrete Wavelet Transform is shift variant due to destruction process in transform. A minor shift in input image can end in extremely different set of wavelet coefficients. The DT-CWT transformation of 2-D signal f(x) in form of wavelet function ψ(n) and scaling function ϕ(n) is as follow: Where Z is the set of natural numbers, j and l refer to the index of shifts and dilations respectively, ϕ j0,l , is the scaling coefficient and S j0,l , is the complex wavelet coefficient with And Where the superscripts r and i denote the real and imaginary parts, respectively [41].

Principle of improved NEDI Technique
There are some artifacts in NEDI such as interpolated pixel values change with the global brightness or the use of larger window improves the conditioning of the C T C matrix but produces blurred images. Such problems are solved in improved NEDI. Improved NEDI is an extension of NEDI. The main aim of improved NEDI is to overcome the problems occurred in NEDI and improve interpolation accuracy. The squared window used by NEDI introduces directional artifacts and makes algorithm non-isotropic. This has been solved by INEDI using circular window. NEDI uses bilinear interpolation while INEDI uses bicubic interpolation. To account for various frequency regions, INEDI can dynamically increase the window radius from a minimum to a maximum value while the normalized residual of the least squares fit decreases. INEDI tries to exclude from the circular window all pixels that do not belong to the local edge using a sort of region growing method that excludes all uniform areas and pixels that are not connected to the edge.

Proposed Approach
High frequency components are mainly damaged after applying interpolation since interpolation caused smoothing on high frequency components. For preserving high frequency components of image, DT-CWT has been applied.
Let us consider the αH * αW high resolution image X H and two sub pixel shifted H * W low resolution images X L1 and X L2 . The objective of resolution enhancement is to estimate the high resolution image X H from two sub pixel shifted low resolution images X L1 and X L2 . DT-CWT decomposes the low resolution images into different frequency sub bands. One level DT-CWT decomposition of H X W low resolution image gives two low frequency sub bands and six high frequency sub bands of size H/2 * W/2 as a result. Low resolution image is obtained by applying low pass filter to high resolution image. Low frequency sub-band images are the low resolution of the input image which consists less information compared to input image so input image is used instead of low frequency sub bands. In proposed interpolation technique, INEDI is applied on high frequency sub-bands for factor α/2 and input image is interpolated with factor α/4. After interpolating high frequency sub-bands images, the ultimate high resolution image is constructed by applying inverse DT-CWT on these interpolated sub bands and interpolated input image. Using this approach, αH/2 * αW/2 interpolated images X H1 and X H2 are generated from low resolution images X 11 and X 12 . Feature based image registration algorithm is applied on X H1 and X H2 to determine shift between them. First, pixels of X H1 and X H2 images are mapped to high resolution grid according to the sub pixel shift between them. After mapping of pixels, unknown pixel values are computed by shift adaptable bilinear interpolation based on the shift between input images. The output H * W high resolution image will hold keen edges than edges in interpolated image received by interpolation of low resolution image directly. This occurs due to the verity that interpolation of high frequency sub-band images will conserve more high frequency components after the interpolation of the respective sub bands separately than interpolating low resolution image directly. In summary, the proposed interpolation technique interpolates high frequency sub band images obtained by DT-CWT. Inverse DT-CWT is applied on interpolated high frequency sub bands and interpolated input image to get high resolution image. Two sub pixel shifted low resolution images are interpolated by this process. These interpolated images are registered by feature based image registration. The final high resolution image is computed using shift adaptable bilinear interpolation on previously interpolated images.

Experimental Results
The results shown in this section are performed in MATLAB 2014b. In this paper, enhancement of image resolution of satellite image is shown with interpolation factor two and four. The effectiveness of proposed approach is proved by comparing it with different methods like bilinear, bicubic, NEDI, DWT-RE [42], DT-CWT-RE [43] , DT-CWT-NLM-RE [44] for different satellite images in 7. In this paper, feature based image registration algorithm is used to find the shift between two shift variant input images. Different contrast measuring parameters like Entropy, gradient energy, laplacian energy and Brenners measure are used for performance analysis of different algorithms. If any three out of these four has higher value then it shows that the technique gives better result.
Brenners measure: A Brenner's measure is image quality measurement algorithm based on the second difference of the image gray-levels. The mathematical function to calculate Brenner's result for the image I can be described as [45].
The Brenners measure indicates quality characteristics of an image. Higher value of ϕ indicates higher resolution. Entropy: The entropy of image is calculated using the histogram of the image. The entropy of Image I is defined as [45].
Where P k is the relative frequency of the kth gray-level. Like Brenners measure, higher the entropy indicates greater the quality enhancement of the image.
Energy of Gradient: The energy of gradient is image quality measure which can be calculated by the sum of squares of the first derivative in the x and y directions. The mathematical representation of energy of gradient of Image I is written as [45].
Laplacian energy: The laplacian energy can be obtained using the second derivative of the image. The laplacian energy can be calculated as [45].
Where △I is image laplacian obtained by convolving image I with laplacian mask.  Figure 8 represents the result images of different methods for interpolation factor two. Figure 9 represents the result images of different shift adaptable methods for interpolation factor two. Figure 10 shows output images of 626 AN EFFICIENT SATELLITE IMAGE SUPER RESOLUTION TECHNIQUE FOR different approaches for interpolation factor four. Table 1, Table 2, Table 3 show comparison of different contrast measures for images in Figure 8, Figure 9 and Figure 10 respectively. The comparison of different contrast measures for image dataset 2 is shown in Table 4. It can be observed from Figure 8 and Table 1 that DT-CWT plus INEDI gives better results for single images. It can be observed from Figure Figure 9 and Table 2 that shift adaptable bilinear interpolation gives better result for two sub pixel shifted images. It can be observed from Figure 10, Table  3 and Table 4 that proposed approach is better in terms of both visualization and quality measure parameters compared to existing methods.

Conclusion
In this paper, a novel innovative approach is proposed to enhance resolution of image. In this approach, using DT-CWT, low and high frequency sub-bands are generated. High frequency sub band images are interpolated with improved NEDI which is NEDI with circular window and dynamic window size as it gives better results on high frequency components. Then inverse DT-CWT is applied on interpolated sub bands and interpolated input image to reconstruct high resolution image. Two sub pixel shifted low resolution images are individually interpolated using DT-CWT and INEDI method. The output images are registered and then shift adaptable bilinear interpolation is applied on both output images to generate high resolution image. The proposed approach is verified for different interpolation factors and for different satellite images. The accuracy of proposed approach is verified by several contrast features. The algorithm proposed in this paper outperforms in comparison to state of the art algorithms.