PDF Links
PubReader
ePub Link1. Introduction
Dynamic textures are videos that exhibit a stationary property with respect to time (i.e., they have the patterns that repeat themselves over a large number of frames). Examples of such dynamic textures are flame, pond, grass, etc. Doretto et al. [1] have proposed a dynamic texture model that exploits the temporal property of a dynamic texture as a linear dynamic system driven by an independent and identical process. The proposed model is especially attractive as it provides a mathematical model that can be used for the analysis and synthesis of a dynamic texture. This model is demonstrated with several examples in [1], and is capable of effectively representing dynamic textures with the model of a low dimensional space. This approach exploits the temporal correlation among the pixels, but it fails to exploit the spatial and chromatic correlation among the pixels. The model proposed by Abraham et al. [2] deals with the spatial and temporal correlation among the dynamic texture frames. The drawbacks of this model have higher computational and storage complexities. The models by Doretto et al. [1] and Abraham et al. [2] do not consider chromatic redundancy within the dynamic texture frames.
In this paper, a technique that exploits the spatial, temporal, and chromatic correlation of the different dynamic texture frame sequences is proposed. Instead of considering the pixel intensity for the linear dynamic system, a discrete wavelet transform is applied to the individual dynamic texture frames. This results in frames having wavelet coefficients in four different frequency bands, namely LL, LH, HL, and HH. The main idea of this paper is to apply content based threshold filtering based on Set Partitioning in a Hierarchical Tree (SPIHT) [3,4], so as to have more significant information in fewer parameters for singular value decomposition (SVD). Also, instead of arranging the whole frame into column vectors for SVD [1], the different bands of dynamic texture frames are arranged into different column matrices for SVD.
The paper is organized as follows: In Section 2, related work is given, which comprised of different papers that lists existing algorithms advantages and disadvantages. In Section 3, a brief overview of the dynamic texture analysis and synthesis model proposed by Doretto et al. [1] is given. In Section 4, our proposed method of incorporating wavelet coefficients and SPIHT is described. In Section 5, experiments comparing the proposed approach against the available approaches are presented. Finally, the conclusions are given in Section 6.
2. Related Work
A lot of work has been done on the dynamic texture synthesis and a number of methods have been proposed. Each method has its advantages and disadvantages that consider different measuring factors, such as the memory required to store the synthesis parameter, the visual quality of the synthesized dynamic texture, time complexity, etc. The understanding of different scenarios requires the study of various synthesis algorithms along with transform algorithms like Fourier, discrete cosine transform, and Wavelet. The proposed model requires a study of the various basic concepts that include different color formats, SPIHT, and conversions among them.
Stefano Soatto et al. [5] presented the characterization of a dynamic texture that poses the problem of analysis and synthesis on a firm analytical footing. The borrowed tool from system identification captures the essence of a dynamic texture. This can be done by using models. The proposed model has the predictive power to extrapolate the synthesis sequence to infinity. This paper does not deal with the spatial correlation among the pixels.
Filip et al. [6] proposed a novel hybrid method for colour dynamic texture modelling. The method is based on Eigen analysis of dynamic texture images and the subsequent pre-processing and modelling of temporal interpolation. Eigen coefficients use a causal auto-regressive model. This model demonstrates good performance for most of the tested dynamic textures, which depends mainly on the properties of the original sequence. This proposed algorithm does not exploit the spatial, temporal, and chromatic correlation among the pixels.
The model by Costantini et al. [7] is flexible and natural, as it causes dimension reduction in the spatial, temporal, and chromatic domain. The model is well suited to dynamic texture synthesis on devices limited by computational power and memory. Here, the analysis part is more expensive as compared to the existing algorithms.
The algorithm by Li et al. [8] is general and automatic, and it works well on different types of textures. Experimental results demonstrate that the approach can reconstruct dynamic texture sequences with promising visual quality and fewer coefficients. The scheme is incapable of searching for a global optimal solution, which would improve the visual synthesis quality.
3. Dynamic Texture Modelling System
It has been shown that dynamic textures can be represented as an output of the linear dynamic system [1]. These patterns can be easily tracked by a linear dynamic system. The approach described by [1] uses raw image vectors for the synthesis of a dynamic texture. The modeling of a dynamic texture includes analysis and synthesis. An analysis consists of; 1) finding an appropriate subspace that represents the trajectory motion, and 2) identifying the trajectory with the help of the dynamical system theory. The appropriate subspace can be found by retaining the first n singular values of the decomposition.
Assume that C = Un and X = SnVnT, then a generic column vector is given by:
where xk is the kth column vector of matrix X and e is a residual error. The equation above can be written as:
where ci with i=1,…,n indicates the ith column vector of matrix C.
Identification of system dynamics starts from matrix X∈Rn x τ, which is the time evolution of a dynamic texture. The evolution of xk is done using the MAR model.
where A∈Rn x n and B∈Rn x nv are the matrix computed from matrix X and Vk is the Gaussian random noise. Finally, a dynamic texture can be synthesized using the following equations:
4. Dynamic Texture Model Along with Wavelet Coefficients and Spiht Based Thresholding
The approach described above uses raw image vectors for the analysis and synthesis of a dynamic texture. As mentioned earlier it does not deal with the spatial correlation among the pixels and thus, results into a large order system. In order to have a minimum order system a discrete wavelet transform is applied before the analysis and synthesis of a linear dynamic texture are carried out. The discrete wavelet transform results in different frequency coefficients. Most of the frequency coefficients in LH, HL, and HH frequency bands are not useful for the overall quality of a synthesized dynamic texture. So, the wavelet coefficients, which significantly contribute in the quality of synthesized frames, are considered in this model.
Roberto Costantini et al. [7] synthesized a dynamic texture using different color spaces instead of RGB color space and achieved a similar synthesis performance by using half of the model coefficients and less computational power. The proposed model considers all three redundancies (i.e., spatial, temporal, and chromatic). The proposed approach can be interpreted as a four stage modelling process, which is as follow: 1) chromatic correlation is exploited using different color spaces; 2) the spatial correlation of the pixels is exploited using a discrete wavelet transform; 3) content based threshold filtering based on SPIHT; and 4) time correlation among the different dynamic texture frames is exploited by learning about time evolution of an operator using the model proposed by [1]. Fig. 1 represents the overall flow of the proposed model. The idea of the proposed model is described in more detail next.
In the proposed dynamic texture analysis and synthesis model, frames are extracted from a dynamic texture. Color space conversion is applied to these extracted frames [8] where RGB color components are converted to YCbCr color components [9] and where Y is luminance and Cb and Cr are chromatic blue and chromatic red color components, respectively. These color spaces help to analyze the luminance and chrominance components separately. A discrete wavelet transform is then applied to YCbCr color space frames and it leads to four different bands, namely LL, LH, HL, and HH [10]. The LL band contains low frequency or significant components and HH contains high frequency or insignificant components.
According to the SPIHT concept [11], firstly wavelet transform is applied then threshold is calculated based on the transform data. Generally, a set of fixed threshold values has an impact on the efficiency of model coefficients. The set of minimum selected threshold values leads to a lot of insignificant data coefficients. On the other hand, a set of maximum selected threshold values leads to a loss of most of the significant data coefficients [12]. So, the best threshold must be data dependent. In the SPIHT coding technique for threshold value section, the maximum value is selected from wavelet transform coefficients of an image. Then ‘N’ (i.e., number of passes) is obtained by logarithms of the maximum selected transformed value with base 2. Mathematically, it is calculated as:
where Cmax is the maximum selected transform coefficient and N is the number of passes.
The threshold for each pass is calculated as:
For each successive pass, the value of N is decremented by 1 and the threshold value is calculated as shown above. Here, the number of passes is dependent on the available bandwidth and quality evaluation parameters.
In the proposed model, the maximum value is selected from all of the dynamic texture frames. To have uniformity in threshold selection, the minimum value is selected from all of the maximum selected values. The number of passes and threshold for each pass is calculated as shown above. The number of passes should be incremented until better quality is achieved.
The human visual system is more sensitive to variations in brightness (i.e., luminance) than color (i.e., chrominance) [13]. Therefore, these color components (i.e., Cb and Cr ) are decimated by a factor of 2, while retaining the full luminance resolution. This helps in achieving a more compact model for the dynamic texture.
Doretto et al. [1] and Soatto et al. [5] have shown that a dynamic texture can be expressed as an output of the linear dynamic system. In the proposed system discrete wavelet transform based bands are then given as an input to the linear dynamic system (i.e., SVD). The same band of all dynamic texture frames are temporally correlated with each other. So, instead of applying SVD on dynamic texture frames by arranging all of the frequency bands of a frame into single column vectors, it is applied onto different frame bands independently. Fig. 2 shows an insight working of the proposed model. This proposed technique helps to attain more flexibility with component selection.
The LL band contains most of the significant information, while the LH, HL, and HH bands contain less significant information. Also, the SVD arranges information in descending order (i.e., from the most significant information to the least significant information). So, selecting a fewer number of components from the LH, HL, and HH bands of the discrete wavelet transform and a greater number of components from the LL band of the wavelet transform provides a more compact representation of a dynamic texture with better visual quality. Skipping more components from the LH, HL, and HH bands of the discrete wavelet transform does not affect the visual quality, as these are less important coefficients.
The compact model after LDS can be stored or sent to the receiver for synthesis. Mathematically, synthesis can be done as shown in Eqs. (5) and (6). Finally, the up sample of the Cb and Cr color components should be performed by adding a factor of 2 and applying an inverse discrete wavelet transform to get the synthesized frames. The proposed model was also implemented using the YIQ color model and results in the same peak signal-to-noise ratio (PSNR) and model size with some variations.
In the proposed system, threshold filtering is based on the SPIHT and LDS parameter learning blocks, which require more computational time. As such, these two blocks were implemented on the graphics processing unit (GPU).
The steps for the proposed algorithm are listed below.
Extract frames from dynamic texture.
Apply the YCbCr or YIQ color encoding technique.
Apply the discrete wavelet transform.
Select the maximum intensity (i.e., the Cmax value) from each dynamic texture frame.
Take N= [log2 Cmax] of all intensity values and select the minimum of all the values.
The threshold value is calculated as: Threshold=2N
Apply filtering based on the calculated threshold value.
Down sample the Cb and Cr color components.
Apply the SVD on different bands of the wavelet transform individually.
The number of components selected is based on different bands of the wavelet transform.
Different blocks are synthesized using analyzed parameters.
Up sample the Cb and Cr color components.
Finally, apply an inverse discrete wavelet transform to get the reconstructed dynamic texture frames.
5. Results
The synthesis of a dynamic texture introduces some distortion in the reconstructed frames. Therefore, the quality evaluation parameter is an important issue in image processing. Several experiments have been performed using the proposed dynamic texture analysis and synthesis model. The execution of the proposed model was also done on GPU [14, 15]. GPU is designed to take advantage of parallel computation (i.e., the same program is executed on many data elements in parallel). This can be achieved with the help of thousands of GPU cores. As the same program is executed for each data element, there are fewer requirement for flow control. As the same program is executed on many data elements and has high arithmetic intensity, the memory access time is hidden in calculation instead of inside a big data cache.
CPU has very few cores and it is optimized for serial processing, while GPU has thousands of smaller and efficient cores that are designed for parallel processing. CPU + GPU is a powerful combination because a serial portion of the code is run on CPU, while a parallel portion of the code, which is independent of each other, runs on GPU [16]. In the proposed model, threshold filtering based on SPIHT and a dimension reduction technique is implemented on GPU to reduce time complexity.
Dynamic texture sequences, which were used for testing purposes, were taken from the MIT temporal texture database and are available at [17]. The raw sequences include the flowing water sequence (352 pixels × 288 pixels × 251 frames), steady sequence (352 pixels × 288 pixels × 251 frames), flush water sequence (352 pixels × 288 pixels × 251 frames), flame sequence (320 pixels × 240 pixels × 89 frames), grass sequence (224 pixels × 144 pixels × 100 frames), tides sequence (352 pixels × 288 pixels × 250 frames), river water sequence (352 pixels × 288 pixels × 251 frames), canal water sequence (352 pixels × 288 pixels × 251 frames), shower sequence (352 pixels × 288 pixels × 251 frames), bird sequence (320 pixels × 240 pixels × 135 frames), tex1 sequence (352 pixels × 288 pixels × 71 frames), and the pond sequence (368 pixels × 240 pixels × 150 frames). The available methods were compared with the proposed method in different aspects. Table 1 represents the results obtained from the proposed model on different dynamic textures.
Table 2 compares the proposed method with the LDS [1], and the Fourier descriptor method [2] for the flame sequence. While setting the same PSNR for all of the models, the number of model coefficients for the synthesis of a dynamic texture using the proposed model was significantly less than the LDS and Fourier descriptor model.
Fig. 3 represents the model coefficients of a reconstructed flame sequence against an average PSNR. Fig. 3 shows the performance comparison between the proposed method, the LDS method [1], and the Fourier descriptor method [2] by using same average PSNR (dB) for all models. In Fig. 3, model coefficients and average PSNR (dB) of different models are taken from Table 2.
Table 3 compares the model size of the proposed method with LDS+Wavelet, and higher-order SVD (HOSVD) [7] by setting the same average PSNR (dB) for a different dynamic texture sequence.
Table 4 shows the time required to synthesize a dynamic texture on CPU and GPU. The execution of a dynamic texture on CPU requires more time because of LDS and SPIHT based threshold filtering. So, these two blocks are implemented on the GPU by parallel processing and thus, require less time as compared to CPU. Fig. 4 shows a graph representing the time required for CPU and GPU for a different dynamic texture. Fig. 5 shows the frames sampled (frame no 1, 17, 49, and 88) from an original flame sequence and from different models [1, 2, 7]. The proposed model provides a better visual quality of reconstructed frames. The proposed model was implemented on the CPU and GPU configurations listed below.
6. Conclusion
Dynamic texture analysis and the synthesis model using SVD along with wavelet transform and SPIHT has been proposed. The proposed scheme uses wavelet coefficients of frames in the sequence, instead of the frames themselves, to learn the dynamics of the texture. For the efficient use of wavelet transform, SPIHT based threshold filtering is applied to transformed wavelet coefficients. In doing so, a higher compression ratio is obtained than is available in the linear dynamic system, the linear dynamic system along with wavelet transform, and the HOSVD. SPIHT based threshold filtering helps to get a more compact representation of a dynamic texture. The results obtained from LDS, LDS along with Fourier transform, LDS along with wavelet transform, and LDS along with wavelet transform, and SPIHT in combination with their comparison is shown in Section 5. The experimental results demonstrate that the proposed system describes the dynamic texture in fewer parameters and the synthesized dynamic texture sequence visually resembles the original one. YCbCr, and YIQ color coding deals with the chromatic correlation among the pixels and thus, helps in obtaining a higher compact representation of a dynamic texture.
