雨雪图像清晰化处理

【实例简介】
频域中雨雪图像的清晰化处理。本文为CMU大牛的文章（ Analysis of Rain and Snow in Frequency Space）。欢迎也做这个方向的朋友们和我联系相互探讨。
Int J Comput Vis Once detected we are then able to either decrease the or snowflakes are less important than in-focus ones, because amount of rain and snow by subtraction, or increase it by they have a milder effect on images. (The appearance model sampling and cloning. Other researchers have also devel- in Sect. 2.2 implicitly handles slightly out-of-focus parti- oped methods for rain and snow synthesis. Rain can be gen- cles. erated via various approximate methods (langer and zhang The lengths of streaks depends on how fast the particles 2003; Langer et al. 2004; Reeves 1983; Starik and Werman are falling and how far they are from the camera. Because 2003; Tatarchuk and Isidoro 2006), but physically accurate they are so small, wind resistance is a major factor, and their rain synthesis(Garg and Nayar 2006)involves accurately terminal velocities depend on their sizes For common alti- modeling how a raindrops deforms and refracts light as it tudes and temperatures, a raindrop's speed s can be approx falls. Well-designed rain textures combined with a particle imated by a polynomial in its diameter a(Foote and duToit system can be used to create realistic scenes(Tariq 2007) 1969) thesis is that when the scene is uniformly illuminated. no s(a)=-02+5.0a-09a2+0.1a3 additional scene analysis is required; the streaks are already finding the speed of snow flakes is more difficult( Magono correctly formed and illuminated. Instead of using separat tools to remove and to render rain and snow, we present a and Nakamura 1965; Bohm 1989), because they have more framework that does both complex shapes. But since our detection algorithm uses a range of streak sizes it is not necessary to obtain exact bounds on individual snow hakes as a result snowflakes 2 Image-Space Analysis can be assumed to fall half as fast as raindrops of similar size. If the ratio between size and speed is approximately A frame from a movie m acquired during a storm can be correct then the streaks can still be detected decomposed into two components: a clear image c and a If the camera and particle are moving at constant veloc rain/snow image r. Generally, a background scene point is ties and the particle stays at a uniform distance, then it will occluded by raindrops or snowFlakes for only a short time, be imaged as a straight streak a falling particle imaged over therefore we can approximate their effect as being purely a cameras exposure time e creates a streak of length /: additive For location(x, y) at time t, we have m(x,y,1)≈c(x,y,1)+r(x,y,1) (1)l(a,3x)=(a+s(a)e In this paper, we develop an algorithm to find r, based 2.2 The Appearance of a Rain or Snow Streak on the overall appearance and statistical properties of rain and snow. Although it is sometimes possible to create clear videos by increasing the camera aperture and exposure time The appearance of a raindrop or snowflake depends on the particle's shape and reflectance, and the lighting in the en (Garg and nayar 2005), this paper focuses on cases where vironment. As shown in Fig. 2(a), under common lighting this is not possible, such as when the entire scene needs to be in focus or when there are fast-moving objects that should conditions, a falling drop will produce a horizontally sym- metric streak not be blurred. When in focus and not blurred by a long A completely accurate prediction of a streaks coloring exposure time, rain and snow appear in images as bright streaks. We begin the analysis in image-space by creating would require extensive physical modeling, as was done an appearance model of a streak to render rain in Garg and Nayar(2006), and is shown in Fig 2(c)and(d). But in most cases, the breadth is only a few 2. 1 The Shape of a rain or snow Streak pixels, and it is not necessary to form an exact model of light reflecting off a snowflake or determine the exact distorted Raindrops and snow flakes can have complex shapes. How- image that will appear in a tiny drop garg and Nayar 2004 ever in a typical video sequence, their shapes are not promi- van de hulst 1957). Instead, we use a simple, analytical nently visible, therefore we ignore any variation in their model that is fast to compute and well-behaved in frequency shape and consider them to be symmetric particles. At a space given instant in time, a camera with focal lengthf images an To begin, the image of a raindrop or snowflake is approx in-focus particle of diameter a at a distance from the camera imated as a gaussian, which appears similar to a slightly z as an image with breadth out-of-focus sphere. As the particle moves in space, the image it creates is a linear motion blurred version of the b(a, z) (2) original Gaussian. If the sphere is larger or closer to the camera, the Gaussian will have a higher variance. If it is If a particle is not in focus, then its image will be broader. falling faster, then it will be blurred into a longer streak. The For the purposes of image analysis, out of focus raindrops equation of a blurred Gaussian g, centered at image loca J Comput V created by all N visible drops ∑g(x 6d,风d) (6) For a given time t, each of ad, zd, ed, and ud are drawn from different distributions, Drops are equally likely to ap- pear at any location in space, so the x and y positions in ud are drawn from uniform distributions. Because a greater volume is imaged further from the camera, more drops are liable to be imaged at greater depths, so zd is drawn from a simple quadratic distribution. Streak orientation has a mean orientation 8d, with a slight variance. The most problematic Fig. 2 Raindrops and snowflakes create streaks of different appear- parameter is the drop size ad. Fortunately, many researchers ances, depending on factors such as the environmental illumina- in the atmospheric sciences have studied the expected num tion, their depth from the camera, and how much they are in focus. ber of each size of raindrop or snowflake, and we draw upon (a)A streak from a real water drop under illumination from a broad source.(b) The streak's appcarance can be modeled by a blurred their conclusions Gaussian(5).(c)A rendered streak from Garg and Nayar(2006) with It is well known that in a single storm, there will be par- broad environmental lighting. If the lighting is from a point source, ticles of various sizes. Size distributions are commonly used then the streak would appear as in the point lighting example ( d), which for raindrops (Marshall and Palmer 1948; Ulbrich 1983 is also from Garg and Nayar(2006). In this paper, we use the blurred Gaussian, because it has approximately the correct appearance and is Feingold and Levin 1986), snowflakes(Gunn and Marshall efficient to compute 1958: Ohtake 1965), and various other hydrometeors, such as graupel and hail(auer 1970; Auer 1972). Previous works tion u=Lux, uy, with orientation 8, variance given on rain removal( garg and Nayar 2004, 2005) have used the breadth b of the streak, and motion blurred over the length l Marshall-Palmer(1948)distribution. For more information, of the streak is given by Microphysics of Clouds and Precipilation by Pruppacher and Klett(1997)is a good general resource for the physics g(x,y;a,z,日, of precipitation. Unfortunately, as discussed by several authors (Jameson and Kostinski 2001, 2002; Desaulniers (a,x) (x-cos()y-u2)2+(y-sin()y-1)2 Soucy et al. 2001; Desaulniers-Soucy 1999), size distr b( butions can be inaccurate. Nevertheless, they give useful general bounds. Both size distributions and observational The values for diameter a and depth z are combined with studies show that drops rarely grow larger than 3 mm. (2),(3), and (4) to compute the correct values of breadth addition, drops smaller than. I mm cannot be seen individ b and length l. in the notation a semicolon is used to dif- ually. Although not accurate for every storm, we find that ferentiate between the parameters of image location versus using a uniform distribution between 1 mm and 3 mm is all others. For example,(x, y;a, z, 8, u) means at location sufficientLy accurate (, y), with parameters u.z., 6, u With all of the variables sampled from their distributions An example of this appearance model is shown in generating images with rain or snow is straightforward with Fig. 2(b). With broad environmental lighting from the sky, this model. But determining if part of an image is rain or the variations due to the drop oscillations discussed in Garg snow would require a search across all(x, y), with each pos and Nayar(2006), Tokay and Beard(1996), Kubesh and sible N,ud, ad, and zd. Performing this search in image Beard(1993)are subtle, so Fig o b/, and(c)appear space would be prohibitive, so we instead perform a simpli- almost identical. even though blurred gaussians are an fied search in frequency space inaccurate approximation of raindrops illuminated with a point light source, most outdoor scenes are not lit with point sources during the day, so the effects of oscillation can be 3 Frequency-Space Analysis ignored Since rain and snow streaks create repeated patterns, it is 2.3 The Appearance of multiple streaks natural to examine them in frequency space. Rather than attempting to find each pixel of each streak, we can The pixel intensity due to rain or snow in one movie frame instead find their general effect on the Fourier transforms at a given location (x, y)should be the sum of the streaks of the images But applying the Fourier transform to(6)does Springer enter column is hree consecutive frames of rain acquired at times t= 1, 2, 3 The left column is three wO-dimensional fourier transforms. onc for cach of the images. The right column is a single three-dimensional transform of all three frames with temporal frequency W=-1,0,. As expected the w= l and w=-1 frequencies re mirror images. But what is interesting is that all of the Fourier transform images appear similar. due to the statistica properties of rain 2D Fourier transforms Rain lmages 3D Fourier transform not make it easier to analyze images For this we make three there are the same number of streaks of each length at each key observations of the magnitude of the fourier transform location of rain and snow Observation 3 The magnitude is approximately constant Observation 1 The shape of the magnitude does not depend across the temporal frequencies. This rightmost column strongly on streaks' locations in an image. Figure 3 shows of Fig. 3 shows the three-dimensional fourier transform an example of the Fourier transform of a sequence with real of all three frames. Interestingly, apart from a few arti rain. The middle column is a sequence of three consecutive facts, the magnitude appears similar across temporal fre frames. They were generated from a sequence of heavy rain quency w. This observation allows us to predict that the with a stationary scene and with an almost stationary cam- three-dimensional Fourier transform will be constant in tem era, by finding the difference of each pixel with the median poral frequency a of itself and its two temporal neighbors These observations will allow us to create a simplified M(r, y,t) model of the frequencies of rain and snow. Instead of find median(M(x, y,t-1),M(,,t), M(x,y, t+ 1) ing every streak individually, we can fit this model by onl The left column of Fig 3 is three separate two- (7) estimating a few parameters 3.1 A Frequency-Space Model of Rain and Snow sional Fourier transforms, one for each image. Notice that even though the streaks are in different locations in differ As shown in(6), an image full of rain or snow is the sum effect of a group of streaks. The same is true in frequency ent frames, the magnitudes appear similar. Appendix a con- space, where the magnitude of the Fourier transform of (6) tains a derivation and simulation that shows the magnitude is only weakly dependent on the number and positions of streaks. We find that although the expansion of the magni tude of the fourier transform of rain and snow can be arbi g(x, v; ad, zd, ed, po 8) trarily complex, it can still be well behaved, which explains the phenomenon seen in the left column which is equivalent to the sum of the Fourier transforms of each streak g: Observation 2 The shape of the magnitude is similar for dif- ferent numbers of streaks. Although the exact Fourier trans forms of images with different numbers of streaks are dif- G(u, v; ad, zd,d,ud ferent from each other, changing the number of streaks has d=1 a similar effect to multiplying all frequencies by a scalar. (Note that in this work, we use only the main lobe of the This pattern is shown in Fig 4. Appendix A also contains blurred Gaussian G, which has a similar appearance to a a validation of this observation, for the special case where standard oriented Gaussian. Int J Comput vis (a) Fig. 4 Three examples of images with streaks rendered by Garg (c)300 streaks. To make them appear similar, each Fourier transform is and Nayar(2006)and their corresponding two dimensional Fourier multiplied by a scalar. Apart from being scaled differently, their mag- transforms. The images have approximately (a)50,(b)100, and nitude appear similar Based on the Observations I and 2 in the previous sec- From Observation 3, we can predict that the magnitude tion, (9)can be simplified as will be constant in temporal frequency w: N ∑G(,;a,xa,D)‖ (10) R*(u, v, W; A, emax, 0min=R*(u, v; A,max, min)(12) The scalars for rotation 0 and brightness A are based on the quation (10)is simpler, but still depends on the num- specific movie. In the next section, we show how to fit 0 ber of streaks n in the image. This is where the statisti- cal properties of rain and snow discussed in Sect. 2.3 be come helpful. Since determining the exact value of each fre 3.2 Fitting the Frequency-Space Model to a Video quency is not vital, we can simplify (10) further, based on three assumptions. First, each spalial location [zmin, marl is equally likely to have a raindrop or snowflake. In a per Only a single intensity A needs to be estimated per frame, spective camera, the volume imaged at a given depth is rel- and often only one orientation 0 per sequence. To estimate ative to the depth squared. This means that in a perspec- these parameters, we can use the fact that rain and snow tive camera, the number of drops imaged at a given depth cover a broad part of the frequency space. Most objects are will also be relative to the depth squared second the result clustered around the lowest frequencies, while rain and snow ing streaks are equally likely to have any orientation within are spread out much more evenly This means that even if the the range [0min, 0max ]. Third, a given particle is equally total energy of the rain or snow is low, a trequency chosen likely to be any size between amin = I mm and amax at random is fairly likely to contain a strong rain or snow mm component. This is especially true if we only examine the Instead of trying to determine the properties of each of non-zero temporal frequencies, which are those that corre the N streaks, we use a model r that has frequencies pro- spond to changes between frames portional to the mean streak and scaled by overall bright- The model parameters can be estimated with two heuris- ness A tics. The scalar multiplier A should be such that the rain/snow model is approximately the same magnitude as R(u,v; A, bmax, emin) the rain or snow in the movie. For the fourier transform of ACC~⊥ a small block of frames, a can be estimated by taking a ra- IG(u,U,a, 2, )dz da de () tio of the median of all frequencies, except for the constant temporal frequ These integrals can be approximated by sampling across 6 a,and z, yielding an estimate of the frequencies of rain and median( M(u, U, w)D median(R*( A=l, 0max. emin)) (13) Int J Comput Vis Fig. 5 Examples of the model for three video sequences. From black background, plus 300 rendered streaks per image. (b) The same top to bottom, we have the original image, its two dimensional Fourier 300 streaks, but now against a moving background with a moving cam transform, and the corresponding rain/snow model (a)A sequence of a era. (c)Real snow and a moving camera Taking the median is effective. because as discussed in ob argmin n Jo (R"(u,U;A,6max,6min)‖-R(u,υ)2dudu servation 3 in the beginning of the section, rain and snow are strong in non-zero temporal frequencies, while most of the (15) scene is concentrated in the zero temporal frequencies Because the search space is one dimensional and bounded The streak orientation can be automatically computed if an exhaustive search can be performed. Different raindrop there is a short subsequence where only rain and snow are generally fall in almost the same direction, so min=0max moving. Again using Observation 3, we expect that indi- for rain. But since snow has a less consistent pattern, a range vidual rain and snow frequencies will change greatly, even of orientations is needed. Using en, 2 radians is though their overall effect stays the same. To find orienta- effective for most videos with snow tion we do not need to find the correct values for each fre- Figure 5 shows the models obtained by fitting to three quency, we only need to determine which are due to rain videos. The frequencies corresponding to the rain are easy to see in(a) and(b), but it easier to see the snow frequen and snow. Therefore, rather than using the median of the ciesinthemoviefor(c)availableathttp://www.cs.cmu frequencies as in(13), we use the standard deviation across eau -barnum/rain/barnumo8frequency html time as a more robust estimator An estimate r of the impor tant frequencies can be obtained by computing the standard deviation over time for each spatial frequency, for T frames: 4 Applications The model that we developed in the previous sections can ∑ (M(u, v,t)-M(u, u)D (14) be used to either decrease or increase the amount of rain =1 and snow. For both cases, the first step is detection, which requires an analysis across entire images, and is performed The correct 0 is found by minimizing the difference be- in frequency space. Once detected, the rain or snow can ei tween the inodel and the estimate ther be directly removed by subtraction, or else the detected Int J Comput vis (a) b f) Fig 6 Rain can be detected in several ways, with the same frequency the ground truth magnitude has any errors is because our method of model. Subfigure(a) shows a frame from the original sequence, which computing the rain/snow component does not generally allow a com has rendered rain streakS.(b) With detection based on a single frame, plete separation of rain/ snow and the clear image. This example shows the rain is segmented fairly accurately, but there are many false de the theoretical limit of using a ratio of magnitudes for a single frame tections. Even the fairly textureless ground is mistakenly detected, b e) Shows detection based on three consecutive frames, with similar cause it shares many of the low frequencies of the model. In(c), de tection still uses a single frame, but a random 50% of the model's fre- accuracy to one frame. The best results are in(f)when dctection is quencies are set to zero. The effect of setting some frequencies to zero performed on a single framc, then refined over thrcc frames. The exact more evident in the videos on the website. )The true magnitude of frequencies of rain and snow change from frame to frame, and using he rain is used instead of our model in (d). The rain is still detected the two step estimation finds only those frequencies that are both rain accurately, although there are fewer erroneous detections. The reason like and rapidly changing pixels can be blended with their temporal neighbors. Alter- of M(u, u), p2 is the estimate based on a single image at nately, to increase the amount of rain or snow, individual time t streaks can be found by matching with blurred Gaussian in image space and then copied onto another image p2(x,y 1)=F-1R*(u, v; A, emax, emin) 4.1 Detecting Rain and snow Using frequency space ×exp(i中{M(u,U)} (16) The output is an image that is bright only where rain or snow The frequency model can be used to detect rain and snow is detected. When R*(u, v)is less than M(u, v), then the ra in a similar way to notch filtering (Gonzalez and woods tio is the estimated percentage of rain and snow at that fre 2002). Intuitively, we want to highlight those frequencies quency. For example. for a given(u, v), if R(u, u)=3 and corresponding to rain and snow while ignoring those corre- M(u, v)=10, then R/M=3. This means that we believe sponding to objects in the scene. This can be done with a that thirty percent of the energy at(u, u)is due to rain and simple ratio. For example, suppose that the model predicts snow. If R*(u,u)> M(u, u), then the ratio is greater than a low value for a given frequency, but the actual value is one, which is not meaningful. The ratio of r* over M is high. Something besides rain or snow is likely causing the therefore capped at one. This capping is both semantically high value. The frequencies that are mostly due to rain and valid as well as practical, in that it prevents frequencies with snow should be found first, as estimated by the ratio of the a very high value for R"/M from dominating the result predicted value to the true value Figure 6(b) shows the one-frame estimation. For visual Detecting streaks in a single frame is done by taking the comparison, Fig. 6(c)shows the result if 50% of the fre inverse transform of the estimate of the proportion of energy quencies in the model are set to zero before using(16) due to rain or snow. Where m(u, v) is the two dimensional figure 6(d)show s the result if the ground truth of the rain Fourier transform of one movie frame and is the phase magnitude is used in place of the rain model Springer Int J Comput Vis By performing a three-dimensional transform of the im- case, c is the per-pixel temporal median of c-.Where ages and using(u, v, w)instead of (u, v),(16)can be used P3 is the detection from (17)as applied to cl-l, the next for a three dimensional transform of multiple consecutive iteration is frames, shown in Fig. 6(e). Using multiple frames improves the accuracy, but not significant c"(x,y,1)=(-a3(x,y,1)n(x,y,t) We found through experimentation that the best approach +ap(x, y, t)c"(x, y, 1) (19) is to perform a three dimensional analysis on a series of consecutive two dimensional estimates , a three dimen sional Fourier transform is applied to p2(, y, t) to ob lecting a good value for a is not difficult. We use a fixed a=3 for all the results in this paper. And as with(18), each P2(u, v,w), and the resulting rain/snow estimation is then: iteration is required to be less than or equal to the original Although the result from each subsequent iteration is p3(x,y,(=F-1R*(u, v,w; A, e, max,min more clear than the previous, the amount of rain and snow ‖P2(l,U,U川 removed decreases per iteration. This means that it may be necessary to iterate many times to remove most of the ×exp(iφ{M(u,,)) (17) streaks. Since this is time consuming, the process can be it- erated only a few times and subsequent cs can be linearly Figure 6(f) shows the results from this method. At first extrapolated from the final two iterations. Figure 7 shows glance, it appears even better than the single frame ground the results from the iterative removal method on four exam- truth in Fig. 6(d ). But the ground truth magnitude actually ple sequences with moving cameras and scenes correctly identifies streaks more precisely, even if it has more false detections. But since the ground truth is not gen 4.3 Increasing Rain and Snow in Image Space erally known, we use p3 as our final estimate of the location of the rain and snow Although the rain and snow can be detected using only the frequency magnitude, creating new streaks requires manip- 4.2 Reducing Rain and snow Using the Frequency Space ulation of phase as well. The main ad vantage of working in Model frequency space was that the locations of the streaks could be ignored. But since we need them for rendering, it is sim- Once detected, the rain or snow pixels can be removed by pler to work in image space. Our approach is to use the replacing them with their temporal neighbors. The detected blurred gaussian model to sample real rain and snow rain and snow p3(, y, t)is used as a mixing weight between We start with the rain/snow estimate m(x, t, y) the original image m and an initial estimate c of the clear im- c"(,y, t). Large streaks are detected by filtering the rain/ age c We find that a per-pixel temporal median filter works snow estimate with a bank of size derivatives of blurred vell for C, although it could be the output of any rain/snow Gaussians, similar to scale detection in(Mikolajczyk and removal algorithm. The detection P3(x, y, t)is multiplied Schmid 2001). For an average a and z with orientation A by the removal rate a, where the product of ap3(x, y, t) is the size derivative is given by capped at one: f(x,y:r1, 22, a,z, 0) c(x, y,t)=(1-ap3(,y, t))m(r, y, t) g(y1x,y1y;a,x,6)-8(2x,y2y;a,x,) (20) +ap3(r, y,t)c(r, y, t) where y ,y1<r2, an the blurred Since rain and snow are brighter than their background, Gaussian terms are normalized to sum to one. Each image c(x, y, t) is required to be less than or equal to m(x, y, t). is filtered with a set of different ys. The filter with the max For a large a, ap3(x, y, t)equals 1 for all(x, y, t), there- imum response corresponds to the size of the streak at that fore c will approach c location Images created with this equation will be temporally This detection method will find many strong streaks, but blurred only where the rain and snow is present. But the dis- will have a few false matches as well. Since we do not need advantage is that it can never remove more rain and snow to find every streak, several steps are taken to cull the se than the initial estimate C If removal is more important than lection. First, locations that appear to have very large and smoothness then we can iterate the detection and removal very small scales are eliminated, and non-maxima suppres- The first iteration cl is the result from(18)on the orig- sion is performed in both location and scale. Next, to ensure inal sequence. Subsequent iterations are based on the last that only bright streaks are used, only the streak candidates lear estinate cn-I and the last initial estimate Cn-I. In this with the most total energy are kept. (The total energy is the J Comput V (a)The mailbox sequence: There are objects at various ranges, between approximately I to 30 meters from the camera. The writing on the mailbox looks similar to snow most of the snow can be removed although there are some errors on the edges of the mailbox and on the bushes (b) Walkers in the snow: This is a very difficult sequence with a lot of high frequency textures, very heavy snow, and multiple moving objects Much of the snow is removed, but the edges of the umbrella and parts of the people's legs are misclassified (c) Sitting man scqucnce: This scene is from the movic Forrest Gump. The rain streaks arc fairly large, as is common in films. The rain can bc completely removed, although the letters and windows in the upper portion of the images are misclassified ( d)A windowed building: The rain is not very heavy, but this sequence is difficult, because there are a large number of straight, bright lines from the window frames and the branches. almost all of the rain is removed but parts of the window frames and the bushes are erroneously detected Fig. 7 Several examples of rain and snow removal based on spatio-temporal frequency detection. Some of the sequences have several movin objects, others have a cluttered foreground with high frequency textures, and all of them are taken with a moving camera Springer 【实例截图】
【核心代码】