1 | % * This code was used in the following articles:
|
---|
2 | % * [1] Learning 3-D Scene Structure from a Single Still Image,
|
---|
3 | % * Ashutosh Saxena, Min Sun, Andrew Y. Ng,
|
---|
4 | % * In ICCV workshop on 3D Representation for Recognition (3dRR-07), 2007.
|
---|
5 | % * (best paper)
|
---|
6 | % * [2] 3-D Reconstruction from Sparse Views using Monocular Vision,
|
---|
7 | % * Ashutosh Saxena, Min Sun, Andrew Y. Ng,
|
---|
8 | % * In ICCV workshop on Virtual Representations and Modeling
|
---|
9 | % * of Large-scale environments (VRML), 2007.
|
---|
10 | % * [3] 3-D Depth Reconstruction from a Single Still Image,
|
---|
11 | % * Ashutosh Saxena, Sung H. Chung, Andrew Y. Ng.
|
---|
12 | % * International Journal of Computer Vision (IJCV), Aug 2007.
|
---|
13 | % * [6] Learning Depth from Single Monocular Images,
|
---|
14 | % * Ashutosh Saxena, Sung H. Chung, Andrew Y. Ng.
|
---|
15 | % * In Neural Information Processing Systems (NIPS) 18, 2005.
|
---|
16 | % *
|
---|
17 | % * These articles are available at:
|
---|
18 | % * http://make3d.stanford.edu/publications
|
---|
19 | % *
|
---|
20 | % * We request that you cite the papers [1], [3] and [6] in any of
|
---|
21 | % * your reports that uses this code.
|
---|
22 | % * Further, if you use the code in image3dstiching/ (multiple image version),
|
---|
23 | % * then please cite [2].
|
---|
24 | % *
|
---|
25 | % * If you use the code in third_party/, then PLEASE CITE and follow the
|
---|
26 | % * LICENSE OF THE CORRESPONDING THIRD PARTY CODE.
|
---|
27 | % *
|
---|
28 | % * Finally, this code is for non-commercial use only. For further
|
---|
29 | % * information and to obtain a copy of the license, see
|
---|
30 | % *
|
---|
31 | % * http://make3d.stanford.edu/publications/code
|
---|
32 | % *
|
---|
33 | % * Also, the software distributed under the License is distributed on an
|
---|
34 | % * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
|
---|
35 | % * express or implied. See the License for the specific language governing
|
---|
36 | % * permissions and limitations under the License.
|
---|
37 | % *
|
---|
38 | % */
|
---|
39 | function [] = OcclusionRefinement(defaultPara, Wrlname, PairList, PostFixStrAfter, Type, PostFixStr) |
---|
40 | |
---|
41 | % This function do three thing |
---|
42 | % 1) detect the occlusion from the current model |
---|
43 | % 2) decide redundent plane |
---|
44 | % 3) Re-Render by not rendering the redundent plane information |
---|
45 | |
---|
46 | % Input: |
---|
47 | % 1) defaultPara - all parameter share for the whole stitching3d folder |
---|
48 | % 2) Wrlname - the name of the model, different by the PairList |
---|
49 | % 3) PairList - the linearly order of image that been added to the model |
---|
50 | |
---|
51 | % Fixed constant --- need to be put in defaultPara by Min later |
---|
52 | depthratioMin = 0.01; |
---|
53 | depthratioMax = 100; |
---|
54 | NegI = diag([1 1 -1]); |
---|
55 | if nargin <6 |
---|
56 | PostFixStr = 'NonMono';%'NonMonoOccluMatched' |
---|
57 | end |
---|
58 | if nargin <4 |
---|
59 | PostFixStrAfter = 'OccluRemove'; |
---|
60 | end |
---|
61 | DisCtsThre = 0.1; % fractional error |
---|
62 | OcclusionRatioThre = 1.1; |
---|
63 | SurfFlag = 0; % default not using surf Feature point to find occlusion |
---|
64 | if strcmp( lower(Type), 'matches') |
---|
65 | SurfFlag = 1; |
---|
66 | end |
---|
67 | |
---|
68 | % 1) Run occlusion detection and Matcheing for every pair of images |
---|
69 | |
---|
70 | ImgList = unique(PairList); |
---|
71 | NumImg = length(ImgList); |
---|
72 | |
---|
73 | if defaultPara.Flag.FlagOccluDetect |
---|
74 | |
---|
75 | tic; % start timing the time for the OccluDetection Step |
---|
76 | |
---|
77 | % copy the ImgInfo of each image to another "PostFixStrAfter".mat file |
---|
78 | if defaultPara.Flag.FlagResetModeltoNomoModel |
---|
79 | for i = 1:NumImg |
---|
80 | load([defaultPara.Fdir '/data/' ImgList{i} '/' Wrlname '_' ImgList{i} '_' PostFixStr '.mat']); |
---|
81 | save([defaultPara.Fdir '/data/' ImgList{i} '/' Wrlname '_' ImgList{i} '_' PostFixStrAfter '.mat'],'model'); |
---|
82 | end |
---|
83 | end |
---|
84 | |
---|
85 | % detect occlusion for every pair of image |
---|
86 | for i = 1:NumImg |
---|
87 | for j = (i+1):NumImg |
---|
88 | |
---|
89 | % [Img1Index] = ImgInfoIndexFromName(ImgInfo, ImgList{i}); |
---|
90 | % [Img2Index] = ImgInfoIndexFromName(ImgInfo, ImgList{j}); |
---|
91 | |
---|
92 | [ImgInfo1 ImgInfo2 Img1Index Img2Index Pair GlobalScale] = ... % What to do with Pairs haven't been matched ??????????????????? |
---|
93 | LoadDataForFindOcclu(defaultPara, Wrlname, ImgList{i}, ImgList{j}, ... |
---|
94 | 0, PostFixStrAfter); % load all imformation needed for occlusion detection |
---|
95 | |
---|
96 | % Define variables |
---|
97 | H = 2274; |
---|
98 | V = 1704; |
---|
99 | Img1 = ImgInfo1.ExifInfo.IDName |
---|
100 | Img2 = ImgInfo2.ExifInfo.IDName |
---|
101 | I1=imreadbw([defaultPara.Fdir '/pgm/' Img1 '.pgm']); % function from sift |
---|
102 | I2=imreadbw([defaultPara.Fdir '/pgm/' Img2 '.pgm']); % function from sift |
---|
103 | ImgScale1 = size(I1); |
---|
104 | ImgScale2 = size(I2); |
---|
105 | |
---|
106 | % time consuming about 5mins |
---|
107 | if ~defaultPara.Flag.FlagPreloadOccluDetect |
---|
108 | [PointPix1 PointDepth1 FaceSetPickedIND1 POriReprojM1 FieldOccluPix1 OccluDist1 OccluedFaceSetIND1 OccluedFaceSetIDRemained1... |
---|
109 | PointPix2 PointDepth2 FaceSetPickedIND2 POriReprojM2 FieldOccluPix2 OccluDist2 OccluedFaceSetIND2 OccluedFaceSetIDRemained2] = ... |
---|
110 | FindOccluPair(defaultPara, ImgInfo1, ImgInfo2, Pair, GlobalScale, SurfFlag); % detect occlsion (use Ray as detector not the surf Features) |
---|
111 | % data Struction define: |
---|
112 | % PointPix PointDepth FaceSetPickedIND POriReprojM FieldOccluPix OccluDist OccluedFaceSetIND |
---|
113 | % -- allof the size that single ray pass through both FaceSet |
---|
114 | % FaceSetPickedIND1 used in DepthMap size or surfFeature size |
---|
115 | % OccluedFaceSetIND1 used in DepthMap size |
---|
116 | if SurfFlag |
---|
117 | save([ defaultPara.Fdir '/data/' Img1 '_' Img2 '_' PostFixStrAfter '_OccluDetect_Match.mat'],... |
---|
118 | 'PointPix1','PointDepth1','FaceSetPickedIND1','POriReprojM1','FieldOccluPix1','OccluDist1',... |
---|
119 | 'OccluedFaceSetIND1','OccluedFaceSetIDRemained1', ... |
---|
120 | 'PointPix2','PointDepth2','FaceSetPickedIND2','POriReprojM2','FieldOccluPix2','OccluDist2',... |
---|
121 | 'OccluedFaceSetIND2','OccluedFaceSetIDRemained2'); |
---|
122 | else |
---|
123 | save([ defaultPara.Fdir '/data/' Img1 '_' Img2 '_' PostFixStrAfter '_OccluDetect_Ray.mat'],... |
---|
124 | 'PointPix1','PointDepth1','FaceSetPickedIND1','POriReprojM1','FieldOccluPix1','OccluDist1',... |
---|
125 | 'OccluedFaceSetIND1','OccluedFaceSetIDRemained1', ... |
---|
126 | 'PointPix2','PointDepth2','FaceSetPickedIND2','POriReprojM2','FieldOccluPix2','OccluDist2',... |
---|
127 | 'OccluedFaceSetIND2','OccluedFaceSetIDRemained2'); |
---|
128 | %'PointPix1','PointDepth1','FaceSetPickedIND1','POriReprojM1','FieldOccluPix1','OccluDist1','OccluedFaceSetIND1',... |
---|
129 | %'PointPix2','PointDepth2','FaceSetPickedIND2','POriReprojM2','FieldOccluPix2','OccluDist2','OccluedFaceSetIND2'); |
---|
130 | end |
---|
131 | else |
---|
132 | if SurfFlag |
---|
133 | load([defaultPara.Fdir '/data/' Img1 '_' Img2 '_' PostFixStrAfter '_OccluDetect_Match.mat']); |
---|
134 | else |
---|
135 | load([defaultPara.Fdir '/data/' Img1 '_' Img2 '_' PostFixStrAfter '_OccluDetect_Ray.mat']); |
---|
136 | end |
---|
137 | end |
---|
138 | |
---|
139 | % Define occlusion if OccluDist > CentainThreshold |
---|
140 | if defaultPara.Flag.FlagRecipicalOccluDetection |
---|
141 | Mask1 = PointDepth1./OccluDist1 > OcclusionRatioThre | OccluDist1./PointDepth1 > OcclusionRatioThre; |
---|
142 | Mask2 = PointDepth2./OccluDist2 > OcclusionRatioThre | OccluDist2./PointDepth2 > OcclusionRatioThre; |
---|
143 | else |
---|
144 | Mask1 = PointDepth1./OccluDist1 > OcclusionRatioThre;%defaultPara.OccluDistThre; |
---|
145 | Mask2 = PointDepth2./OccluDist2 > OcclusionRatioThre;%defaultPara.OccluDistThre; |
---|
146 | end |
---|
147 | |
---|
148 | % Action: find new matches or decide plane to remove given the occlusion infomation |
---|
149 | switch lower(Type) |
---|
150 | case 'matches' |
---|
151 | [matches]=MatchPointsGivenOcclusion(defaultPara, ImgInfo1, ImgInfo2, ImgScale1, ImgScale2, Img1, Img2, Img1Index, Img2Index, ... |
---|
152 | Pair, GlobalScale, Wrlname, PostFixStrAfter, ... |
---|
153 | POriReprojM1(:,Mask1), FieldOccluPix1(:,Mask1), FaceSetPickedIND1(:,Mask1), ... |
---|
154 | POriReprojM2(:,Mask2), FieldOccluPix2(:,Mask2), FaceSetPickedIND2(:,Mask2) ); |
---|
155 | case 'corrmatches' |
---|
156 | if ~isempty(Pair.lamda) |
---|
157 | depthratio = Pair.lamda(1,:)./Pair.lamda(2,:); |
---|
158 | % Min add to remove outliers (should be already removed when doing PoseEst.m) |
---|
159 | Inliers = depthratio > depthratioMin & depthratio < depthratioMax; |
---|
160 | depthratio = depthratio(Inliers); |
---|
161 | maxRatio = max(depthratio); |
---|
162 | minRatio = min(depthratio); |
---|
163 | else |
---|
164 | maxRatio = max([GlobalScale(1)/GlobalScale(2) GlobalScale(2)/GlobalScale(1)]); |
---|
165 | minRatio = 1/maxRatio; |
---|
166 | end |
---|
167 | [Matches1 CoeffM1 Inliers1]=CorrolationMatch( defaultPara, Pair, I1, I2, PointPix1(:,Mask1), POriReprojM1(:,Mask1), FieldOccluPix1(:,Mask1), [minRatio maxRatio]); |
---|
168 | Pair2_1.R = Pair.R'; |
---|
169 | Pair2_1.T = -Pair.R'*Pair.T; |
---|
170 | if defaultPara.Flag.FlagRefinementDisp |
---|
171 | disp('Start Reverse CorrolationMatches Hold on'); |
---|
172 | whos |
---|
173 | end |
---|
174 | [Matches2 CoeffM2 Inliers2]=CorrolationMatch( defaultPara, Pair2_1, I2, I1, PointPix2(:,Mask2), POriReprojM2(:,Mask2), FieldOccluPix2(:,Mask2), [1/maxRatio 1/minRatio]); |
---|
175 | if defaultPara.Flag.FlagRefinementDisp |
---|
176 | disp('CorrolationMatch.m Finished'); |
---|
177 | end |
---|
178 | Matches1 = Matches1(:,Inliers1); |
---|
179 | Matches2 = Matches2(:,Inliers2); |
---|
180 | CoeffM1 = CoeffM1(:,Inliers1); |
---|
181 | CoeffM2 = CoeffM2(:,Inliers2); |
---|
182 | |
---|
183 | % Check if the Matches are not mutual discard the one with less Coeff(Cross-Corrolation value) =============== |
---|
184 | Matches = [ Matches1 [Matches2(3:4,:); Matches2(1:2,:)]]; |
---|
185 | CoeffM = [ CoeffM1 CoeffM2]; |
---|
186 | % Min used different algorithm than SurFeature Matches |
---|
187 | [Inliers] = CleanMatch(Matches, CoeffM(1,:)); % choose the matches with higher Coeff is the matches is not mutual |
---|
188 | [InliersReverse] = CleanMatch(Matches(:,Inliers), CoeffM(1,Inliers)); % choose the matches with higher Coeff is the matches is not mutual |
---|
189 | Inliers = Inliers(InliersReverse); |
---|
190 | Matches = Matches(:,Inliers); |
---|
191 | CoeffM = CoeffM(:,Inliers); |
---|
192 | |
---|
193 | if defaultPara.Flag.FlagRefinementDisp |
---|
194 | figure; plotmatches(I1,I2,Matches(1:2,:), Matches(3:4,:),repmat(1:size(Matches,2),2,1), 'Stacking','v','Interactive', 3); |
---|
195 | end |
---|
196 | |
---|
197 | save([defaultPara.Fdir '/data/' Img1 '_' Img2 '_' PostFixStrAfter 'BeforeFiltering.mat'],'Matches','CoeffM'); |
---|
198 | |
---|
199 | % use Coeff as threshould to filter out error matches |
---|
200 | CoeffMask = CoeffM(1,:) > defaultPara.CoeffMThre; |
---|
201 | [inlier, Residual] = EpipoPrune(defaultPara, Pair, Matches, ImgScale1); |
---|
202 | EpipolarResidualMask = Residual < defaultPara.ResidualThre; |
---|
203 | CoeffRationMask = CoeffM(2,:)./CoeffM(1,:) < defaultPara.coeffratioThre; |
---|
204 | CoeffRationMask( CoeffM(1,:) == 0) = false;% CoeffM(1,:) == 0 means not satisfy the epipolar constrain |
---|
205 | Mark = CoeffMask & CoeffRationMask & EpipolarResidualMask; |
---|
206 | |
---|
207 | save([defaultPara.Fdir '/data/' Img1 '_' Img2 '_' PostFixStrAfter '.mat'],'Matches','CoeffM','Mark'); |
---|
208 | |
---|
209 | % ======================================================================= |
---|
210 | if ~isempty(Matches(:,Mark)) |
---|
211 | tempf1 = Matches(1:2,Mark); |
---|
212 | tempf2 = Matches(3:4,Mark); |
---|
213 | x_calib = [ inv(defaultPara.InrinsicK1)*[ tempf1; ones(1,size(tempf1,2))];... |
---|
214 | inv(defaultPara.InrinsicK2)*[ tempf2; ones(1,size(tempf2,2))]]; |
---|
215 | [ lamda1 lamda2 Error] = triangulation( defaultPara, Pair.R, Pair.T, x_calib); |
---|
216 | % notice lamda re-scale to local model scale |
---|
217 | lamda1 = lamda1./GlobalScale(1); |
---|
218 | lamda2 = lamda2./GlobalScale(2); |
---|
219 | % Storage the match result for later ReInference |
---|
220 | AddMatch2Model(defaultPara, Wrlname, lamda1, Matches(1:2,Mark), ImgInfo1, ImgScale1, Img1Index, Img2Index, PostFixStrAfter, Error); |
---|
221 | AddMatch2Model(defaultPara, Wrlname, lamda2, Matches(3:4,Mark), ImgInfo2, ImgScale2, Img2Index, Img1Index, PostFixStrAfter, Error); |
---|
222 | % Storage the New Matches |
---|
223 | if defaultPara.Flag.FlagRefinementDisp |
---|
224 | disp('Storaging Occlusion Surf Features Matches'); |
---|
225 | save([defaultPara.Fdir '/data/' Img1 '_' Img2 '_' PostFixStrAfter '.mat'],'Matches','CoeffM','Error','Mark'); |
---|
226 | end |
---|
227 | end |
---|
228 | |
---|
229 | |
---|
230 | case 'remove' |
---|
231 | RemovePlaneGivenOcclusion(defaultPara, ImgInfo1, ImgInfo2, Pair, GlobalScale, Img1, Img2, Wrlname, PostFixStrAfter,... |
---|
232 | FaceSetPickedIND1, OccluedFaceSetIND1, FaceSetPickedIND2, OccluedFaceSetIND2, Mask1, Mask2); |
---|
233 | end |
---|
234 | end |
---|
235 | end |
---|
236 | TimeForOccluDetectionFirstStep = toc; |
---|
237 | if defaultPara.Flag.FlagRefinementDisp |
---|
238 | disp(['Finish OccluDetectionFirstStep in ' num2str(TimeForOccluDetectionFirstStep) ' seconds']); |
---|
239 | end |
---|
240 | |
---|
241 | end % end of if defaultPara.Flag.FlagOccluDetect |
---|
242 | |
---|
243 | % 2) Render of each image individiually ----------------------------------------------------------------------------------- |
---|
244 | if defaultPara.Flag.FlagRefinementReInference |
---|
245 | |
---|
246 | tic; % timing the Refinement Step |
---|
247 | |
---|
248 | for i = 1:NumImg |
---|
249 | |
---|
250 | switch lower(Type) |
---|
251 | case 'remove' |
---|
252 | % Data preparing and ReInfernece |
---|
253 | Default.Wrlname{1} = [Wrlname '_' ImgList{i} '_' PostFixStrAfter]; |
---|
254 | Default.filename{1} = ['../' ImgList{i}]; |
---|
255 | load([defaultPara.Fdir '/data/' ImgList{i} '/' Wrlname '_' ImgList{i} '_' PostFixStrAfter '.mat']); |
---|
256 | Default.OutPutFolder = [ defaultPara.OutPutFolder ImgList{i} '/']; |
---|
257 | R = LoadModelStatus( defaultPara.Fdir, Wrlname, ImgList{i}, 'R'); |
---|
258 | T = LoadModelStatus( defaultPara.Fdir, Wrlname, ImgList{i}, 'T'); |
---|
259 | |
---|
260 | if false |
---|
261 | DiffHori = conv2(model.PlaneParaInfo.FitDepth, [1 -1], 'valid'); |
---|
262 | DiffVert = conv2(model.PlaneParaInfo.FitDepth, [1; -1], 'valid'); |
---|
263 | maskH = [ (DiffHori./model.PlaneParaInfo.FitDepth(:,2:end) > DisCtsThre) zeros(size(model.PlaneParaInfo.FitDepth,1),1)]; |
---|
264 | maskH(end,:) = 0; |
---|
265 | maskV = [ (DiffVert./model.PlaneParaInfo.FitDepth(2:end,:) > DisCtsThre); zeros(1, size(model.PlaneParaInfo.FitDepth,2))]; |
---|
266 | maskV(:,end) = 0; |
---|
267 | model.PlaneParaInfo.SupOri(logical( maskH) ) = 0; |
---|
268 | model.PlaneParaInfo.SupOri(logical( maskV) ) = 0; |
---|
269 | end |
---|
270 | |
---|
271 | WrlFacestHroiReduce(model.PlaneParaInfo.Position3DFited, model.PlaneParaInfo.PositionTex, model.PlaneParaInfo.SupOri, ... |
---|
272 | Default.filename{1}, Default.Wrlname{1}, ... |
---|
273 | Default.OutPutFolder, 0, 0); |
---|
274 | |
---|
275 | case {'matches','corrmatches'} |
---|
276 | % Data preparing and ReInfernece |
---|
277 | Default.Wrlname{1} = [Wrlname '_' ImgList{i} '_' PostFixStrAfter]; |
---|
278 | load([defaultPara.Fdir '/data/' ImgList{i} '/' Wrlname '_' ImgList{i} '_' PostFixStrAfter '.mat']); |
---|
279 | R = LoadModelStatus( defaultPara.Fdir, Wrlname, ImgList{i}, 'R'); |
---|
280 | T = LoadModelStatus( defaultPara.Fdir, Wrlname, ImgList{i}, 'T'); |
---|
281 | Scale = LoadModelStatus(defaultPara.Fdir, Wrlname, ImgList{i}, 'Scale'); |
---|
282 | SingleImgReInference(defaultPara, model, ImgList{i}, LoadModelStatus(defaultPara.Fdir, Wrlname, ImgList{i}, 'GroundLevel'),Wrlname, ... |
---|
283 | R, T, Scale ,PostFixStrAfter); |
---|
284 | end |
---|
285 | |
---|
286 | % Build Meta Wrl file ========= |
---|
287 | Path = [ defaultPara.OutPutFolder Wrlname '_' PostFixStrAfter '.wrl']; |
---|
288 | InLinePath = ['./' ImgList{i} '/' Default.Wrlname{1} '.wrl']; |
---|
289 | Q = Rotation2Q(NegI*R*NegI);% for Wrl (-) z component; |
---|
290 | if any(isnan(Q)) |
---|
291 | Q = zeros(4,1); |
---|
292 | end |
---|
293 | WRLT = T; |
---|
294 | WRLT(3) = -WRLT(3);% for Wrl (-) z component; |
---|
295 | Scale = LoadModelStatus(defaultPara.Fdir, Wrlname, ImgList{i}, 'Scale'); |
---|
296 | if i == 1 |
---|
297 | BuildVrmlMetaModel(1, defaultPara.OutPutFolder, Path, InLinePath, Q, WRLT, repmat(Scale,3,1)); |
---|
298 | else |
---|
299 | BuildVrmlMetaModel(0, defaultPara.OutPutFolder, Path, InLinePath, Q, WRLT, repmat(Scale,3,1)); |
---|
300 | end |
---|
301 | % =============================== |
---|
302 | |
---|
303 | end |
---|
304 | |
---|
305 | TimeForRefinementStep = toc; |
---|
306 | if defaultPara.Flag.FlagRefinementDisp |
---|
307 | disp(['Finish RefinementStep in ' num2str(TimeForRefinementStep) ' seconds']); |
---|
308 | end |
---|
309 | end % end of if defaultPara.Flag.FlagRefinementReInference |
---|
310 | |
---|
311 | return; |
---|