-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
phase_linking stage takes too long #90
Comments
Hi @gjustin40 |
Hi, I would like to ask how best to optimize the phase-linking process for my machine. My CPU is 4.5 GHz with 16 cores 32 threads. However, some of the patches (200 x 200, 135 slc images in stack) take >40 hours to run (see below example of job 1). With another 31 jobs still in the queue, this stack may take >1 month to complete. My AOI is 4000x104000 pixels. Do you have any advice for improving efficiency, or are the default parameters the best I can hope for with a large stack of SLC images (135)? Thank you for any help you can provide. 20240928:180440 * phase_linking.py --work_dir /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy --range_window 15 --azimuth_window 15 --patch_size 200 --method sequential_EMI --test ks --num_worker 32 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --mask /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/mask_hills.h5 --index 0 --slc_stack /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5 [======================= 72% =======> ] 29000/40000 pixels, patch 27 0s / 0s Phase inversion of PATCH_0024 is Completed in 00 mins 41 secs [======================= 90% ================> ] 36000/40000 pixels, patch 27 0s / 0s Phase inversion of PATCH_0025 is Completed in 00 mins 41 secs [======================= 98% ====================> ] 39000/40000 pixels, patch 30 0s / 0s Phase inversion of PATCH_0023 is Completed in 00 mins 42 secs [==================================================] 40000/40000 pixels, patch 30 Phase inversion of PATCH_0028 is Completed in 00 mins 42 secs [==================================================] 40000/40000 pixels, patch 27
[==================================================] 40000/40000 pixels, patch 21 Phase inversion of PATCH_0021 is Completed in 03 mins 37 secs [==================================================] 40000/40000 pixels, patch 20 Phase inversion of PATCH_0020 is Completed in 186 mins 37 secs [==================================================] 40000/40000 pixels, patch 18 Phase inversion of PATCH_0018 is Completed in 452 mins 47 secs [==================================================] 40000/40000 pixels, patch 19 Phase inversion of PATCH_0019 is Completed in 459 mins 26 secs [==================================================] 40000/40000 pixels, patch 0 Phase inversion of PATCH_0000 is Completed in 489 mins 44 secs [==================================================] 40000/40000 pixels, patch 17 Phase inversion of PATCH_0017 is Completed in 768 mins 15 secs [==================================================] 40000/40000 pixels, patch 16 Phase inversion of PATCH_0016 is Completed in 1140 mins 35 secs [==================================================] 40000/40000 pixels, patch 15 Phase inversion of PATCH_0015 is Completed in 1169 mins 03 secs [==================================================] 40000/40000 pixels, patch 14 Phase inversion of PATCH_0014 is Completed in 1206 mins 00 secs [==================================================] 40000/40000 pixels, patch 9 Phase inversion of PATCH_0009 is Completed in 1852 mins 08 secs [==================================================] 40000/40000 pixels, patch 10 Phase inversion of PATCH_0010 is Completed in 1990 mins 24 secs [==================================================] 40000/40000 pixels, patch 8 Phase inversion of PATCH_0008 is Completed in 2015 mins 18 secs [==================================================] 40000/40000 pixels, patch 1 Phase inversion of PATCH_0001 is Completed in 2097 mins 30 secs [==================================================] 40000/40000 pixels, patch 5 Phase inversion of PATCH_0005 is Completed in 2110 mins 36 secs [==================================================] 40000/40000 pixels, patch 4 Phase inversion of PATCH_0004 is Completed in 2140 mins 56 secs [==================================================] 40000/40000 pixels, patch 11 Phase inversion of PATCH_0011 is Completed in 2221 mins 22 secs [==================================================] 40000/40000 pixels, patch 2 Phase inversion of PATCH_0002 is Completed in 2223 mins 13 secs [==================================================] 40000/40000 pixels, patch 13 Phase inversion of PATCH_0013 is Completed in 2338 mins 40 secs [==================================================] 40000/40000 pixels, patch 6 Phase inversion of PATCH_0006 is Completed in 2360 mins 34 secs [==================================================] 40000/40000 pixels, patch 3 Phase inversion of PATCH_0003 is Completed in 2363 mins 04 secs [==================================================] 40000/40000 pixels, patch 12 Phase inversion of PATCH_0012 is Completed in 2500 mins 21 secs [==================================================] 40000/40000 pixels, patch 7 Phase inversion of PATCH_0007 is Completed in 2500 mins 51 secs Using default MintPy Path: /home/seis/anaconda3/envs/insar/lib/python3.10/site-packages [=============> 28% ] 11000/40000 pixels, patch 37 0s / 0s Phase inversion of PATCH_0036 is Completed in 00 mins 36 secs |
a typo in the above dimensions of my AOI, it is actually 4000x10400 pixels. |
@Jesse-kearse |
Thanks for your reply, @mirzaees . Which version of scipy do you suggest I downgrade to? are "new versions" 1.0 and newer? |
I have the following version installed in my miaplpy conda environment: scipy==1.10.1; numpy==1.23.1 I am still experiencing extremely slow phase_linking. Are the version listed above perhaps still to new? I have used "conda install scipy==1.10.1" to downgrade to these versions, but do I need to re-install Miaplpy as well, I wouldn't think so? Thanks for any help you can provide. |
Phase linking takes a lot of time to compute because DS-InSAR is very time consuming. After reducing numpy and scipy to the version you describe, if it is still very slow at this point, that is normal. If you don't downgrade numpy and scipy to the current version, it will take 10 times as long as it does now. DS-InSAR is suitable for very small study areas, and large areas can take months to years to process. |
Thanks for your comment. I do understand that the DS-InSAR is computationally expensive. My confusion is that after Miaplpy separates the study area into patches of 200x200 elements, a single patch is taking up to 50 hours to run - which I thought would be independent of the study area size, as it still needs to loop through all the other patches with the subsequent jobs. Is this correct. Maybe I am not understanding something more fundamental about the computations. Thanks again for any further help (and please ignore my comment on the other thread) |
Does it still take around 50 hours for a single patch after lowering the scipy version and numpy version, normally it takes around 5 hours for a single patch. |
Please output the following code in miaplpy environment |
Here is an example of job 4 (32 patches processed in parallel) from start to finish. you can see that it took 4720 minutes for the final patch to be processed, before beginning job 5 (which I terminated sometime after 5600 minutes). 20241009:200325 * phase_linking.py --work_dir /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy --range_window 15 --azimuth_window 15 --patch_size 200 --method sequential_EMI --test ks --num_worker 32 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --mask /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/mask_hills.h5 --index 4 --slc_stack /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5 [======================= 81% ===========> ] 32500/40000 pixels, patch 149 0s / 0s Phase inversion of PATCH_0154 is Completed in -1 mins 59 secs [==================================================] 40000/40000 pixels, patch 155 Phase inversion of PATCH_0151 is Completed in -1 mins 59 secs [======================= 98% ====================> ] 39000/40000 pixels, patch 149 0s / 0s Phase inversion of PATCH_0157 is Completed in -1 mins 59 secs [======================= 96% ===================> ] 38500/40000 pixels, patch 159 0s / 0s Phase inversion of PATCH_0146 is Completed in -1 mins 59 secs [==================================================] 40000/40000 pixels, patch 159 Phase inversion of PATCH_0152 is Completed in -1 mins 59 secs [==================================================] 40000/40000 pixels, patch 149 Phase inversion of PATCH_0150 is Completed in -1 mins 59 secs
[==================================================] 40000/40000 pixels, patch 144 Phase inversion of PATCH_0144 is Completed in 1556 mins 40 secs [==================================================] 40000/40000 pixels, patch 137 Phase inversion of PATCH_0137 is Completed in 2192 mins 40 secs [==================================================] 40000/40000 pixels, patch 135 Phase inversion of PATCH_0135 is Completed in 2206 mins 49 secs [==================================================] 40000/40000 pixels, patch 138 Phase inversion of PATCH_0138 is Completed in 2285 mins 20 secs [==================================================] 40000/40000 pixels, patch 140 Phase inversion of PATCH_0140 is Completed in 2413 mins 60 secs [==================================================] 40000/40000 pixels, patch 136 Phase inversion of PATCH_0136 is Completed in 2747 mins 28 secs [==================================================] 40000/40000 pixels, patch 141 Phase inversion of PATCH_0141 is Completed in 2883 mins 44 secs [==================================================] 40000/40000 pixels, patch 139 Phase inversion of PATCH_0139 is Completed in 2885 mins 07 secs [==================================================] 40000/40000 pixels, patch 143 Phase inversion of PATCH_0143 is Completed in 2906 mins 15 secs [==================================================] 40000/40000 pixels, patch 142 Phase inversion of PATCH_0142 is Completed in 3135 mins 05 secs [==================================================] 40000/40000 pixels, patch 134 Phase inversion of PATCH_0134 is Completed in 3518 mins 49 secs [==================================================] 40000/40000 pixels, patch 132 Phase inversion of PATCH_0132 is Completed in 3721 mins 16 secs [==================================================] 40000/40000 pixels, patch 131 Phase inversion of PATCH_0131 is Completed in 4114 mins 37 secs [==================================================] 40000/40000 pixels, patch 133 Phase inversion of PATCH_0133 is Completed in 4456 mins 39 secs [==================================================] 40000/40000 pixels, patch 130 Phase inversion of PATCH_0130 is Completed in 4690 mins 04 secs [==================================================] 40000/40000 pixels, patch 129 Phase inversion of PATCH_0129 is Completed in 4719 mins 57 secs [==================================================] 40000/40000 pixels, patch 128 Phase inversion of PATCH_0128 is Completed in 4720 mins 15 secs Using default MintPy Path: /home/seis/anaconda3/envs/insar/lib/python3.10/site-packages [=================> 35% ] 14000/40000 pixels, patch 171 0s / 0s Phase inversion of PATCH_0161 is Completed in -1 mins 12 secs [======================= 91% =================> ] 36500/40000 pixels, patch 167 0s / 0s Phase inversion of PATCH_0166 is Completed in -1 mins 13 secs
[==================================================] 40000/40000 pixels, patch 172 Phase inversion of PATCH_0172 is Completed in 333 mins 39 secs [==================================================] 40000/40000 pixels, patch 173 Phase inversion of PATCH_0173 is Completed in 2518 mins 14 secs [==================================================] 40000/40000 pixels, patch 191 Phase inversion of PATCH_0191 is Completed in 4533 mins 06 secs [==================================================] 40000/40000 pixels, patch 190 Phase inversion of PATCH_0190 is Completed in 4558 mins 10 secs [==================================================] 40000/40000 pixels, patch 189 Phase inversion of PATCH_0189 is Completed in 5258 mins 04 secs [==================================================] 40000/40000 pixels, patch 187 Phase inversion of PATCH_0187 is Completed in 5565 mins 12 secs [==================================================] 40000/40000 pixels, patch 185 Phase inversion of PATCH_0185 is Completed in 5665 mins 23 secs [======================= 57% ] 23000/40000 pixels, patch 181354929s / 267753s^C |
this was after I had downgraded to the versions listed above |
|
You can try installing the following version. |
Thanks for your suggestion. I will install these versions and retry phase linking. "When entering phase linking, the command line gives an estimate of the processing time and compares it to see if the speed has improved." I am not familiar with this command line output? The example that I gave above is the direct command line print out. |
There has been no improvement when using the versions listed above. Unfortunately, these extremely long processing times make the work I want to do untenable. I will have to look at other options. |
Thanks again for your help |
Hi again. I have run phase-linking over a region of 400,000 pixels (10 200x200 patches) and I varied the number of SLC images in the stack to see how strongly this affects the run time of phase_linking. The SLCs are full resolution. Using 1-year of data (31 SLC images), it took 11 minutes to complete phase linking, with 2-year of data (53 SLC images) it took 4.5 hours, and with 3-year (77 SLC images) it is estimating ~25 hours. Does this seem correct to you? It seems that the depth of the SLC is the most important factor for phase_linking computation time? I have another query regarding phase_linking computation time: I am wondering how the temporal/spatial coherence of the wrapped phase contributes to phase_linking computation time - for areas that are very coherent both spatially and temporally, would phase_linking be significantly quicker than for areas that are very noisy and contain low signal to noise ratio @mirzaees ? Thanks for any help |
Hi @Jesse-kearse |
Hi @mirzaees |
Also is it practical to do a the processing in two parts. For example, Would this be possible/practical? and do the processes of "phase_linking" require ISCE2 to be installed? |
@Jesse-kearse yes of course you can transfer your data after loading step |
I am performing analysis using miaplpy.
Attempting to run full-resolution without applying Multi-looking in ISCE2.
The configuration used is as follows.
If there is a trade-off between accuracy and processing time, what option can I modify to reduce the time?
Are there any options other than reducing the resolution?
Thank you.
The text was updated successfully, but these errors were encountered: