Study Model for Information Reconstruction on Cloud Contaminated Area for Single Multispectral Remote Sensing Sentinel-2 Imagery using Generative Adversarial Network
Main Article Content
Abstract
Cloud and cloud shadow cause information loss in optical remote sensing analysis. South East Asia, especially Vietnam, Sentinel-2 imagery has short re-visit cycle and observations tend to be contaminated with cloud and cloud shadow. Traditional cloud removal methods require close date multi-temporal data to avoid seasonal land cover changes. In this study, a method of integrating Deep Convolutional Neural Networks (DCNN) and Generative Adversarial Network (GAN) was proposed. This machine learning model estimates the information loss over cloud contaminated areas on a single Sentinel-2 image. The results show that for images with cloud cover rate under 25%, our model can reconstruct cloudless images with PSNR (25 – 40 dB) and SSIM (0.86 – 0.93) compared to real clear images. On the other hand, with cloud cover rate up to 40%, the model performance will be affected heavily by the distribution of cloud and cloud shadow areas. By investigating DCNN and GAN, our method has proven to be an effective tool to remove cloudy images with low and medium rates, which enriches the clear optical remote sensing data sources for environment monitoring.
Keywords:
Cloud and cloud shadow, Remote sensing, Sentinel-2, Generative adversarial network, Single image cloud removal.
References
[1] N. Q. Long, T. V. Anh, B. K. Luyen, Determination of Ground Subsidence By Sentinel-1 Sar Data (2018-2020) over Binh Duong Quarries, Vietnam, Vnu Journal of Science: Earth and Environmental Sciences, Vol. 37, No. 2, 2021, pp. 69-83, https://doi.org/10.25073/2588-1094/vnuees.4605
(in Vietnamese).
[2] C. H. Lin, P. H. Tsai, K. H. Lai, J. Y. Chen, Cloud Removal from Multitemporal Satellite Images Using Information Cloning, in IEEE Transactions on Geoscience and Remote Sensing, Vol. 51,
No. 1, 2013, pp. 232-241, https://doi.org/10.1109/tgrs.2012.2197682.
[3] S. Huanfeng, L. H. Fang, Q. Yan., Z. L. Pei, Y. Q. Qiang, An Effective Thin Cloud Removal Procedure for Visible Remote Sensing Images, Isprs Journal of Photogrammetry and Remote Sensing, Vol. 96, 2012, pp. 224-235, https://doi.org/10.1016/j.isprsjprs.2014.06.011.
[4] N. Q. Huy, P. V. Manh, N. X. Linh et al., Building A Travel Advisory System Using Webgis Technology for Tourism Development in Quang Binh, Vietnam, Vnu Journal of Science: Earth and Environmental Sciences, Vol 35, No. 3, 2019,
pp. 108-114,
https://doi.org/10.25073/2588-1094/vnuees.4399 (in Vietnamese).
[5] P. V. Dong, B. Q. Thanh, N. Q. Huy et al., Application of Deep Learning Algorithm to Build an Automated Cloud Segmentation Model Based on Open Data Cube Framework, VNU Journal of Science: Earth and Environmental Sciences,
Vol. 36, No. 4, 2020, pp. 8-16, https://doi.org/10.25073/2588-1094/vnuees.4441 (in Vietnameses).
[6] O. Hagolle, M. Huc, D. V. Pascual, G. Dedieu, A Multi-temporal Method for Cloud Detection, Applied to FORMOSAT-2, VENUS, LANDSAT and SENTINEL-2 Images, Remote Sensing of Environment, Vol. 114, No. 8, 2010, pp. 1747-1755, https://doi.org/10.1016/j.rse.2010.03.002.
[7] Z. Zhe, W. Shixiong, W. E. Curtis, Improvement and Expansion of the Fmask Algorithm: Cloud, Cloud Shadow, and Snow Detection for Landsats 4–7, 8, and Sentinel 2 Images, Remote Sensing of Environment, Vol. 159, 2015, pp. 269-277, https://doi.org/10.1016/j.rse.2014.12.014.
[8] T. A. Tuan, N. D. Duong, Development of Landsat Cloud Free Image Data for Classification of Land Cover-case Study in Dak Lak Province, VNU Journal of Science: Earth and Environmental Sciences, Vol. 35, No. 4, 2019, pp. 80-87, https://doi.org/10.25073/2588-1094/vnuees.4447.
[9] Q. Cheng, H. Shen, L. Zhang et al., Cloud Removal for Remotely Sensed Images by Similar Pixel Replacement Guided with A Spatio-Temporal MRF Model, ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 92, 2014, pp. 54-68, https://doi.org/10.1016/j.isprsjprs.2014.02.015.
[10] X. Zhang, F. Qin, Y. Qin, Study on the Thick Cloud Removal Method Based on Multi-temporal Remote Sensing Images, International Conference on Multimedia Technology, 2010, pp. 1-3, https://doi.org/10.1109/ICMULT.2010.5631017.
[11] C. Yang, X. Lu, Z. Lin et al., High-Resolution Image Inpainting Using Multi-Scale Neural Patch Synthesis, 2017, pp. 6721-6729,
https://doi.org/10.1109/CVPR.2017.434.
[12] X. Zhu, Y. Qian, X. Zhao et al., A deep Learning Approach to Patch-based Image Inpainting Forensics, Signal Processing: Image Communication, Vol. 67, 2018, pp. 90-99, https://doi.org/10.1016/j.image.2018.05.015.
[13] J. Yu, Z. Lin, J. Yang et al., Generative Image Inpainting with Contextual Attention, 2018,
pp. 5505-5514, https://doi.org/10.48550/arXiv.1801.07892.
[14] G. Liu, F. A. Reda, K. J. Shih et al., Image Inpainting for Irregular Holes Using Partial Convolutions, 2018, pp. 1-15, https://doi.org/10.48550/arXiv.1804.07723.
[15] H. Shen, X. Li, Q. Cheng et al, Missing Information Reconstruction of Remote Sensing Data: A Technical Review, IEEE Geosci Remote Se.ns Mag, Vol 3, No. 3, 2015, pp. 61-85, https://doi.org/10.1109/MGRS.2015.2441912.
[16] H. Shen, J. Wu, Q. Cheng et al., A Spatiotemporal Fusion Based Cloud Removal Method for Remote Sensing Images with Land Cover Changes, IEEE J Sel Top Appl Earth Observations Remote Sensing,
Vol. 12, No. 3, 2019, pp. 862-874,
https://doi.org/ 10.1109/JSTARS.2019.2898348.
[17] A. M. Martínez, E. I. Verdiguier, M. P. Maneta et al., Multispectral High Resolution Sensor Fusion for Smoothing and Gap-filling in the Cloud, Remote Sensing of Environment, Vol 247, 2020, pp. 1-19, https://doi.org/10.1016/j.rse.2020.111901.
[18] X. Zhu, Y. Qian, X. Zhao et al., A Deep Learning Approach to Patch-Based Image Inpainting Forensics, Signal Processing: Image Communication, Vol. 67, 2018, pp. 90-99, https://doi.org/10.1016/j.image.2018.05.015.
[19] I. J. Goodfellow, J. P. Abadie, M. Mirza et al, Generative Adversarial Networks, arXiv: 14062661, 2014, pp. 1-9, https://doi.org/10.48550/arXiv.1406.2661.
[20] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:150504597, 2015, pp. 1-8, https://doi.org/10.48550/arXiv.1505.04597.
[21] J. Korhonen, J. You, Peak Signal-to-noise Ratio Revisited: is Simple Beautiful?, 2012 Fourth International Workshop on Quality of Multimedia Experience, 2012, pp. 37-38,
https://doi.org/10.1109/QoMEX.2012.6263880.
[22] L. Zhang, L. Zhang, X. Mou, D. Zhang, FSIM: A Feature Similarity Index for Image Quality Assessment, in IEEE Transactions on Image Processing, Vol. 20, No. 8, 2011, pp. 2378-2386, https://doi.org/10.1109/TIP.2011.2109730.
(in Vietnamese).
[2] C. H. Lin, P. H. Tsai, K. H. Lai, J. Y. Chen, Cloud Removal from Multitemporal Satellite Images Using Information Cloning, in IEEE Transactions on Geoscience and Remote Sensing, Vol. 51,
No. 1, 2013, pp. 232-241, https://doi.org/10.1109/tgrs.2012.2197682.
[3] S. Huanfeng, L. H. Fang, Q. Yan., Z. L. Pei, Y. Q. Qiang, An Effective Thin Cloud Removal Procedure for Visible Remote Sensing Images, Isprs Journal of Photogrammetry and Remote Sensing, Vol. 96, 2012, pp. 224-235, https://doi.org/10.1016/j.isprsjprs.2014.06.011.
[4] N. Q. Huy, P. V. Manh, N. X. Linh et al., Building A Travel Advisory System Using Webgis Technology for Tourism Development in Quang Binh, Vietnam, Vnu Journal of Science: Earth and Environmental Sciences, Vol 35, No. 3, 2019,
pp. 108-114,
https://doi.org/10.25073/2588-1094/vnuees.4399 (in Vietnamese).
[5] P. V. Dong, B. Q. Thanh, N. Q. Huy et al., Application of Deep Learning Algorithm to Build an Automated Cloud Segmentation Model Based on Open Data Cube Framework, VNU Journal of Science: Earth and Environmental Sciences,
Vol. 36, No. 4, 2020, pp. 8-16, https://doi.org/10.25073/2588-1094/vnuees.4441 (in Vietnameses).
[6] O. Hagolle, M. Huc, D. V. Pascual, G. Dedieu, A Multi-temporal Method for Cloud Detection, Applied to FORMOSAT-2, VENUS, LANDSAT and SENTINEL-2 Images, Remote Sensing of Environment, Vol. 114, No. 8, 2010, pp. 1747-1755, https://doi.org/10.1016/j.rse.2010.03.002.
[7] Z. Zhe, W. Shixiong, W. E. Curtis, Improvement and Expansion of the Fmask Algorithm: Cloud, Cloud Shadow, and Snow Detection for Landsats 4–7, 8, and Sentinel 2 Images, Remote Sensing of Environment, Vol. 159, 2015, pp. 269-277, https://doi.org/10.1016/j.rse.2014.12.014.
[8] T. A. Tuan, N. D. Duong, Development of Landsat Cloud Free Image Data for Classification of Land Cover-case Study in Dak Lak Province, VNU Journal of Science: Earth and Environmental Sciences, Vol. 35, No. 4, 2019, pp. 80-87, https://doi.org/10.25073/2588-1094/vnuees.4447.
[9] Q. Cheng, H. Shen, L. Zhang et al., Cloud Removal for Remotely Sensed Images by Similar Pixel Replacement Guided with A Spatio-Temporal MRF Model, ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 92, 2014, pp. 54-68, https://doi.org/10.1016/j.isprsjprs.2014.02.015.
[10] X. Zhang, F. Qin, Y. Qin, Study on the Thick Cloud Removal Method Based on Multi-temporal Remote Sensing Images, International Conference on Multimedia Technology, 2010, pp. 1-3, https://doi.org/10.1109/ICMULT.2010.5631017.
[11] C. Yang, X. Lu, Z. Lin et al., High-Resolution Image Inpainting Using Multi-Scale Neural Patch Synthesis, 2017, pp. 6721-6729,
https://doi.org/10.1109/CVPR.2017.434.
[12] X. Zhu, Y. Qian, X. Zhao et al., A deep Learning Approach to Patch-based Image Inpainting Forensics, Signal Processing: Image Communication, Vol. 67, 2018, pp. 90-99, https://doi.org/10.1016/j.image.2018.05.015.
[13] J. Yu, Z. Lin, J. Yang et al., Generative Image Inpainting with Contextual Attention, 2018,
pp. 5505-5514, https://doi.org/10.48550/arXiv.1801.07892.
[14] G. Liu, F. A. Reda, K. J. Shih et al., Image Inpainting for Irregular Holes Using Partial Convolutions, 2018, pp. 1-15, https://doi.org/10.48550/arXiv.1804.07723.
[15] H. Shen, X. Li, Q. Cheng et al, Missing Information Reconstruction of Remote Sensing Data: A Technical Review, IEEE Geosci Remote Se.ns Mag, Vol 3, No. 3, 2015, pp. 61-85, https://doi.org/10.1109/MGRS.2015.2441912.
[16] H. Shen, J. Wu, Q. Cheng et al., A Spatiotemporal Fusion Based Cloud Removal Method for Remote Sensing Images with Land Cover Changes, IEEE J Sel Top Appl Earth Observations Remote Sensing,
Vol. 12, No. 3, 2019, pp. 862-874,
https://doi.org/ 10.1109/JSTARS.2019.2898348.
[17] A. M. Martínez, E. I. Verdiguier, M. P. Maneta et al., Multispectral High Resolution Sensor Fusion for Smoothing and Gap-filling in the Cloud, Remote Sensing of Environment, Vol 247, 2020, pp. 1-19, https://doi.org/10.1016/j.rse.2020.111901.
[18] X. Zhu, Y. Qian, X. Zhao et al., A Deep Learning Approach to Patch-Based Image Inpainting Forensics, Signal Processing: Image Communication, Vol. 67, 2018, pp. 90-99, https://doi.org/10.1016/j.image.2018.05.015.
[19] I. J. Goodfellow, J. P. Abadie, M. Mirza et al, Generative Adversarial Networks, arXiv: 14062661, 2014, pp. 1-9, https://doi.org/10.48550/arXiv.1406.2661.
[20] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:150504597, 2015, pp. 1-8, https://doi.org/10.48550/arXiv.1505.04597.
[21] J. Korhonen, J. You, Peak Signal-to-noise Ratio Revisited: is Simple Beautiful?, 2012 Fourth International Workshop on Quality of Multimedia Experience, 2012, pp. 37-38,
https://doi.org/10.1109/QoMEX.2012.6263880.
[22] L. Zhang, L. Zhang, X. Mou, D. Zhang, FSIM: A Feature Similarity Index for Image Quality Assessment, in IEEE Transactions on Image Processing, Vol. 20, No. 8, 2011, pp. 2378-2386, https://doi.org/10.1109/TIP.2011.2109730.