Segment-then-Classify: Few-shot instance segmentation for environmental remote sensing (Papers Track)
Yang Hu (University of California, Santa Barbara); Kelly Caylor (UCSB); Anna S Boser (UCSB)
Abstract
Instance segmentation is pivotal for environmental sciences and climate change research, facilitating important tasks from land cover classification to glacier monitoring. This paper addresses the prevailing challenges associated with data scarcity when using traditional models like YOLOv8 by introducing a novel, data-efficient workflow for instance segmentation. The proposed Segment-then-Classify (STC) strategy leverages the zero-shot capabilities of the novel Segment Anything Model (SAM) to segment all objects in an image and then uses a simple classifier such as the Vision Transformer (ViT) to identify objects of interest thereafter. Evaluated on the VHR-10 dataset, our approach demonstrated convergence with merely 40 examples per class. YOLOv8 requires 3 times as much data to achieve the STC's peak performance. The highest performing class in the VHR-10 dataset achieved a near-perfect mAP@0.5 of 0.99 using the STC strategy. However, performance varied greatly across other classes due to the SAM model’s occasional inability to recognize all relevant objects, indicating a need for refining the zero-shot segmentation step. The STC workflow therefore holds promise for advancing few-shot learning for instance segmentation in environmental science.