Vetnuus | October 2025 19 To effectively deploy GPT-4 and GPT-4o in veterinary ophthalmology, specific contextual adaptations are necessary. These models are configured to act as theoretical consultants, analysing visual data from images and predicting possible veterinary conditions. The setup demands a tailored prompt that provides the models with a context encapsulating the task of diagnosing based on visual cues alone, without supplementary background information. An exemplary prompt used in this scenario is demonstrated in Figure 13. [63], where the models’ output includes diagnostic suggestions and segmentation masks, crucial for practical applications [84]. The outcomes derived from this innovative application of GPT-4 and GPT-4o are directly compared against traditional methods, providing a benchmark for assessing the efficacy of integrating language models in image-based diagnostic tasks. This comparative analysis is essential for validating the models’ utility and effectiveness in real-world scenarios [85]. 3.2.3 Grounding SAM Object detection and segmentation models like YOLO [86] (You Only Look Once) and Mask R-CNN [87] (Region-Based Convolutional Neural Network) have shown remarkable performance in various applications, including medical image analysis. YOLO is known for its speed and efficiency, as it processes an image in a single pass to detect objects, making it highly suitable for real-time applications. However, both YOLO and Mask R-CNN require extensive training on customised datasets specific to the objects they are designed to detect or segment. This means that to achieve high accuracy in a particular domain, such as veterinary ophthalmology, these models must be fine-tuned using domain-specific data, which can be time-consuming and resource-intensive. In contrast, Grounding SAM, a powerful combination of Grounding DINO [88] and SAM [89], operates in a more versatile and adaptable manner. Instead of relying on pre-training for specific objects, Grounding SAM only requires a text prompt to detect and segment objects in an image. This prompt-driven approach eliminates the need for pre-training on customised datasets, making it ideal for applications where labelled data is scarce or difficult to obtain. Grounding SAM builds upon the foundational principles of transformer models and leverages zero-shot learning capabilities, meaning it can generalise across different tasks and images without the need for extensive retraining. This characteristic sets it apart from traditional models like YOLO and Mask R-CNN, which depend on task-specific training. Grounding DINO is the first step in the Grounding SAM framework, responsible for object detection. It operates by taking both images and text prompts as input. The transformer-based architecture of Grounding DINO enables it to process relationships between various objects in the image through its self-attention mechanism. This process allows the model to focus on relevant parts of the image, enhancing its ability to detect objects even in visually complex or noisy environments. The text prompts provided to Grounding DINO are essential for guiding the object detection process. For instance, in the case of diagnosing canine eye diseases, text prompts describing symptoms in layman’s terms (e.g., “extensive tearing”) are used, as medical terminology might not yield effective results due to the model’s limitations in domain-specific vocabulary. Once the objects have been detected by Grounding DINO, the bounding boxes are passed to SAM for instance segmentation. SAM’s task is to convert these bounding boxes into precise segmentation masks, which provide pixel-level accuracy for each detected object. SAM utilises a transformer-based image encoder that can handle a wide range of segmentation tasks with minimal human intervention, making it highly versatile for medical applications, where segmentation accuracy is paramount. In veterinary ophthalmology, SAM’s segmentation masks are particularly useful for delineating fine details of ocular symptoms, such as corneal cloudiness or scleral redness. This level of precision in image segmentation ensures that even subtle abnormalities can be detected and analysed, providing critical data for diagnostic purposes. SAM’s ability to generate accurate segmentation masks in a zero-shot manner—without the need for task-specific training—positions it as a powerful tool in the medical domain. The combined operation of Grounding DINO and SAM is depicted in Figure 14. [63], which illustrates the process of diagnosing canine ocular conditions. The input image, typically a clinical photograph of a dog’s eye, is first processed by Grounding DINO, which identifies areas of interest based on the provided text prompt. Grounding DINO generates bounding boxes around the detected areas, such as the eye or tear ducts, and these are passed to SAM for detailed segmentation. SAM then produces segmentation masks that outline the detected regions with pixel-level precision. The segmented regions, such as areas of corneal opacity or excessive tearing, are used as input for further analysis. A significant challenge in applying Grounding SAM to medical image analysis is prompt engineering, particularly when dealing with complex medical terminology. Grounding DINO and SAM requires prompts that can be interpreted accurately by the model. Medical terms such as “epiphora” or “keratitis” may not yield effective segmentation results due to their specialised nature. Leading Article Figure 13. An example of an input prompt to GPT-4 and GPT-4o requesting segmentation of medical conditions and the most probable diagnosis
RkJQdWJsaXNoZXIy OTc5MDU=