Mastering Picture and Video Segmentation with SAM 2

Introduction

This information will stroll you thru what  Phase Something Mannequin 2  is, the way it works, and the way you’ll put it to use to portion objects in footage and movies. It affords state-of-the-art execution and flexibility in fragmenting objects into footage, making it an essential useful resource for a assortment of pc imaginative and prescient functions. This straight factors to supplying a nitty-gritty, step-by-step walkthrough for establishing and using SAM 2 to carry out image division. By taking this direct, it is possible for you to to supply division covers for footage using each field and level prompts.

Studying Goals

• Describe the important thing options and functions of the Phase Something Mannequin 2 SAM 2 in picture and video segmentation.
• Efficiently configure a CUDA-enabled setting, set up needed dependencies, and clone the Phase Something Mannequin 2  repository for picture segmentation duties.
• Apply SAM 2 to generate segmentation masks for photographs utilizing each field and level prompts and visualize the outcomes successfully.
• Consider how SAM 2 can revolutionize picture and video enhancing by enabling real-time segmentation, automating advanced duties, and democratizing content material creation for a broader viewers.

This text was printed as part of the Knowledge Science Blogathon.

Stipulations

A while just lately you start, assure you’ve received a CUDA-enabled GPU for faster dealing with. Additionally, confirm that you’ve Python put in in your machine. This information assumes you may have some primary information of Python and picture processing ideas.

What’s SAM 2?

 Phase Something Mannequin 2  is an progressed instrument for image division created by Fb AI Inquire about (Cheap). On July 29th, 2024, Meta AI discharged SAM 2, an progressed image and video division institution present. SAM 2 empowers purchasers to produce focuses or containers in an image or video to create division covers for specific objects.

Click on right here to entry it.

Key Options of SAM 2

• Superior Masks Technology: SAM 2 generates high-quality segmentation masks based mostly on person inputs, reminiscent of factors or bounding containers.
• Flexibility: The mannequin helps each picture and video segmentation.
• Pace and Effectivity: With CUDA help, SAM 2 can carry out segmentation duties quickly, making it appropriate for real-time functions.

Core Parts of SAM 2

• Picture Encoder: Encodes the enter picture for processing.
• Immediate Encoder: Converts user-provided factors or containers right into a format the mannequin can use.
• Masks Decoder: Generates the ultimate segmentation masks based mostly on the encoded inputs.

Functions of SAM 2

Allow us to now look into the functions of SAM 2 beneath:

• Photograph and Video Enhancing: SAM 2 permits for exact object segmentation, enabling detailed edits and inventive results in photographs and movies.
• Autonomous Automobiles: In autonomous driving, SAM 2 can be utilized to determine and monitor objects like pedestrians, autos, and street indicators in real-time.
• Medical Imaging: SAM 2 can help in segmenting anatomical buildings in medical photographs, aiding in diagnostics and therapy planning.

What’s Picture Segmentation?

Picture segmentation is a pc imaginative and prescient method that entails dividing a picture into a number of segments or areas to simplify its evaluation. Every phase represents a distinct object or a part of an object throughout the picture, making it simpler to determine and analyze particular components.

Forms of Picture Segmentation

• Semantic Segmentation: Classifies every pixel right into a predefined class.
• Occasion Segmentation: Differentiates between completely different cases of the identical object class.
• Panoptic Segmentation: Combines semantic and occasion segmentation.

Setting Up and Using SAM 2 for Picture Segmentation

We’ll information you thru the method of establishing the Phase Something Mannequin 2 (SAM 2) in your setting and using its highly effective capabilities for exact picture segmentation duties. From making certain your GPU is able to configuring the mannequin and making use of it to actual photographs, every step will likely be coated intimately that can assist you harness the total potential of SAM 2.

Step 1: Verify GPU Availability and Set Up the Atmosphere

First, let’s be certain that your setting is correctly arrange, beginning with checking for GPU availability and setting the present working listing.

# Verify GPU availability and CUDA model
!nvidia-smi
!nvcc --version # Import needed modules
import os # Set the present working listing
HOME = os.getcwd()
print("HOME:", HOME)

Clarification

• !nvidia-smi and !nvcc –model: These instructions examine in case your framework incorporates a CUDA-enabled GPU and present the CUDA kind.
• os.getcwd(): This work will get the present working catalog, which may be utilized for overseeing report methods.

Step 2: Clone the SAM 2 Repository and Set up Dependencies

Subsequent, we have to clone the SAM 2 repository from GitHub and set up the required dependencies.

# Clone the SAM 2 repository
!git clone https://github.com/facebookresearch/segment-anything-2.git # Change to the repository listing
%cd segment-anything-2 # Set up the SAM 2 bundle
!pip set up -e . # Set up extra packages
!pip set up supervision jupyter_bbox_widget

Clarification

• !git clone: Clones the SAM 2 repository to your native machine.
• %cd: Adjustments the listing to the cloned repository.
• !pip set up -e .: Installs the SAM 2 bundle in editable mode.
• !pip set up supervision jupyter_bbox_widget: Installs extra packages required for visualization and bounding field widget help.

Step 3: Obtain Mannequin Checkpoints

Mannequin checkpoints are important, as they comprise the educated parameters of SAM 2. We are going to obtain a number of checkpoints for various mannequin sizes.

# Create a listing for checkpoints
!mkdir -p checkpoints # Obtain the mannequin checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_tiny.pt -P checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_small.pt -P checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_base_plus.pt -P checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt -P checkpoints

Clarification

• !mkdir -p checkpoints: Creates a listing for storing mannequin checkpoints.
• !wget -q … -P checkpoints: Downloads the mannequin checkpoints into the checkpoints listing. Totally different checkpoints signify fashions of various sizes and capabilities.

Step 4: Obtain Pattern Photos

For demonstration functions, we’ll use some pattern photographs. You may also use your photographs by following related steps.

# Create a listing for knowledge
!mkdir -p knowledge # Obtain pattern photographs
!wget -q https://media.roboflow.com/notebooks/examples/canine.jpeg -P knowledge
!wget -q https://media.roboflow.com/notebooks/examples/dog-2.jpeg -P knowledge
!wget -q https://media.roboflow.com/notebooks/examples/dog-3.jpeg -P knowledge
!wget -q https://media.roboflow.com/notebooks/examples/dog-4.jpeg -P knowledge

Clarification

• !mkdir -p knowledge: Creates a listing for storing pattern photographs.
• !wget -q … -P knowledge: Downloads the pattern photographs into the info listing.

Step 5: Set Up the SAM 2 Mannequin and Load an Picture

Now, we’ll arrange the SAM 2 mannequin, load a picture, and put together it for segmentation.

import cv2
import torch
import numpy as np
import supervision as sv from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator # Allow CUDA if obtainable
torch.autocast(device_type="cuda", dtype=torch.bfloat16).__enter__() if torch.cuda.get_device_properties(0).main >= 8: torch.backends.cuda.matmul.allow_tf32 = True torch.backends.cudnn.allow_tf32 = True # Set the system to CUDA
DEVICE = torch.system('cuda' if torch.cuda.is_available() else 'cpu') # Outline the mannequin checkpoint and configuration
CHECKPOINT = "checkpoints/sam2_hiera_large.pt"
CONFIG = "sam2_hiera_l.yaml" # Construct the SAM 2 mannequin
sam2_model = build_sam2(CONFIG, CHECKPOINT, system=DEVICE, apply_postprocessing=False) # Create the automated masks generator
mask_generator = SAM2AutomaticMaskGenerator(sam2_model) # Load a picture for segmentation
IMAGE_PATH = "/content material/WhatsApp Picture 2024-08-02 at 14.17.11_2b223e01.jpg"
image_bgr = cv2.imread(IMAGE_PATH)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB) # Generate segmentation masks
sam2_result = mask_generator.generate(image_rgb)

Clarification

• CUDA Setup: Allows CUDA for quicker processing and units the system to GPU if obtainable.
• Mannequin Setup: Builds the SAM 2 mannequin utilizing the required configuration and checkpoint.
• Picture Loading: Masses and converts the pattern picture to RGB format.
• Masks Technology: Makes use of the automated masks generator to generate segmentation masks for the loaded picture.

Step 6: Visualize the Segmentation Masks

We are going to now visualize the segmentation masks generated by SAM 2.

# Annotate the masks on the picture
mask_annotator = sv.MaskAnnotator(color_lookup=sv.ColorLookup.INDEX)
detections = sv.Detections.from_sam(sam_result=sam2_result)
annotated_image = mask_annotator.annotate(scene=image_bgr.copy(), detections=detections) # Plot the unique and segmented photographs aspect by aspect
sv.plot_images_grid( photographs=[image_bgr, annotated_image], grid_size=(1, 2), titles=['source image', 'segmented image']
)

# Extract and plot particular person masks
masks = [ mask['segmentation'] for masks in sorted(sam2_result, key=lambda x: x['area'], reverse=True)
] sv.plot_images_grid( photographs=masks[:16], grid_size=(4, 4), measurement=(12, 12)
)

Clarification:

• Masks Annotation: Annotates the segmentation masks on the unique picture.
• Visualization: Plots the unique and segmented photographs aspect by aspect and likewise plots particular person masks.

Step7: Use Field Prompts for Segmentation

Field prompts permit us to specify areas of curiosity within the picture for segmentation.

# Outline the SAM 2 Picture Predictor
predictor = SAM2ImagePredictor(sam2_model) # Reload the picture
image_bgr = cv2.imread(IMAGE_PATH)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB) # Encode the picture for bounding field enter
import base64 def encode_image(filepath): with open(filepath, 'rb') as f: image_bytes = f.learn() encoded = str(base64.b64encode(image_bytes), 'utf-8') return "knowledge:picture/jpg;base64,"+encoded # Allow customized widget supervisor in Colab
IS_COLAB = True if IS_COLAB: from google.colab import output output.enable_custom_widget_manager() from jupyter_bbox_widget import BBoxWidget # Create a bounding field widget
widget = BBoxWidget()
widget.picture = encode_image(IMAGE_PATH) # Show the widget
widget

Clarification

• Picture Predictor: Defines the SAM 2 picture predictor.
• Picture Encoding: Encodes the picture to be used with the bounding field widget.
• Widget Setup: Units up a bounding field widget for specifying areas of curiosity.

Step8: Get Bounding Containers and Carry out Segmentation

After specifying the bounding containers, we are able to use them to generate segmentation masks.

# Get the bounding containers from the widget
containers = widget.bboxes
containers = np.array([ [ box['x'], field['y'], field['x'] + field['width'], field['y'] + field['height'] ] for field in containers
])

[{'x': 457, 'y': 341, 'width': 0, 'height': 0, 'label': ''}, {'x': 205, 'y': 79, 'width': 0, 'height': 1, 'label': ''}]

# Set the picture within the predictor
predictor.set_image(image_rgb) # Generate masks utilizing the bounding containers
masks, scores, logits = predictor.predict( field=containers, multimask_output=False
) # Convert masks to binary format
masks = np.squeeze(masks) # Annotate and visualize the masks
box_annotator = sv.BoxAnnotator(shade=sv.Colour.white())
mask_annotator = sv.MaskAnnotator(color_lookup=sv.ColorLookup.INDEX) detections = sv.Detections( xyxy=containers, masks=masks.astype(bool)
) source_image = box_annotator.annotate(scene=image_bgr.copy(), detections=detections)
segmented_image = mask_annotator.annotate(scene=image_bgr.copy(), detections=detections) # Plot the annotated photographs
sv.plot_images_grid( photographs=[source_image, segmented_image], grid_size=(1, 2), titles=['source image', 'segmented image']
)

Clarification

• Bounding Containers: Retrieves the bounding containers specified utilizing the widget.
• Masks Technology: Makes use of the bounding containers to generate segmentation masks.
• Visualization: Annotates and visualizes the masks on the unique picture.

Step9: Use Level Prompts for Segmentation

Level prompts permit us to specify particular person factors of curiosity for segmentation.

# Create level prompts based mostly on bounding containers
input_point = np.array([ [ box['x'] + (field['width'] // 2), field['y'] + (field['height'] // 2) ] for field in widget.bboxes
])
input_label = np.array([1] * len(input_point)) # Generate masks utilizing the purpose prompts
masks, scores, logits = predictor.predict( point_coords=input_point, point_labels=input_label, multimask_output=True
) # Convert masks to binary format
masks = np.squeeze(masks) # Annotate and visualize the masks
point_annotator = sv.PointAnnotator(color_lookup=sv.ColorLookup.INDEX)
mask_annotator = sv.MaskAnnotator(color_lookup=sv.ColorLookup.INDEX) detections = sv.Detections( xyxy=sv.mask_to_xyxy(masks=masks), masks=masks.astype(bool)
) source_image = point_annotator.annotate(scene=image_bgr.copy(), detections=detections)
segmented_image = mask_annotator.annotate(scene=image_bgr.copy(), detections=detections) # Plot the annotated photographs
sv.plot_images_grid( photographs=[source_image, segmented_image], grid_size=(1, 2), titles=['source image', 'segmented image']
)

Clarification

• Level Prompts: Creates level prompts based mostly on the bounding containers.
• Masks Technology: Makes use of the purpose prompts to generate segmentation masks.
• Visualization: Annotates and visualizes the masks on the unique picture.

Key Factors to Bear in mind When Working SAM 2

Allow us to now look into few essential key factors beneath:

Revolutionizing Photograph and Video Enhancing

• Potential to rework the picture and video enhancing business.
• Future enhancements could embrace improved precision, decrease computational necessities, and superior AI integration.

Actual-Time Segmentation and Enhancing

• Evolution might result in real-time segmentation and enhancing capabilities.
• Permits seamless alterations in movies and pictures with minimal effort.

Inventive Potentialities for All

• Opens up new inventive potentialities for each professionals and amateurs.
• Simplifies the manipulation of visible content material, the creation of beautiful results, and the manufacturing of high-quality media.

Automating Advanced Duties

• Automates intricate segmentation duties.
• Considerably accelerates workflows, making subtle enhancing extra accessible and environment friendly.

Democratizing Content material Creation

• Makes high-level enhancing instruments obtainable to a broader viewers.
• Empowers storytellers and evokes innovation throughout numerous sectors, together with leisure, promoting, and schooling.

Affect on VFX Trade

• Enhances visible results (VFX) manufacturing by streamlining advanced processes.
• Reduces the effort and time required for creating intricate VFX, enabling extra formidable tasks and bettering general high quality.

Spectacular Potential of SAM 2

The Phase Something Mannequin 2 (SAM 2) stands poised to revolutionize the fields of picture and video enhancing by introducing vital developments in precision and computational effectivity. By integrating superior AI capabilities, SAM 2 will allow extra intuitive person interactions and real-time segmentation and enhancing, permitting seamless alterations with minimal effort. This groundbreaking know-how guarantees to democratize content material creation, empowering each professionals and amateurs to govern visible content material, create beautiful results, and produce high-quality media with ease.

As SAM 2 automates advanced segmentation duties, it’ll speed up workflows and make subtle enhancing accessible to a wider viewers. This transformation will encourage innovation throughout numerous industries, from leisure and promoting to schooling. Within the realm of visible results (VFX), SAM 2 will streamline intricate processes, decreasing the effort and time wanted to create elaborate VFX. It will allow extra formidable tasks, elevate the standard of visible storytelling, and open up new inventive potentialities within the VFX world.

Conclusion

By following this information, you may have realized methods to arrange and use the Phase Something Mannequin 2 (SAM 2) for picture segmentation utilizing each field and level prompts. SAM 2 gives highly effective and versatile instruments for segmenting objects in photographs, making it a beneficial asset for numerous pc imaginative and prescient duties. Be at liberty to experiment along with your photographs and discover the capabilities of SAM 2 additional.

Key Takeaways

• SAM 2 is a complicated device developed by Meta AI that allows exact and versatile picture and video segmentation utilizing each field and level prompts.
• The mannequin can considerably improve picture and video enhancing by automating advanced segmentation duties, making it extra accessible and environment friendly.
• Organising SAM 2 requires a CUDA-enabled GPU and a primary understanding of Python and picture processing ideas.
• SAM 2’s capabilities open new potentialities for each professionals and amateurs in content material creation, providing real-time segmentation and inventive management.
• The mannequin has the potential to rework numerous industries, together with visible results, leisure, promoting, and schooling, by democratizing high-level enhancing instruments.

Continuously Requested Questions

The media proven on this article is just not owned by Analytics Vidhya and is used on the Creator’s discretion.