The best way to Construct a Picture Recollections App with CLIP

Many well-liked pictures purposes have options that collate pictures into slideshows, typically known as “recollections.” These slideshows are centered round a theme corresponding to a specific location, individual, or an idea widespread throughout your pictures.

Utilizing CLIP, a picture mannequin developed by OpenAI, we are able to construct a photograph recollections app that teams pictures in accordance with a specified theme. We are able to then collate the pictures retrieved by CLIP right into a video which you can share with family and friends.

Right here is the recollections slideshow we make in the course of the tutorial:

With out additional ado, let’s get began!

The best way to Construct a Picture Recollections App with CLIP

To construct our picture recollections app, we’ll:

1. Set up the required dependencies
2. Use CLIP to calculate embeddings for every picture in a folder
3. Use CLIP to search out associated photographs given a textual content question (i.e. “individuals” or “metropolis”)
4. Write logic to show associated photographs right into a video
5. Save the slideshow we have now generated

You might be questioning: “what are embeddings?” Embeddings are numeric representations of photographs, textual content, and different knowledge which you can evaluate. Embeddings are the important thing to this undertaking: we are able to evaluate textual content and picture embeddings to search out photographs associated to the themes for which we need to make recollections.

Step #1: Set up Required Dependencies

Earlier than we are able to begin constructing our app, we have to set up a number of dependencies. Run the next command to put in the Python packages we’ll use in our utility:

pip set up faiss-cpu opencv-python Pillow

(If you’re engaged on a pc with a CUDA-enabled GPU, set up faiss-gpu as an alternative of faiss-gpu)

With the required dependencies put in, we at the moment are prepared to start out constructing our recollections app.

Begin by importing the required dependencies for the undertaking:

import base64
import os
from io import BytesIO
import cv2
import faiss
import numpy as np
import requests
from PIL import Picture
import json

We’re going to use the Roboflow Inference Server for retrieving CLIP embeddings. You’ll be able to host the Inference Server your self, however for this information we’ll use the hosted model of the server.

Add the next fixed variables to your Python script, which we’ll use afterward to question the inference server.

INFERENCE_ENDPOINT = "https://infer.roboflow.com"
API_KEY = "API_KEY"

Change the `API_KEY` worth along with your Roboflow API key. Learn to discover your Roboflow API key.

Now, let’s begin engaged on the logic for our utility.

Step #2: Calculate Picture Embeddings

Our utility goes to take a folder of photographs and a textual content enter. We are going to then return a slideshow that accommodates photographs associated to the textual content enter. For this, we have to calculate two sorts of embeddings:

1. Picture embeddings for every picture, and;
2. A textual content embedding for the theme for a slideshow.

Let’s outline a perform that calls the Roboflow Inference Server and calculates a picture embedding:

def get_image_embedding(picture: str) -> dict: picture = picture.convert("RGB") buffer = BytesIO() picture.save(buffer, format="JPEG") picture = base64.b64encode(buffer.getvalue()).decode("utf-8") payload = { "physique": API_KEY, "picture": {"kind": "base64", "worth": picture}, } knowledge = requests.submit( INFERENCE_ENDPOINT + "/clip/embed_image?api_key=" + API_KEY, json=payload ) response = knowledge.json() embedding = response["embeddings"] return embedding

Subsequent, let’s outline one other perform that retrieves a textual content embedding for a question:

def get_text(immediate): text_prompt = requests.submit( f"{INFERENCE_ENDPOINT}/clip/embed_text?api_key={API_KEY}", json={"textual content": immediate} ).json()["embeddings"] return np.array(text_prompt)

Step #3: Create an Index

The 2 features we wrote within the earlier step each return embeddings. However we haven’t written the logic to make use of them but! Subsequent, we have to calculate picture embeddings for a folder of photographs. We are able to do that utilizing the next code:

index = faiss.IndexFlatL2(512)
image_frames = [] for body in os.listdir("./photographs"): body = Picture.open("./photographs/" + body) embedding = get_image_embedding(body) index.add(np.array(embedding)) image_frames.append(body) with open("image_frames.json", "w+"): json.dumps(image_frames) faiss.write_index(index, "index.bin")

This code creates an “index”. This index will retailer all of our embeddings. We are able to effectively search this index utilizing textual content embeddings to search out photographs for our slideshow.

On the finish of this code, we save the index to a file for later use. We additionally save all the picture body file names to a file. That is necessary as a result of the index doesn’t retailer these, and we have to know with what file every body within the index is related so we are able to make our slideshow.

Step #4: Retrieve Photos for the Slideshow

Subsequent, we have to retrieve photographs for our slideshow. We are able to do that with a single line of code:

question = get_text("san francisco")
D, I = index.search(question, 3)

Within the first line of code, we name the get_text() perform we outlined earlier to retrieve a textual content embedding for a question. On this instance, our question is “san francisco”. Then, we search our picture index for photographs whose embeddings are much like our textual content embedding.

This code will return photographs ordered by their relevance to the question. Should you don’t have any photographs related to the question, outcomes will nonetheless be returned, though they won’t be helpful in making a thematic slideshow. Thus, be sure to seek for themes you understand are featured in your photographs.

The three worth states we wish the highest three photographs related to our textual content question. You’ll be able to improve or lower this quantity to retrieve extra or fewer photographs on your slideshow.

Step #5: Discover Most Picture Width and Peak

There’s another step we have to full earlier than we are able to begin creating slideshows: we have to discover the most important picture width and top values within the photographs we’ll use to create every slideshow. It’s because we have to know at what decision we should always save our video.

To seek out the utmost width and top values within the frames we have now gathered, we are able to use the next code:

video_frames = []
largest_width = 0
largest_height = Zero for i in I[0]: body = image_frames[i] cv2_frame = np.array(body) cv2_frame = cv2.cvtColor(cv2_frame, cv2.COLOR_BGR2RGB) video_frames.prolong([cv2_frame] * 20) top, width, _ = cv2_frame.form if width > largest_width: largest_width = width if top > largest_height: largest_height = top

Step #6: Generate the Slideshow

We’re onto the ultimate step: create the slideshow. The entire items are in place. We now have discovered photographs associated to a textual content question, and calculated the decision we’ll use for our slideshow. The ultimate step is to create a video that makes use of the pictures.

We are able to create our slideshow utilizing the next code:

final_frames = [] for i, body in enumerate(video_frames): if body.form[0] < largest_height: distinction = largest_height - body.form[0] padding = distinction // 2 body = cv2.copyMakeBorder( body, padding, padding, 0, 0, cv2.BORDER_CONSTANT, worth=(0, 0, 0), ) if body.form[1] < largest_width: distinction = largest_width - body.form[1] padding = distinction // 2 body = cv2.copyMakeBorder( body, 0, 0, padding, padding, cv2.BORDER_CONSTANT, worth=(0, 0, 0), ) final_frames.append(body) video = cv2.VideoWriter( "video1.avi", cv2.VideoWriter_fourcc(*"MJPG"), 20, (largest_width, largest_height)
) for body in final_frames: video.write(body) cv2.destroyAllWindows()
video.launch()

This code creates an enormous checklist of all the frames we need to embody in our picture. These frames are padded with black pixels in accordance with the utmost top and width we recognized earlier. This ensures photographs will not be stretched to suit precisely the identical decision as the most important picture. We then add all of those frames to a video and save the outcomes to a file referred to as video.avi.

Let’s run our code on a folder of photographs. For this information, we have now run the recollections app on a sequence of metropolis pictures. Here’s what our video appears like:

We now have efficiently generated a video with photographs associated to “san francisco”.

Conclusion

CLIP is a flexible instrument with many makes use of in pc imaginative and prescient. On this information, we have now demonstrated find out how to construct a photograph recollections app with CLIP. We used CLIP to calculate picture embeddings for all photographs in a folder. We then saved these embeddings in an index.

Subsequent, we used CLIP to calculate a textual content embedding that we used to search out photographs associated to a textual content question. In our instance, this question was “san francisco”. Lastly, we accomplished some post-processing to make sure photographs had been all the identical measurement, and compiled photographs associated to our question right into a slideshow.