Focusing is difficult. In recent times, the quantity of distractions obtainable to us has been rising, and we frequently lose monitor of how a lot we’re distracted.
To assist myself keep engaged, I created a undertaking that precisely tracks what number of instances I am distracted in a sure time period.
On this information, you may learn to construct an app to trace your productiveness in addition to play Spotify music each time you might be distracted utilizing eye monitoring. Earlier than we start, I’ll first break down every thing it’s worthwhile to do at a better stage to raised perceive the undertaking.
To construct the total undertaking we’ll:
- Construct an eye fixed monitoring imaginative and prescient mannequin
- Obtain wanted instruments
- Configure logic and variables
- Arrange the attention monitoring boundaries and Spotify API
- Arrange graphing the code
Use this Colab pocket book to observe alongside and construct your personal productiveness monitoring utility.
Construct an Eye Monitoring Imaginative and prescient Mannequin
First, you may creat a Roboflow account to assist construct a mannequin that may precisely detect your eyes, pupil, cellphone, and face. Consult with this basic Roboflow tutorial video if any duties under are unclear.
- Signal as much as Roboflow and create a brand new workspace.
- Create courses by going to the add courses tab after which add the courses you want.
- Add pictures
To get prime quality pictures of your face, file a video in your laptop computer or cellphone and use a body generator to get pictures of your face. Add pictures or movies right here:
- Begin Annotating pictures
Subsequent go to the Annotate display so as to begin labeling information. Click on the annotate pictures button to get began.
- Annotate the pictures
Draw on the containers round your objects within the annotation display. On this case, I drew containers round my face, pupil, and eye.
Label round 100 pictures of your face. Moreover, have round 25 pictures of you along with your cellphone, and your mannequin ought to do comparatively properly.
Right here had been the statistics from my mannequin, accessible from Roboflow’s Well being Examine characteristic.
- Generate a dataset
After labeling, go to generate within the sidebar, and ensure so as to add a step to preprocessing.
Resize the picture to 2048×2048. By making the picture greater, this helps the mannequin detect the pupils extra precisely.
- Deploy your mannequin
After ending up the steps in producing and after coaching, it is best to see a display like this:
Your mannequin is now prepared. It’s time to put in writing code so as to play Spotify music while you get distracted, and graph your productiveness.
To start out, obtain the next modules by pip putting in them.
- Inference
- Roboflow
- Matplotlib
- Opencv
- Spotipy (Spotipy’s API software)
NOTE: Downloading Opencv could take a very long time (~1 hour) as a consequence of wheels. If there are any points, attempt upgrading setuptools and wheels.
!pip set up opencv-python inference matplotlib spotipy roboflowSubsequent, import the next libraries into a brand new Python file:
import cv2
from typing import Union, Checklist, Non-obligatory
from inference.core.interfaces.digicam.entities import VideoFrame
from inference import InferencePipeline
from datetime import datetime
import matplotlib.pyplot as plt
import spotipy
from spotipy.oauth2 import SpotifyOAuthOutline Variables
On this step, you’ll have to outline variables you want for the undertaking.
Beneath is a quick description of what every variable is used for.
API Key: Roboflow’s API Key’s wanted so as to use the mannequin we created.
Eye_Vals and Pupil_Vals: Shops the values of the 2 eyes and pupils in a dictionary. This dictionary comprises the next data for every eye and pupil:{ “label”: label,
“confidence”: confidence,
“start_point”: start_point,
“end_point”: end_point,
“x”: x,
“y”: y,
“width”: width,
“top”: top
}
Colours: For the totally different colours we’ll use within the mannequin (for the eyes, pupil, face and cellphone coloring)
Whole data: Shops all of the detected gadgets in a single body. Every merchandise is saved in a dictionary that has the next data:{ “label”: label,
“confidence”: confidence,
“start_point”: start_point,
“end_point”: end_point,
“x”: x,
“y”: y,
“width”: width,
“top”: top
}
Prev data: Shops all of the detected gadgets within the earlier body. Identical storing logic as data.
Prev_difference_left and Prev_difference_right: Set at 50, it shops the earlier distinction of how far the attention is from the pupil (to see should you’re wanting away from the display which implies you’re unproductive)
as soon as: Set to True, used to solely play spotify music as soon as
x_point: Shops the time at which you had been unproductive
y_point: Counts what number of instances you had been unproductive
This is similar format and logic for the opposite 4 variables.
api_key = "API_KEY" # Finds the place the pupil and eyes are
pupil_vals = []
eye_vals = [] # Colours
GREEN = (0, 255, 0)
RED = (0, 0, 255)
BLUE = (255, 0, 0)
PURPLE = (128, 0, 128)
ORANGE = (0, 165, 255)
YELLOW = (0, 255, 255) # Will get our eye monitoring values
total_information = []
prev_information = []
prev_difference_left = 50
prev_difference_right = 50
as soon as = True # For graphing
x_point = []
x_point_phone = []
x_point_eyes = []
y_point = [0]
y_point_phone = [0]
y_point_eyes = [0]
Add Draw Capabilities
# Draw features
def draw_box(x, y, width, top, picture, COLOR): start_point = (int(x - width / 2), int(y - top / 2)) end_point = (int(x + width / 2), int(y + top / 2)) cv2.rectangle(picture, start_point, end_point, COLOR, cv2.FILLED)
Within the code above, we write a operate that attracts containers on specified components of a picture.
The Colour scheme of every merchandise is:
Pupils are Purple
Eyes are Blue
Face is Inexperienced
Telephone is Pink
def get_specific_eyes(eye_vals, pupil_vals): if eye_vals[0]['x'] < eye_vals[1]['x']: right_eye = eye_vals[0] left_eye = eye_vals[1] if eye_vals[0]['x'] > eye_vals[1]['x']: right_eye = eye_vals[1] left_eye = eye_vals[0] if pupil_vals[0]['x'] < pupil_vals[1]['x']: right_pupil = pupil_vals[0] left_pupil = pupil_vals[1] if pupil_vals[0]['x'] > pupil_vals[1]['x']: right_pupil = pupil_vals[1] left_pupil = pupil_vals[0] return right_eye, left_eye, right_pupil, left_pupilThe get_specific_eyes operate takes within the two eyes and pupil values (which we outlined within the final step), and appears on the X axis of each. The X axis begins on the high left nook.
Because of this if the primary eye (eye_vals[0]) has a x coordinate lower than the second eye (eye_vals[1]), then the appropriate eye is eye_vals[0] and the left is eye_vals[1]. Nevertheless if the primary eye has a x coordinate higher than the second eye, then the appropriate eye is eye_vals[1] and the left is eye_vals[0].
This logic is mirrored onto the left and proper pupils.
Connect with Spotify API
If music helps you focus, I like to recommend using Spotify’s API instruments so as to add a reminder while you’re distracted. Nevertheless, you will want to have a premium Spotify subscription. If you don’t, you’ll be able to merely skip this step.
First, so as to get your shopper id and shopper secret keys, you’ll must create an app utilizing the “Getting began” part of these directions.
# Change along with your Spotify app credentials
client_id = 'Client_Id'
client_secret = 'Client_Secret'
redirect_uri = 'http://localhost:8888/callback' # Scope to regulate playback
scope = 'user-modify-playback-state user-read-playback-state' # Authenticate and get token
sp = spotipy.Spotify(auth_manager=SpotifyOAuth(client_id=client_id, client_secret=client_secret, redirect_uri=redirect_uri, scope=scope))
# Seek for a playlist
def search_playlist(identify): outcomes = sp.search(q=f'playlist:{identify}', kind='playlist', restrict=1) if outcomes['playlists']['items']: playlist = outcomes['playlists']['items'][0] print(f"Discovered playlist: {playlist['name']} with URI: {playlist['uri']}") return playlist['uri'] else: print("Playlist not discovered") return None
After that, create a search playlist operate that makes use of Spotify’s api to search out the playlist you need. The operate searches for the inputted playlist in an merchandise checklist. If it finds the playlist, it would print “Discovered playlist” together with the playlist identify.
To make use of it, enter a reputation into the operate like what I did right here (I selected classical since that is what makes me focus the most effective):
playlist_name = "Greatest Classical Music"
search_playlist(playlist_name)
Above, we create a operate referred to as play_playlist that can:
- discover the playlist – playlist_uri = search_playlist(playlist_name)
- discover a machine that has the spotify account – if playlist_uri: gadgets = sp.gadgets()
- Shuffle the playlist – sp.shuffle(True, device_id=device_id)
- Begin taking part in the track – sp.start_playback(device_id=device_id, context_uri=playlist_uri)
# Operate to play a monitor
def play_playlist(playlist_name): playlist_uri = search_playlist(playlist_name) if playlist_uri: gadgets = sp.gadgets() if gadgets['devices']: device_id = gadgets['devices'][0]['id'] # Use the primary obtainable machine sp.shuffle(True, device_id=device_id) sp.start_playback(device_id=device_id, context_uri=playlist_uri) print(f"Taking part in playlist on machine: {gadgets['devices'][0]['name']}") else: print("No lively gadgets discovered. Please open Spotify in your machine and take a look at once more.")After that, you’ll be able to check it by working the next code:
play_playlist("Greatest Classical Music")
Combining each features, it is best to have the ability to begin taking part in this playlist: (make sure that to love the playlist first if it is not working.)
If you wish to play your personal private playlist, substitute “Greatest classical Music” with the playlist of your alternative.
Run the Mannequin
On this step, we create the primary logic for every thing we did within the earlier 5 steps. The operate will:
- Draw out the predictions
- Present the picture
- Create the attention monitoring logic
- Use Spotify’s API to play music when one is distracted
- Add data to graphic logic
Right here is the primary code. Within the following code snippets, code is repeated so as to clarify every half clearly.
# Primary prediction operate
def on_prediction(
predictions: Union[dict, List[Optional[dict]]],
video_frame: Union[VideoFrame, List[Optional[VideoFrame]]],
) -> None:
if not isinstance(predictions, checklist):
predictions = [predictions]
video_frame = [video_frame]
world as soon as, start_time, prev_information, prev_difference_left, prev_difference_right
for prediction, body in zip(predictions, video_frame):
if prediction is None:
proceed # SOME PROCESSING
picture = body.picture
for obj in prediction['predictions']:
# Get bounding field coordinates
x = obj['x']
y = obj['y']
width = obj['width']
top = obj['height']
label = obj['class']
confidence = obj['confidence'] start_point = (int(x - width / 2), int(y - top / 2))
end_point = (int(x + width / 2), int(y + top / 2)) data = {
"label": label,
"confidence": confidence,
"start_point": start_point,
"end_point": end_point,
"x": x,
"y": y,
"width": width,
"top": top
} if label == "cellphone" and "cellphone" not in prev_information:
x_point.append(datetime.now()-start_time)
x_point_phone.append(datetime.now()-start_time)
y_point.append(y_point[len(y_point)-1]+1)
y_point_phone.append(y_point_phone[len(y_point_phone)-1]+1) if as soon as == True:
play_playlist("Greatest Classical Music")
as soon as = False
total_information.append(data)
# Put together label textual content
label_text = f"{label}: {confidence:.2f}" # Get textual content dimension
(text_width, text_height), baseline = cv2.getTextSize(label_text, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1) # Draw background rectangle for the label
cv2.rectangle(picture, (start_point[0], start_point[1] - text_height - baseline),
(start_point[0] + text_width, start_point[1]), (0, 255, 0), cv2.FILLED)
cv2.putText(picture, label_text, (start_point[0], start_point[1] - baseline),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 1)
for i in vary(len(total_information)):
if total_information[i]['label'] == 'cellphone':
draw(total_information[i]['x'], total_information[i]['y'], total_information[i]['width'], total_information[i]['height'], RED, picture)
elif total_information[i]['label'] == 'face':
draw(total_information[i]['x'], total_information[i]['y'], total_information[i]['width'], total_information[i]['height'], GREEN, picture)
elif total_information[i]['label'] == 'eye':
draw(total_information[i]['x'], total_information[i]['y'], total_information[i]['width'], total_information[i]['height'], BLUE, picture)
eye_vals.append(total_information[i])
for j in vary(len(total_information)):
if total_information[j]['label'] == 'pupil':
draw(total_information[j]['x'], total_information[j]['y'], total_information[j]['width'], total_information[j]['height'], PURPLE, picture)
pupil_vals.append(total_information[j]) if len(eye_vals) == 2 and len(pupil_vals) == 2:
right_eye, left_eye, right_pupil, left_pupil = get_specific_eyes(eye_vals, pupil_vals)
right_side_eye = right_eye["x"] - right_eye["width"]/2
left_side_eye = left_eye["x"] + left_eye["width"]/2
right_side_pupil = right_pupil["x"]
left_side_pupil = left_pupil["x"]
difference_right = right_side_pupil - right_side_eye
difference_left = left_side_eye - left_side_pupil if ((difference_left < 38) or (difference_right < 38)) and (prev_difference_left > 38 and prev_difference_right > 38):
x_point.append(datetime.now()-start_time)
x_point_eyes.append(datetime.now()-start_time)
y_point.append(y_point[len(y_point)-1]+1)
y_point_eyes.append(y_point_eyes[len(y_point_eyes)-1]+1)
if as soon as == True:
play_playlist("Greatest Classical Music")
as soon as = False prev_difference_right = difference_right
prev_difference_left = difference_left
if total_information:
prev_information.clear()
for information in total_information:
prev_information.append(data["label"])
else:
prev_information = []
total_information.clear()
eye_vals.clear()
pupil_vals.clear() cv2.imshow('body', picture) if (datetime.now()-start_time).total_seconds() > 120:
pipeline.terminate()
pipeline.be part of()
cv2.destroyAllWindows()
break
if cv2.waitKey(1) & 0xFF == ord('q'):
break
First, make sure that predictions are an inventory or dictionary object. Do the identical for video_frame. After, set each to lists.
def on_prediction(
predictions: Union[dict, List[Optional[dict]]],
video_frame: Union[VideoFrame, List[Optional[VideoFrame]]],
) -> None:
if not isinstance(predictions, checklist):
predictions = [predictions]
video_frame = [video_frame]Subsequent, set all of the wanted variables to be world so as to use them within the operate.
world as soon as, start_time, prev_information, prev_difference_left, prev_difference_rightBe sure to skip the logic if there are not any detections (this protects time to output a body).
for prediction, body in zip(predictions, video_frame): if prediction is None: proceedOn this code block, we get the picture by means of the body. Utilizing “picture = body.picture”
# SOME PROCESSING picture = body.picture for obj in prediction['predictions']: # Get bounding field coordinates x = obj['x'] y = obj['y'] width = obj['width'] top = obj['height'] label = obj['class'] confidence = obj['confidence'] start_point = (int(x - width / 2), int(y - top / 2)) end_point = (int(x + width / 2), int(y + top / 2)) data = { "label": label, "confidence": confidence, "start_point": start_point, "end_point": end_point, "x": x, "y": y, "width": width, "top": top } total_information.append(data)
Then, utilizing the predictions in that body (gotten from prediction[‘predictions’]), we’re in a position to get the x, y, width, top, label, and confidence of the field.
After, we calculate the start_point and end_point and add that data in addition to the beforehand obtained data similar to x, y, width, and many others. to the “total_information” checklist.
Subsequent, we add one of many methods to trace one’s productiveness. If one of many labels within the body is a cellphone, we add the time of once I was unproductive to the x_point checklist. Since y_point tracks the instances I used to be unproductive, add yet another to y_point so as to monitor how productive I’m.
if label == "cellphone" and "cellphone" not in prev_information: x_point.append(datetime.now()-start_time) x_point_phone.append(datetime.now()-start_time) y_point.append(y_point[len(y_point)-1]+1) y_point_phone.append(y_point_phone[len(y_point_phone)-1]) if as soon as: playlist_name = "Greatest Classical Music" search_playlist(playlist_name) play_playlist("Greatest Classical Music") as soon as = FalseLastly, we verify if that is the primary time I’m unproductive. If that’s the case, the “if as soon as == True” assertion is true and it performs the Spotify music. Take away the if assertion logic if you don’t want to play Spotify music.
# Put together label textual content label_text = f"{label}: {confidence:.2f}" # Get textual content dimension (text_width, text_height), baseline = cv2.getTextSize(label_text, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1) # Draw background rectangle for the label cv2.rectangle(picture, (start_point[0], start_point[1] - text_height - baseline), (start_point[0] + text_width, start_point[1]), (0, 255, 0), cv2.FILLED) cv2.putText(picture, label_text, (start_point[0], start_point[1] - baseline), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 1) Now, draw the label code for every detected bounding field. An instance of what the code does is proven under:
for i in vary(len(total_information)): if total_information[i]['label'] == 'cellphone': draw(total_information[i]['x'], total_information[i]['y'], total_information[i]['width'], total_information[i]['height'], picture, RED) elif total_information[i]['label'] == 'face': draw(total_information[i]['x'], total_information[i]['y'], total_information[i]['width'], total_information[i]['height'], picture, GREEN) elif total_information[i]['label'] == 'eye': draw(total_information[i]['x'], total_information[i]['y'], total_information[i]['width'], total_information[i]['height'], picture, BLUE) eye_vals.append(total_information[i]) for j in vary(len(total_information)): if total_information[j]['label'] == 'pupil': draw(total_information[j]['x'], total_information[j]['y'], total_information[j]['width'], total_information[j]['height'], picture, PURPLE) pupil_vals.append(total_information[j])
The code above is used to attract out every picture utilizing the operate and knowledge gotten from earlier code. As a substitute of drawing the objects within the loop for predictions, by drawing every object in a specified order, we’re in a position to affirm no overlap in bounding containers because the eyes could probably cowl the pupils.
Within the two pictures under, the left one specifies the order of drawing, which is why we’re in a position to see the pupils (because it was drawn final). The opposite picture (proper) didn’t specify, which meant the attention’s had been drawn above the pupil, blocking them from our view.
Then, we add the data from total_information to the pupil_vals and eye_vals lists.
if len(eye_vals) == 2 and len(pupil_vals) == 2: right_eye, left_eye, right_pupil, left_pupil = get_specific_eyes(eye_vals, pupil_vals) right_side_eye = right_eye["x"] - right_eye["width"]/2 left_side_eye = left_eye["x"] + left_eye["width"]/2 right_side_pupil = right_pupil["x"] left_side_pupil = left_pupil["x"] difference_right = right_side_pupil - right_side_eye difference_left = left_side_eye - left_side_pupil
Subsequent, so as to monitor the distinction between the pupil and eye size, I tracked the space from the outer sides of each eyes to the midpoint of the pupil. These lengths of variations are often called “difference_left” and “difference_right”.
if ((difference_left < 38) or (difference_right < 38)) and (prev_difference_left > 38 and prev_difference_right > 38): x_point.append(datetime.now()-start_time) x_point_eyes.append(datetime.now()-start_time) y_point.append(y_point[len(y_point)-1]+1) y_point_eyes.append(y_point_eyes[len(y_point_eyes)-1]+1) if as soon as == True: playlist_name = "Greatest Classical Music" search_playlist(playlist_name) play_playlist("Greatest Classical Music") as soon as = FalseUtilizing the 2 variations, I monitor how small the size of the space is on one of many pupils. If one pupil’s distance is smaller than 38, then the classical music will play and x_point will add the time. Just like the cellphone logic, if that is the primary time (indicated if as soon as continues to be True), then we’ll play spotify music.
prev_difference_right = difference_right prev_difference_left = difference_left if total_information: prev_information.clear() for information in total_information: prev_information.append(data["label"]) else: prev_information = [] total_information.clear() eye_vals.clear() pupil_vals.clear()
On this code block, we implement the earlier variable logic by defining earlier variations and former data. We additionally clear the values for complete data, eye_vals, and pupil_vals as they’re outdated variables.
cv2.imshow('body', picture)
Lastly, we are able to present the body of the picture. Which appears one thing like this!
if (datetime.now()-start_time).total_seconds() > 30:
pipeline.terminate()
pipeline.be part of()
cv2.destroyAllWindows()
break
if cv2.waitKey(1) & 0xFF == ord('q'):
breakThe previous couple of items of the operate are for when the pipeline shuts down. For now, I set a timer that leaves the operate when it reaches a time of over 30 seconds. Change the quantity “30” to any variety of seconds you need the code to cease at.
Run the Actual-time Pipeline
To run our predictions we’ll use an Inference pipeline. Inference pipelines are used primarily for real-time object detection fashions, which implies it’s excellent for our use case.
pipeline = InferencePipeline.init( video_reference=0, model_id="productivity-tracker/9", # Change with your personal mannequin max_fps = 30, confidence=0.6, api_key=api_key, on_prediction=on_prediction,
) start_time = datetime.now()
pipeline.begin()Now, we are able to lastly outline the pipeline and run it. Additionally make sure that to outline the start_time so as to calculate the instances at which you had been distracted.
Plot Outputs on a Graph
Within the final part, we make the most of matplotlib’s graphing capabilities so as to plot three graphs:
- Total instances distracted
- Instances distracted by cellphone
- Instances distracted by means of gazing off
if x_point: x_point = [delta.seconds for delta in x_point]
y_point.pop(0) plt.determine(1)
plt.plot(x_point, y_point)
plt.xlabel("Time (seconds)")
plt.ylabel("Instances distracted")
plt.legend() if x_point_eyes:
x_point_eyes = [delta.seconds for delta in x_point_eyes]
y_point_eyes.pop(0) plt.determine(2)
plt.plot(x_point_eyes, y_point_eyes)
plt.xlabel("Time (seconds)")
plt.ylabel("Instances distracted by eyes") if x_point_phone:
x_point_phone = [delta.seconds for delta in x_point_phone]
y_point_phone.pop(0) plt.determine(3)
plt.plot(x_point_phone, y_point_phone)
plt.xlabel("Time (seconds)")
plt.ylabel("Instances distracted by cellphone") plt.present()
else:
print("100% productive!")To graph the chart of distractions, I first must verify if I used to be distracted in any respect. If that’s the case, it would get the x_points and y_points and plot the gradual enhance in distraction extra time. In any other case, it would print that I used to be not distracted a single time all through the time interval given. The opposite two graphs even have related logic as if the checklist of x_point eyes / x_point cellphone was not empty, then it would graph a chart.
Conclusion
Through the use of laptop imaginative and prescient, we’re in a position to construct a real-time productiveness tracker. On this instance, we used object detection with logic to set off notifications and ship information for monitoring. Utilizing a number of libraries similar to Matplotlib and Spotify, we’re in a position to graph and add methods of strengthening productiveness inside our private workspace. Total, by means of using laptop imaginative and prescient, we are able to efficiently monitor how distracted we’re inside a time period.
This similar course of may be utilized to construct any real-time monitoring utility.
