Week 1 Documentation of Scribble AI
In this project, we will be training and building an AI model that is able to detect scribbles and be able to tell what object the scribble represents.
In general, scribble is a game where every member of a group that is playing take turns to draw (or scribble) something on a canvas, and every other non-scribbling member has to guess what that object is.
Take a look at: https://skribbl.io/. This is the game. Play a little to get familiar with it. It's always best to play with a couple of friends so you get the idea of the entire game.
This project will be pretty similar to the game. The players should be able to choose a word randomly from a list of words that the AI will not know of. Afterwards, every player will be required to scribble the word as best as they can and the AI will try and predict what the word is. Simple idea. There will be no scribble-drawing AI, however.
What we will be using to complete this project will be Python for our base programming language, as well as Streamlit, which is a Python library that allows us to build web-ready apps without writing any web dev code. This will allow us to focus solely on Python and the AI. Since this project focuses on learning some foundational AI, we will be making use of Neural Networks to train the AI model, as opposed to using something like OpenAI API to have GPT recognize the image for us. We will be using scikit-learn for the AI.
1) If on Windows, install Python from https://www.python.org/ and download the latest version.
2) On Windows, make sure to check the box that says "Set Python to PATH" while installing.
3) On MacOS or Linux, Python should be pre-installed, so update it using Homebrew: [Insert MacOS instructions]
Install streamlit using pip inside a terminal or command prompt:
pip install streamlit
Now, we get our feet a little wet by writing some basic Python code that uses the streamlit library to display content on a webpage. After creating a new .py file, write the following code:
import streamlit as st
st.write("Hello World")
Then, it can easily be run from the terminal using the command:
streamlit run your-file-name.py
At first, we import the library. As we progress, you will come across this import statement a lot of times as we need to import external modules and libraries to help us achieve our goal.
Secondly, the st.write() function takes whatever is put inside the parentheses and then displays them on the browser. This is a really simple approach where you do not have to deal with HTML, CSS or JS code and jump straight into Python while also having a web app to display your project. Just like st.write(), there are a plethora of other streamlit functions and programming styles that we will look at in the future weeks.
For your final project, you’ll be creating an interactive web application inspired by Google’s Quick, Draw!. This is an engaging game where an AI tries to recognize your doodles in real time. You’ll be using Streamlit, a powerful tool you’ve been learning about, to build this application. The goal is to design an app that can interpret the user’s doodles and make accurate guesses about what they represent. It’s a fantastic opportunity to apply your coding skills, explore AI, and create something fun and interactive. I encourage you to visit Google’s Quick, Draw! to get a feel for what you’ll be building.
Check it out at: https://quickdraw.withgoogle.com/
The very first thing you might have noticed on Google Quickdraw is the big white canvas where you have to draw the doodles. To build a project similar to that, our first task, hence, should be to get a canvas up and running. And not any canvas would do, we need a drawable canvas!
Luckily for us, we don't need to code out a drawable canvas ourselves. There exists lots of external libraries for streamlit - called streamlit "components" - and one of them is a drawable canvas! This should give us a head start so that we can put all of our focus onto the AI features of this project.
We'll install it right away:
pip install streamlit-drawable-canvas
After we install it, we are free to get started. To begin, we will use the starter code provided by this library to get a canvas up and running:
import streamlit as st
from streamlit_drawable_canvas import st_canvas
# Specify canvas parameters in application
stroke_width = st.sidebar.slider("Stroke width: ", 1, 25, 3)
stroke_color = st.sidebar.color_picker("Stroke color hex: ")
bg_color = st.sidebar.color_picker("Background color hex: ", "#eee")
realtime_update = st.sidebar.checkbox("Update in realtime", True)
# Create a canvas component
canvas_result = st_canvas(
fill_color="rgba(255, 165, 0, 0.3)", # Fixed fill color with some opacity
stroke_width=stroke_width,
stroke_color=stroke_color,
background_color=bg_color,
update_streamlit=realtime_update,
height=500,
drawing_mode="freedraw",
point_display_radius=0,
key="canvas",
)
# Do something interesting with the image data and paths
# The image from the canvas is available as an ndarray and can be accessed via "canvas_result.image_data"
if canvas_result.image_data is not None:
# Do something here
That is a lot of code! Fear not!!
We'll go through them together. Let's look at the imports first
import streamlit as st
from streamlit_drawable_canvas import st_canvas
So, we are importing streamlit as before, but now we are also importing the drawable canvas that we just installed.
Let's look at the first three lines:
stroke_width = st.sidebar.slider("Stroke width: ", 1, 25, 3)
stroke_color = st.sidebar.color_picker("Stroke color hex: ")
bg_color = st.sidebar.color_picker("Background color hex: ", "#eee")
In these lines, we are making use of the .sidebar() functionality of streamlit. It is just like .write() or any other streamlit function we have looked at before. It is used to control the sidebar of our webapp. Then, with the sliders and color pickers we are setting the stroke width and color, and the background color.
Now:
realtime_update = st.sidebar.checkbox("Update in realtime", True)
This line dictates whether the canvas is updated in real time or not. It is a checkbox in the sidebar. By default, it is set to true. For our purposes, we will always want this to be true.
At the very end:
canvas_result = st_canvas(
fill_color="rgba(255, 165, 0, 0.3)", # Fixed fill color with some opacity
stroke_width=stroke_width,
stroke_color=stroke_color,
background_color=bg_color,
update_streamlit=realtime_update,
height=500,
drawing_mode="freedraw",
point_display_radius=0,
key="canvas",
)
This is the code that draws the canvas on the webpage. You can see the various options it is taking, the stroke width and color, the background color, whether to update in realtime or not, everything we did before is now passed to this function to dictate how the canvas will look like.
Pretty much all the code we have covered till now should remain the same and will only give you a canvas. The main task (and challenge) now lies in doing something with the image produced by the canvas.
The image by the canvas is available in canvas_result.image_data. We can do whatever we want with the image right now.
Until next time, keep going over the things learned today, namely:
1) What Scribble is
2) Playing around with Python to get familiar with it as a programming language
3) Looking a little bit deeper into st.write() from the streamlit docs.
