15 (Symbol code for “Dragon” class)
27 (Symbol code for “Cactus” class)March 2022
The goal of this project is to develop one of the building blocks of a larger pipeline, which aims at monitoring Dobble games.
As the Dobble game and the overall idea of the project were introduced during the lecture, we will focus here on what you have to produce. Of course, you can watch this video if you want to know more.
The specific goal of this project is to design a symbol classifier for Dobble symbols, which must:
Examples (note the various orientations and sizes):
| Sample input | Expected output |
|---|---|
|
15 (Symbol code for “Dragon” class) |
|
27 (Symbol code for “Cactus” class) |
About the input:
About the output:
-1 when you fail to recognize a symbol, you may return only one symbol code, or return codes and associated probabilities. This is up to you to specify a correct system which effectively solves the problem.You will need to divide your pipeline into two stages:
📦 Click here to download the train set.
We provide you with a set of 285 images to help you design and train your system.
We keep 155 images for the final test set.
The train set is available for all the duration of the project, and the test set will be accessible only during the defense: you will have to process the test images and share the result with us, so we can compute the performance of your system.
The features you design must be scale- and rotation- invariant. Ideally they should also be viewpoint- and contrast- invariant, but we will deal with a simplified problem here.
We strongly suggest that you implement two kinds of features:
You need to try several classifiers and explain your choices, in particular regarding the kind of feature you process.
Please stick with scikit-learn classifiers, and refrain from using deep learning techniques: one of the main goals of this course is to teach you the processing stages which are still present but integrated and hidden in a deep learning pipeline.
We expect you to try at the very least 2 classifiers.
We expect you to:
Please pay some attention to the quality of your tables and plots. We are particularly sensitive to proper axis naming, title choice, and legend content for figures, and column/row naming, units, number of significant digits for tables.
Evaluation is a critical part in this project, do not underestimate it: a proper evaluation is the only way to avoid data processing project becoming nightmares.
During your defense, we will share with you a set of images that you will have to process. This set will be a gzipped tar file, containing no folder and only PNG files like the ones used for training.
For each file you must produce the correct label (same name as the directories of the train set) and save them in a text file with this format (without header):
filename,predicted_class_idYour score will be computed as (\(N\) is the number of test cases): \[ S = 1 - \frac{1}{N} \begin{cases} 0 & (\text{predicted} = \text{expected}) \\ 0.5 & (\text{predicted} = -1) \\ 1 & (\text{predicted} \neq \text{expected}) \end{cases} \]
The closer you get to 1, the more performant your system is. The special -1 label can be used to mark uncertain samples and obtain a lower (halved) penalty for misclassification.
BEFORE THE DEFENSE, you must submit through Moodle the following deliverables:
DURING THE DEFENSE, you must submit through Teams the following deliverables:
Checklist for grading (may be slightly adjusted):
| Description | Grading |
|---|---|
| Feature extraction | |
| Implemented color histogram | 1 point(s) |
| Implemented some shape descriptor | 1 point(s) |
| Implemented extra feature extractors | 1 point(s) |
| Experimental setup | |
| Separated a validation set | 1 point(s) |
| Chose appropriate metrics | 0.5 point(s) |
| Classification | |
| Tried a dummy baseline | 0.5 point(s) |
| Train a linear classifier | 1 point(s) |
| Train a non-linear classifier | 1 point(s) |
| Train 3+ classifiers | 1 point(s) |
| Justified classifier choices | 1 point(s) |
| Tried explicit feature fusion | 1 point(s) |
| Evaluation, benchmark and calibration | |
| Performed meta-parameter optimization, if applicable | 1 point(s) |
| Obtained more than random performance | 0.5 point(s) |
| Obtained more than average performance | 1 point(s) |
| Evaluated color descriptor alone | 0.5 point(s) |
| Evaluated shape descriptor alone | 0.5 point(s) |
| Evaluated color and shape descriptors together | 0.5 point(s) |
| Description | Grading |
|---|---|
| Slides | |
| Followed recommended presentation outline | 1 point(s) |
| Produced correct and understandable result summaries | 1 point(s) |
| Talk | |
| Gave a clear summary of their work | 1 point(s) |
| Answered correctly to questions | 1 point(s) |
| Demo | |
| Processed the test set during the defense | 2 point(s) |
This is a bonus/penalty grading.