Difference between revisions of "Evaltex"
From LRDE
Line 120: | Line 120: | ||
= Output format= |
= Output format= |
||
The evaluation results are given in two forms: |
The evaluation results are given in two forms: |
||
+ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
+ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
+ | |||
⚫ | |||
+ | |||
⚫ | |||
⚫ | |||
⚫ | |||
+ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
+ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
+ | |||
+ | |||
* local evaluation ''.txt'' file for each image |
* local evaluation ''.txt'' file for each image |
||
Line 198: | Line 238: | ||
GT object 6<br/> |
GT object 6<br/> |
||
Coverage = 1 Accuracy = 0.960362 Split = 1<br/> |
Coverage = 1 Accuracy = 0.960362 Split = 1<br/> |
||
⚫ | |||
⚫ | |||
− | |||
− | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
− | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
− | |||
⚫ | |||
− | |||
⚫ | |||
⚫ | |||
⚫ | |||
− | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
− | |||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
</small> |
</small> |
||
|} |
|} |
Revision as of 11:37, 8 July 2016
EvaLTex (Evaluating Text Localization) is a unified evaluation framework used to measure the performance of text detection and text segmentation algorithms. It takes as input text objects represented either by rectangle coordinates or by irregular masks. The output consists of a set of scores, at local and global levels, and a visual representation of the behavior of the analysed algorithm through quality histograms.
For more details on the evaluation protocol, read the scientific paper published in the Image and Vision Computing Journal. Details on the visual representation of the evaluation can be found in the article published in the Proc. of International Conference in Document Analysis and Recognition.
Please cite the IVC paper in all publications that use the EvaLTex tool and the ICDAR paper in all publications that use the histogram representation. |
Evaluation performance measurements
Local evaluation
For each matched GT object we assign two quality measures: Coverage (Cov) and Accuracy (Acc);
- Cov computes the rate of the matched area with respect to the GT object area
- Acc computes the rate of the matched area with respect to the detection area
Global evaluation
The Recall () computes the amount of detected text. We compute 3 measures: a global , a quantitative that measures the amount detected objects (regardless of the matched area) and a qualitative that corresponds to the rate of the detected text area with respect to the number of true positives ().
The Precision () computes the rate of detections that have a match in the GT. Similarly to , we compute 3 measures: a quantitative that measures the amount of valid detections (regardless of the matched area) and a qualitative that corresponds to the rate of the detected text area with respect to the number of total detections, computed as the sum of and
Input format
The framework takes as input .txt files containing the coordinates of the text bounding boxes and labeled images containing text masks, depending on the evaluation task. To unify the input format
Text detection tasks
Text detection results can be represented through boxes and masks. For text detection tasks using bounding boxes, a .txt file is enough. If the text objects are represented by irregular masks, then an additional labeled image will be needed.
Boxes
Ground truth format
The GT format contains the following attributes:
- img name
- image height, image width
- text object
- ID: unique text object ID
- region ID: region ID to which the object belongs to
- "transcription": can be empty
- text reject: option that decides if a text object should be counted or not; can be set to f (default) or t (not take into account)
- x: x coordinate of the bounding box
- y: y coordinate of the bounding box
- width: width of the bounding box
- height:x height of the bounding box
e.g.
img_1 960,1280 |
Detection result format
The detection .txt file format contains the following attributes:
- img name
- text object
- ID: unique text object ID
- "transcription": can be empty
- x: x coordinate of the bounding box
- y: y coordinate of the bounding box
- width: width of the bounding box
- height:x height of the bounding box
e.g.
img_1 1,"",272,264,392,186 |
Masks
If a more precise evaluation is needed, one can also evaluate the detection of text masks. To do so, we need TODO add explinations.
Ground truth format
Detection result format
Text segmentation tasks
Ground truth format
Similar to text detection tasks using masks, to evaluate text segmentation we use, in addition to the .txt file a labeled image (each character is labeled differently). Each GT object is represented by a character. Character-level GT objects cannot be grouped into regions and consequently each text object has a different region tag. The x, y, width and height will define the coordinates of the bounding box of each character.
e.g.
img_1 960,1280 |
Detection result format
Output format
The evaluation results are given in two forms:
- global evaluation for an entire dataset
EvaLTex statistics Global results FScore_noSplit=0.725095 Quantity results Quality results EMD results |
- local evaluation .txt file for each image
EvaLTex statistics - image img_1 |
Global results FScore_noSplit=0.572144 |
Quantity results |
Quality results |
EMD results FScore_noSplit=0.578853 |
Local evaluation |
Run the evaluation
The executable EvaLTex takes as input two .txt files, one for the GT and one for the detection/segmentation.
Usage
./EvaLTex -g gt.txt -d det.txt -o output_dir/ [options]
|
Datasets
ICDAR 2013
Born-digital
- ground truth .txt
- labeled images
Natural scene
- ground truth .txt
- labeled images
Downloads
Credits
EvaLTex was written by Ana Stefania CALARASANU. Please send any suggestions, comments or bug reports to calarasanu@lrde.epita.fr
Please cite the ICV paper in all publications that use the EvaLTex tool and the ICDAR paper in all publications that use the histogram representation. |