As part of ICFHR 2016 (http://www.nlpr.ia.ac.cn/icfhr2016/competitions.htm), we propose a competition on Script Classification.
1. Competition: task and context
1.1 Task under evaluation
The task to be evaluated in the present competition is the classification of 1000 images of Latin Scripts, from handwritten books dated 500 C.E. to 1600 C.E.
The organizers provide a training data-set consisting in 2000 images of well defined script types. They will evaluate the results on a 1000 additional images which are not included in the training data-set, or/and on a set of 2000 images of mixed script type images.
The complete data-set is thereafter named CLaMM : classification of Latin Medieval Manuscripts
There are 12 pre-defined classes in CLaMM according to the script style.
We propose two possible tasks: task 1 named “Crisp Classification” and task 2 named “Fuzzy Classification”.
Both tasks are independent, and participants have to announce if they want to perform task 1, task 2, or tasks 1 and 2.
- Task 1 is divided in two steps:
- The participants will have to provide a “distance matrix” between pairs of images
- The participants will have to associate a single label to each image
- Task 2 is also divided in two steps
- The participants will have to provide a “distance matrix” between pairs of image
- The participants will have to associate a multi-weighted labeling to each image
Participants are expected to provide the executable files, capable of producing the results of steps 1 and 2 of the respective tasks according to the format that is described in section 2.
Automated analysis and classification of handwritings applied to the written production of the European Middle Ages is a new challenge and a “frontier in handwriting recognition”.
Digital libraries from Cultural Heritage institutions contain literally ten-thousands of digitized manuscripts of the European Middle Ages. Some examples:
- more than 12,000 fully digitized manuscripts in Manuscripta Mediaevalia (http://www.manuscripta-mediaevalia.de/);
- 10,000 in Gallica (http://gallica.bnf.fr), the digital library of the French National Library;
- 7,700 in the BVMM – Bibliothèque Virtuelle des Manuscrits Médiévaux (http://bvmm.irht.cnrs.fr), or Virtual Library of Medieval Manuscripts, run by the Institut de Recherche et d’Histoire des Textes (CNRS);
- 4,000 in Teca Digitale (http://teca.bmlonline.it/), digital library of the Biblioteca Medicea Laurenziana;
- 2,000 “Digitised Manuscripts” on the website of the British Library (http://www.bl.uk/manuscripts/);
- 1,400 in e-codices – Virtual Manuscript Library of Switzerland (http://www.e-codices.ch/en);
- 400 in the Digital Walters (http://www.thedigitalwalters.org/), digital library of the Walters Art Museum (Baltimore)
- 300 in the “Beinecke Digital Collections” (http://brbl-dl.library.yale.edu/) of the Beinecke Rare Book & Manuscript Library (Yale University Library);
- as well as 5447 (often partially) digitised manuscripts in Digital Scriptorium (http://vm136.lib.berkeley.edu/BANC/digitalscriptorium/)
The overwhelming majority of manuscripts in there are written in Latin script.
In this context, there is a need for an automated “tagging” or “cataloguing” of the handwriting on the images, not only to allow for historical research (when and how which text is written), but primarily because it is a pre-requisite for handwritten text recognition (HTR) or automated indexing and data mining. To perform HTR on the digitized manuscripts, one “numerical model” is necessary to recognize the text for each script type and the identification of the script type is the first step.
This has been stated for the modern handwriting styles . The medieval millennium extending from 500 C.E. to 1600 C.E. shows that the Latin script evolved and took very different forms, much more diverse than all the writing styles of the 19th to the 21st century.
1.3 Importance of the CLaMM dataset
The participants of this competition will get the only available reference data-set covering the European Middle Age and tagged as regards script types and production date.
In real-life conditions and beyond the challenges of material degradations, segmentation, etc., one of the difficulties is that there is a historical continuum in the evolution of scripts so that there are mixed types and many scripts that could pertain to two or more categories. In this regard, classification of scripts addresses the subjectivity of the human mind, so that, as in art history, all attributions remain subject to debate and discussion.
In this competition, in task 1 the training data-set and the test data-set encompass well defined script types, in order to make the evaluation possible. In task 2, the test data-set also encompasses mixed script types, which illustrates evolution of Latin scripts.
1.4 Related topics and previous work
The present competition on the Classification of Medieval Handwritings in Latin Script is related but differs from:
- Segmentation and text detection on an image;
- Image Feature Extraction;
- Sorting out different scripts (Latin / Arabic / Greek / Hebrew, etc.);
- Performing scribal identification within a homogenous corpus or within a particular manuscript.
The latter topic is the closest and has been dealt with by numerous competitions and publications –.
As for the Classification of Medieval Handwritings in Latin Script specifically: the first attempt at automating the classification of medieval Latin scripts was made by the Graphem research project (Grapheme based Retrieval and Analysis for PalaeograpHic Expertise of medieval Manuscripts) funded by the French National Research Agency (ANR-07-MDCO-006, 2007-2011). The results are published in , .
Further research has been conducted on a theoretical level by one of the organizers and several teams in Computer Science–. Nevertheless none of the teams had access to the labelled data-set and the latter has not been made available anywhere.
2.1 Image data-set
Both training and test data-sets consists of grey-level images in TIFF format at 300 dpi, picturing a 100 x 150 mm part of a manuscript.
The training set consists of 2000 images. The test set consists of 1000 images for task 1, and 2000 images for task 2.
The list of classes is provided in a CSV file with 2 columns: “FILENAME,SCRIPT_TYPE”.
The image collection used for the competition is mainly based on the collection of 9800 images from the French catalogues of dated and datable manuscripts–, increased with the on-line documentation from the BVMM (http://bvmm.irht.cnrs.fr/) and Gallica (http://gallica.bnf.fr/) in order to build classes of the same size.
The images of the training set are tagged according to 12 labels. The division of scripts is based on morphological differences and allographs, as defined in standard works on Latin scripts , .
The following script types are described:
- Mostly capital script
- Almost no ascender or descender (d, F, h, p, q overlap only minimally)
- Reduced A; d with shaft sloping to the left; rounded, often opene/Eand M;
- Minuscule script (ascenders and descenders clearly between the lines)
- mainly open letter a; vertical d,3 or z-shaped letter g; minuscule form of m (three minims); both N and n may be used; mostly long s (ſ = U+017F); t with loop on the left;
- The corpus encompasses more or less cursive types of half-uncial scripts
- Minuscule script (ascenders and descenders clearly between the lines)
- uncial roundeda, half-uncial d, g with a small round eye; f, i, r and long s may have (short) descenders, flat-topped t;
- vertical uncial a; half-uncial d and uncial d; f and long s stand on the line; long s and round s may be used simultaneously
- script based on Caroline type, characterized by its lateral compression (narrower letters and fusions) and regularization of strokes, esp. equalizing tendency in the treatment of feet.
- two-compartment a; f and long s on the line; loopless ascenders (b, h, k, l);
- Southern Textualis (Rotunda):
- two-compartment a; f and long s on the line; loopless ascenders (b, h, k, l)
- This script is the Mediterranean forms of Textualis. It is characterized by the roundness of its bows, visible especially in b, c, d, e, h, o, p, q and round r
- single-compartment a; f and long s below the line; loops on ascenders (b, h, k, l);
- single-compartment a; f and long s below the line; loopless ascenders (b, h, k, l);
- Semitextualis: Textualis (cf. supra) with single-compartment a
- Semihybrida: Hybrida with irregularly loopless ascenders (b, h, k, l);
- Humanistic: Imitated from Caroline scripts, with dotted i
- Humanistic Cursive: Humanistic script with cursiva forms and ‘italic’ slant
2.2.2 Ideal models of the script types
2.2.3 Samples from the CLaMM data set
3.1 Expected outputs
3.1.1 Executable file
Participants will provide
- an executable file
- a description of the required environment, resources and the expected processing time for 1000 images.
The executable file should:
- read the images from a folder
- a “ symmetrical distance matrix” of the images present in the folder (for task 1 as well as for task 2). The sum of all entries of this matrix will be equal to 1.0.
- a CSV file with 2 columns: “FILENAME, SCRIPT_TYPE ” for task 1 (SCRIPT_TYPE, an integer between 1 and 12, indicating the belonging class).
- a CSV file with 13 columns: FILENAME,SCRIPT_TYPE1, …., SCRIPT_TYPE12” for task 2 (SCRIPT_TYPE1, …., SCRIPT_TYPE12 are real values, and each row has a sum of its values equal to 1.0). This information will be named Belonging Matrix in the following sections.
Nota: Participants are allowed to submit several independent proposals
Based on the test data-set, the evaluation will be given as follow:
- Accuracy per script type
- Global accuracy for fuzzy results
- Normalized distance matrix analysis
- Processing time
For each task, two rankings will be done. The first one will be based on the average global accuracy, the second based on the symmetrical distance matrix.
The “accuracy per script type” is given according to the ground truth, which has one label for each script image in the evaluation data set. The ranking will be based on the average accuracy.
As far as the distance matrix is concerned, the evaluation will be done on the average intraclasse distance that will yield to the second ranking.
3.2.2 Task 2
The “global accuracy for fuzzy results” will be evaluated as follow:
- The ground truth indicates one or two labels for each script image, from the Belonging Matrix, only the two highest membership degrees will define SCRIPT_TYPE1 and SCRIPT_TYPE2.
- Scores will be attributed as follow:
- 4 points if SCRIPT_TYPE1 AND SCRIPT_TYPE2 match the labels given to the image in the ground truth.
- 2 points if SCRIPT_TYPE1 matches one of the labels given to the image in the ground truth, but not SCRIPT_TYPE2.
- 1 point if SCRIPT_TYPE2 matches one label given to the image in the ground truth, but not SCRIPT_TYPE1.
- (-2) points if SCRIPT_TYPE1 AND SCRIPT_TYPE2 do not match any of the labels given to the image in the ground truth.
The ranking will be based on the average accuracy.
As far as the distance matrix is concerned, for each image, according to the ground truth, we consider the one or two classes with highest membership degree. Then the sum of distances to the images of the same ground truth classes will be computed and averaged all over the test base. The ranking will be done with respect to this global value.
3.3 Additional information
The distance matrix will be used to further analyze the results. Some classes are acknowledged to be closer to one another for historical and phylogenetic reasons. The building of the data-set and the discretization of classes make the accuracy rate a sufficient measure. It is however relevant for future work (e.g. HTR) which classes are intermingled and which may be clearly separated.
4 Registration and access to data
To register in this contest send an e-mail to dominique_DOT_stutzmann_AT_irht_DOT_cnrs_DOT_fr and CC: to m_DOT_helias_AT_irht_DOT_cnrs_DOT_fr with the subject ICFHR 2016 CLaMM competition registration.
In the message you must provide the following data:
- Group name and acronym
- Participants and e-mail
- Contact person
- Registration for task 1 OR/AND task 2
22 February 2016: Competitions published on ICFHR2016 website and open to participants.
15 March 2016: Training data-set available. Participants have to announce their participation by emailing the organizers and will receive login/password to download the training data set.
Nota bene: Participants are welcome, even after March 15th, but the following deadline will remain unchanged!
21 May 2016: Delivery of executable files with one-page description of method
<!– update –>
Deadline extended to 27 May 2016
01 July 2016: Deadline for submission for review of full papers describing the competitions. Papers must be sent directly to the Competitions Chairs.
01 October 2016: CLaMM data set released (metadata + images) on the website of the Institut de Recherche et d’Histoire des Textes (http://www.irht.cnrs.fr) and will be maintained by the institution.
6.1 Dominique Stutzmann (dominique [dot] stutzmann [at] irht [dot] cnrs [dot] fr)
Senior Researcher at Centre National de la Recherche Scientifique (French National Center for Scientific Research).
6.2 Véronique Eglin (veronique [dot] eglin [at] insa-lyon [dot] fr)
Full professor in computer science at INSA de Lyon (Engineer school). Board member of GRCE (since 2000).
6.3 Nicole Vincent (nicole [dot] vincent [at] mi [dot] parisdescartes [dot] fr)
Full professor at Paris Descartes University, France.
Member of advisory committees of conferences: Cifed 12, 14, 16, EGC11, 12, 13 ; Member of the Program Committees of conferences: Cifed 08,10, ICFHR 12, 14, 16, ICDAR 13, 15 ; Co-Chair of the Program committee of EGC 05 ; Chair of the organizing committee of EGC05; Member of the editorial board of journal Pattern Recognition
6.4 Florence Cloppet (florence [dot] cloppet [at] mi [dot] parisdescartes [dot] fr)
Assistant professor in Computer Science at Paris Descartes university (Paris-
6.5 Marlène Helias-Baron (m [dot] helias [at] irht [dot] cnrs [dot] fr)
Research ingeneer at Institut de Recherche et d’Histoire des Textes (CNRS), Paris
7 Works Cited
 R. Niels and L. Vuurpijl, “Generating Copybooks from Consistent Handwriting Styles,” in Ninth International Conference on Document Analysis and Recognition, 2007. ICDAR 2007, 2007, vol. 2, pp. 1009–1013.
 L. R. B. Schomaker, K. Franke, and M. L. Bulacu, “Using codebooks of fragmented connected-component contours in forensic and historic writer identification,” Pattern Recognition Letters, vol. 28, no. 6, pp. 719–727, 2007.
 A. A. Brink, J. Smit, M. L. Bulacu, and L. R. B. Schomaker, “Writer identification using directional ink-trace width measurements,” Pattern Recognition, vol. 45, no. 1, pp. 162–171, 2012.
 S. He, M. Wiering, and L. R. B. Schomaker, “Junction detection in handwritten documents and its application to writer identification,” Pattern Recognition, vol. 48, pp. 4036–4048, 2015.
 D. Muzerelle and M. Gurrado, Eds., Analyse d’image et paléographie systématique : travaux du programme “Graphem” : communications présentées au colloque international “Paléographie fondamentale, paléographie expérimentale : l’écriture entre histoire et science” (Institut de recherche et d’histoire des textes (CNRS), Paris, 14-15 avril 2011). Paris: Association Gazette du livre médiéval, 2011.
 D. Stutzmann and M. Gurrado, “Mesure et histoire des écritures médiévales,” in Mesure et histoire médiévale, Actes du XLIIIe Congrès de la SHMESP, Paris: Publications de la Sorbonne, 2013, pp. 153–166.
 N. Vincent, A. Seropian, and G. Stamon, “Synthesis for handwriting analysis,” Pattern Recognition Letters, vol. 26, no. 3, pp. 267–275, 2005.
 G. Joutel, V. Eglin, and H. Emptoz, “Generic scale-space process for handwriting documents analysis,” in 19th International Conference on Pattern Recognition, 2008. ICPR 2008, 2008, pp. 1–4.
 I. Siddiqi, F. Cloppet, and N. Vincent, “Contour Based Features for the Classification of Ancient Manuscripts,” presented at the 14th Conference of the International Graphonolics Society, (IGS), Dijon, 2009.
 G. Joutel, V. Eglin, and H. Emptoz, “Generic Scale-Space Architecture for Handwriting Documents Analysis, chapter 15,” in Pattern Recognition Recent Advances, A. Herout, Ed. InTech, 2010, pp. 293–312.
 F. Cloppet, H. Daher, V. Églin, H. Emptoz, M. Exbrayat, G. Joutel, F. Lebourgeois, L. Martin, I. Moalla, I. Siddiqi, and N. Vincent, “New Tools for Exploring, Analysing and Categorising Medieval Scripts,” Digital Medievalist, no. 7, 2011.
 H. Daher, V. Églin, S. Brès, and N. Vincent, “Étude de la dynamique des écritures médiévales. Analyse et classification des formes écrites,” Gazette du livre médiéval, vol. 56–57, pp. 21–41, 2011.
 I. Siddiqi, F. Cloppet, and N. Vincent, “Writing property descriptors. A proposal for typological groupings,” Gazette du livre médiéval, vol. 56–57, pp. 42–57, 2011.
 V. Eglin, D. Gaceb, H. Daher, S. Bres, and N. Vincent, “Outils d’analyse de la dynamique des écritures médiévales pour l’aide à l’expertise paléographique,” Document Numérique, vol. 41, no. 1, pp. 81–104, 2011.
 D. Stutzmann, “Clustering of medieval scripts through computer image analysis: towards an evaluation protocol,” Digital Medievalist, vol. 10, 2015.
 C. Samaran, R. Marichal, M.-C. Garand, and J. Metman, Catalogue des manuscrits en écriture latine: portant des indications de date, de lieu ou de copiste, Tome I: Musée Condé et bibliothèques parisiennes, 2 vols. Paris: Centre national de la recherche scientifique, 1959.
 C. Samaran, R. Marichal, M.-C. Garand, and M. Mabille, Catalogue des manuscrits en écriture latine: portant des indications de date, de lieu ou de copiste, Tome II: Bibliothèque Nationale, fonds latin Nos.1 à 8000, 2 vols. Paris: CNRS, 1962.
 C. Samaran, R. Marichal, M.-C. Garand, and M. Mabille, Catalogue des manuscrits en écriture latine: portant des indications de date, de lieu ou de copiste, Tome V: Est de la France, 2 vols. Paris: CNRS, 1964.
 M.-C. Garand and M. Mabille, Catalogue des manuscrits en écriture latine portant des indications de date, de lieu ou de copiste. Tome VI, Bourgogne, centre, sud-est et sud-ouest de la France. Paris: CNRS, 1968.
 C. Samaran, R. Marichal, and M. Mabille, Catalogue des manuscrits en écriture latine: portant des indications de date, de lieu ou de copiste, Tome III: Bibliothèque Nationale, fonds latin Nos. 8001 à 18613, 2 vols. Paris: CNRS, 1974.
 C. Samaran, R. Marichal, M.-C. Garand, M. Mabille, and D. Muzerelle, Catalogue des manuscrits en écriture latine: portant des indications de date, de lieu ou de copiste, 4: Bibliothèque nationale Fonds latin (Supplément), nouvelles acquisitions latines, petits fonds divers, 2 vols. Paris: Centre national de la recherche scientifique, 1981.
 C. Samaran and R. Marichal, Catalogue des manuscrits en écriture latine: portant des indications de date, de lieu ou de copiste, Tome VII: Ouest de la France et pays de Loire, 2 vols. Paris: CNRS, 1985.
 D. Muzerelle, Manuscrits datés des bibliothèques de France. 1. Cambrai. Paris: CNRS Editions, 2000.
 D. Muzerelle, Manuscrits datés des bibliothèques de France, 2: Laon, Saint-Quentin, Soissons, 1 vols. Paris: CNRS éd, 2013.
 B. Bischoff, Paläographie des römischen Altertums und des abendländischen Mittelalters, 4., durchgesehene und erw. Aufl. Berlin: E. Schmidt, 2009.
 A. Derolez, The Palaeography of Gothic Manuscript Books from the Twelfth to the Early Sixteenth Century. Cambridge: Cambridge University Press, 2003.
4 réflexions sur « ICFHR2016 Competition on the Classification of Medieval Handwritings in Latin Script »
These data sets should be made publicly available to anyone, not just to academic “teams”.
They are available to be downloaded by anyone for non-commercial use