OCR is one of the earliest addressed computer vision tasks. But there are few solutions available for its application in specific domains such as parsing of mathematical formulas. Thus we will solve this problem in a simple educative way, providing a comprehensive introduction into Computer Vision(CV) field with a possibility to extend the base solution. The resulting program incorporates segmentation of input image into characters and then character recognition itself based on Convolutional Neural Network(CNN).
Firstly, prepare the data by executing the following commands from the project's root folder:
cd data
unzip emnist.zip
unzip crohme.zip
And then run Main.ipynb notebook
In the root folder, you can see multiple *.ipynb and *.py files. Below is a detailed description of each of them
- References.ipynb - notebook with reference materials related to the project,
- Troubleshooting.ipynb - notebook with a list of typical problems, which you may encounter while running the project, and their resolutions
- DataPreprocessing.ipynb - notebook for preprocessing datasets. It cleans, merges and enhances data from EMNIST and CHRONME datasets, which located under
datadirectory - CharacterSegmentation.py - python module with realisation of line segmentation functionality. It takes its input from the
inputfolder and saves the segmented characters intosegmentedfolder. - RecognitionModelTraining.ipynb - notebook which trains CNN model to parse characters. Trained models are saved under the
modeldirectory. - Main.ipynb - notebook with a main pipeline of the project. It orchestrates all modules by firsly segmenting the line into characters, then parsing them, and then converting parsed characters to Latex.
For project details, please check out supporting report