Added support for the new table recognition algorithm
Minor general changes needed to properly support the algorithm
Table recognition - new algorithm
A new algorithm is available in the v3/text and v3/pdf endpoints. It can be enabled by specifying "enable_tables_fallback": true as one of the request arguments.
We care deeply about backwards compatibility. The new algorithm will only be used if both of the following conditions are fulfilled:
Our standard algorithm failed to recognize a table.
"enable_tables_fallback": true is specified as a request argument.
We have ensured that there will be no computational overhead for customers who do not specify this option, so response times will not be affected.
We have invested in a hybrid approach that will be able to tackle complicated cases like:
extremely large tables (e.g. tables with hundreds of cells)
tables with very complex structures (e.g. tables with many \multirow and \multicolumn cells)
tables featuring table cells with complex content like:
table cells with complex math like large matrices or several aligned equations
tables cells containing whole paragraphs of text
tables containing text in languages that are more challenging to recognize properly compared to English:
this includes languages with rich alphabets like Chinese, Japanese, Hindi, Hebrew, Arabic, and others
tables containing cells with rotated text
tables containing diagrams inside cells like:
table cells with chemistry diagrams (note that these can be converted to SMILES)
table cells with natural images or similar:
table will still be recognized and contain the image link for the diagram inside its cells
We will also support all combinations of the above cases.
The algorithm we are releasing now might still struggle with:
tables with cells containing complex math like large matrices or several aligned equations
tables containing diagrams inside cells
empty grids of cells without textual content
tables with rotated text are partially supported:
in v3/pdf cells containing rotated text will be embedded as images
We will add improvements that will cover specified cases shortly.
Some differences in output produced by the new algorithm compared to the standard one:
Column alignment is always central.
All cells have all borders (top, bottom, right, and left).
February 1, 2023
Deployed new version, RSK-110.
Overview of changes
Added support for new Latex commands: \measuredangle, \grave, \bumpeq, and \amalg.
Improved recognition of constructs expressed with \lceil, \rceil, \lfloor, and \rfloor in combination with \left and \right.
Improved formatting of equations in text mode (see bellow for details).
Improved recognition of equations that contain large sub-equations in subscripts.
Improved recognition of handwritten French.
Improved recognition of handwritten German.
Improved recognition of handwritten Chinese.
Improved recognition of handwritten Japanese.
General improvements (new data iteration).
Formatting changes
The default “text” output has been changed, for example see the following equation:
from the current “text” output:
\( y=mx+b \)\n\( x=y^{2}-1 \)
and 2 asciimath equation outputs, to:
\( \begin{array}{l} y = mx + b \\ x=y^{2}-1 \end{array} \)
with 1 single asciimath equation output:
{:[y=mx+b],[x=y^(2)-1]:}
which will make the “text” derived formats more consistent with what is currently returned for equations with a left brace:
Since we are already emitting in certain cases (eg, equations with no braces aligned around the “=” sign instead of being left aligned) asciimath for v3/text that looks like:
{:[2x+8y=21],[6x-4y=14]:}
we consider this update to be an inconsistency bugfix instead of a new feature with the potential to break backwards compatibility.
In general, it is desirable for our API for small changes in input to result in small changes in output. For example, removing the left brace from equation 2 will simply change the v3/text asciimath from:
{[2x+8y=21],[6x-4y=14]:}
to:
{:[2x+8y=21],[6x-4y=14]:}
which is a smaller change than the previous behavior, in which subtracting a left brace results in two equations instead of 1.
January 30, 2023
Deployed new version, RSK-109.
Improved recognition of isolated symbols.
January 12, 2023
Deployed new version, RSK-108.
Improved handling of images that contain mixed math and text in Russian.
December 2, 2022
Deployed new version, RSK-107p2
This update features two changes:
changes in spacing of arrays, aligned arrays and similar, & and \\ now always have spaces around them (even with rm_spaces in the request)
visually unpleasant blocks of equations are being converted to left alignment instead of keeping the wrong alignment
November 25, 2022
Deployed new version, RSK-107.
Improvements related to worksheet crops, small images with strong or dashed border near the content.
November 22, 2022
Deployed new version, RSK-106.
Improvements related to formatting of references in PDF pages, especially pages with green/red link boxes.
November 15, 2022
Deployed new version, RSK-105.
General improvements to handling zoomed out and zoomed in images. No changes to output formatting or error characteristics.
November 14, 2022
Deploying new version RSK-104p1.
Formating of block math is fixed in certain cases where the equations were wrongly kept in the text mode.
November 11, 2022
Deploying new version RSK-104.
Incremental improvement of image parsing module. Includes fixes for images with many lines of text. Accuracy improvements on handwritten data. No changes to output formatting.
November 3, 2022
Deploying new version RSK-103p1 which fixes string post processing issues.
In this version, we have changed the default Markdown / LaTeX for the following character:
# -> \#
While # works fine in Markdown and has the same behavior as \#, the former causes LaTeX compilation issues, whereas \# succeeds in LaTeX without any problem. We chose to always emit LaTeX \# instead of # so that our output would be more compatible and less likely to cause issues. The updated character \# is compatible with Markdown as well as LaTeX.
Unescaped # will simply no longer appear in OCR Markdown / LaTeX outputs.
Alternative math formats such as Asciimath are not affected by the change, this is a Markdown / LaTeX change only.
October 21, 2022
New enable_spell_check option to the v3/text and v3/pdfs endpoints greatly improves handwriting OCR for English (other languages coming soon)
June 6, 2022
Resolved a critical bug that impacted PDF processing of 2 column PDFs
You can now request that only certain subsets of pages are processed in a PDF, via the new page_ranges field
Pushed latency improvements that benefit all endpoints, reducing processing time by 30% on average
Updated how line data is represented for PDFs from using rectangular regions to polygonal contours (this is helpful for handwritten PDFs where text lines are generally not rectangular)
Added page dimensions to the line-by-line data structures
There are two available data structures for line-by-line data:
Raw PDF lines data: this is the ideal data structure for searching; does not contain contextual annotations for titles, abstracts, etc.
Context enhanced PDF mmd lines data: you can use this to re-create the full document, including contextual annotations for titles, abstracts, etc. (see here for syntax)
Published a Github repo which contains client-side code for live drawing with the Mathpix digital ink API containing a fully working example of leveraging user actions like scribbling and strikethrough to delete content
April 18, 2022
Added an EU server region (AWS region eu-central-1) to decrease latencies for European customer and also for adherance to GDPR
You can now use app-tokens for authenticating requests inside client side app code
The new app-tokens route provides a include_strokes_session_id flag, which when true, returns a strokes_session_id string that can be used inside calls to v3/strokes, enabling digital ink sessions with live updates
Pricing for the strokes endpoint when using session_ids can be found here
Add OCR support for basic handwritten PDFs
March 28, 2022
You can now get detailed line-by-line data for PDFs, including geometric coordinates, via the new GET v3/pdf/<pdf_id>.lines.json endpoint.
Better robustness for our v3/text endpoint:
Our ability to correctly interpret complex layouts involving math and text has improved, with much-improved edge case handling and handling of line text for skewed images and other image distortions that occur frequently in consumer photo search applications.
March 14, 2022
Our new OCR models feature stringent guarantees of syntactic correctness, resolving a rare but long-standing problem of occasionally malformed LaTeX strings, resulting in rendering errors due to double subscripts, double superscripts, malformed tables, and other syntax issues. This has been fixed at a fundamental level. Syntax issues are essentially completely fixed.
Deprecated \atop command in favor of \substack
February 8, 2022
We have recently switched to a new, faster database to save image and PDF data. Next week, we will decommission our old database. This will result in OCR API image results log data from before December 1st, 2021 becoming unavailable via the GET v3/ocr-results endpoint. Note that we have already migrated all PDF data to the new database, so there will be no data loss for PDF data.
November 15, 2021
Deployed incremental update to our re OCR engine, resulting in:
significantly improved handwriting recognition, including disambiguating symbols based on context
improved table parsing accuracy
notably fewer errors
September 2, 2021
Deployed a core algorithm update for our image parsing module, resulting in significantly better accuracy and edge case behavior for all endpoints
Added support for Tamil, Telugu, Gujarati, and Bengali
Updated our OCR to use a more effective representation of Chinese characters, leading to higher accuracy, and better coverage.
Added support for \bigcirc
July 19, 2021
Support for sending image binaries for lower image upload latencies
Support for tags which allow you to associate an attribute with your requests and subsequently retrieve the associated requests by using tags in a /v3/ocr-results query
PDF processing updates:
Fixed a bug where pages were getting skipped
Improved processing of PDFs with foreign languages
Servers in Singapore for faster latencies for API customers in Asia
Triangle diagram OCR now supported for diagrams commonly found in trigonometry textbooks
Added InChI option for chemical diagram OCR
April 2, 2021
Added a include_word_data parameter to the v3/text endpoint, which when set to true, returns word by word information, with separate results, confidences values, and contour coordinates for each word
significant accuracy improvements to text / math localization performance for v3/latex and v3/batch for both ocr=["math"] (default) as well as ocr=["math", "text"], resulting in lower error rates and fewer bad results
November 12, 2020
added autorotation for v3/text
Images like this now work in v3/text:
The goal of automatic rotation is to pick correct orientation for received images before any processing is done. The result of auto rotation looks like:
We will soon add these features to v3/latex and v3/batch as well. We implemented a very conservative rotation confidence threshold, meaning you should still try to call the API with a properly oriented image if possible!
First of all, we trained our models on a larger dataset, resulting in a general accuracy increase.
Secondly, we improved the precision of predictions, at the potential cost of slightly decreasing prediction recall in some circumstances. Here is an example of an image, where previously our v3/text tried to read the bottom, cut off parts of the image:
Now, v3/text ignores these sections, resulting in a much cleaner output than before. The endpoint will still try to read everything in an image (vs the v3/latex endpoint which tries to read the main equation), but will be slightly less aggressive in reading unusual image sections in order to avoid garbage outputs.
chemistry diagram detection
We have added a new field in our LineData object, subtype, so that we can return more information about diagrams to API clients. Currently subtype can only be chemistry, but more diagram subtypes are coming soon.
