Genie Enterprise Data Scientist Report –Study, analyse and interpret documents with GenieReader

Genie Enterprise Data Scientist Report –

Study, analyse and interpret documents with GenieReader

By Nazeer Basha Shaik

Welcome to the Digital century. Never wondered how digitization would change our lives and make them run faster? Digitization has transformed handwritten documents to editable documents, this made creation of documents easier. Creating, editing, adding graphics, deleting documents and transferring them has become smoother and a lot easier than ever before.

That's when things went out of control, in the name of automation, writing or creating documents became simpler, just with a click a document of thousand pages is getting created in seconds, I would say. But then, who is going to read them? Let’s keep it this way: How long you might need to read them? How long you might need to analyze them? How long you might need to draw insights out of them? How long you might need to connect the dots in the document? Let's give it a bigger shot, how long you might need to assess thousands of such documents. Sounds interesting, but reading all those documents would probably be boring for a human. An intelligent program however wouldn’t be bothered. Did I just reveal that there is a solution for the problem? I guess I did, we Genie Enterprise have a one stop solution, GENIEREADER. 

Genie Reader is an important digital tool used to study, analyse and interpret text. The digital world has left us with bountiful raw data, allowing us to mine and discover interesting patterns that would have been impossible before.

For a human to read large amounts of text and understand all of the important and nuanced patterns is monotonous and time consuming. This type of work takes precious time from highly paid experts. However, intelligent algorithms can instantly make sense of reams of documents.

Genie Reader has three unique skills which make it better than any other tool available; it can read documents, analyse and represent them. With the help of advanced learning algorithms Genie Reader can analyse digital text and interpret it efficiently.

Genie Reader parses and reads the content of the documents, analyzing them for specific aspects and validates the information following a rule-based approach. In simpler words, the parser converts unstructured data to structured data. The parsed text is evaluated using state of the art preprocessing techniques that later feed to our pipeline.

Our pipeline works as follows: after converting the raw and unstructured text to a syntactically parsed and annotated data type, we apply state of the art pre-processing techniques to remove the noise out of the data. These pre-processing techniques include the removal of stopwords, stemming, and the replacing of special characters. Later the pre-processed text is used to run three parallel processes.

One process is dedicated to learning and knowledge creation, for which the preprocessed text is forwarded to coreference resolution. By applying coreference resolution, we resolve the grammatical references with nouns and pronouns in the context of the structured data. The extracted text is later used to build a Knowledge Graph in the form of triplets. The triplet structure consists of subject, predicate and object. The Knowledge Graph is later plotted and visually represented with a user-friendly interactive interface. Once we have the knowledge graph extracted from the context we can use the cutting edge querying technologies like SparkQL, GraphQL etc. These querying languages provide us with solutions for our complex queries.

Another parallel process is context and reference extraction from the whole document. Explicit and implicit references are extracted in this process. Once the references are extracted, we use them to plot a user-interactive graph.

Once the references are extracted, the third Genie Reader process uses technologically advanced similarity models to create a corpus. In this process each pre-processed text segment is encoded using transformer models. Based on the encoding a similarity score is generated and those scores are used for the results on the search page displayed in Genie Reader.

This is the basics about the technology behind GenieReader in a nutshell. If you would like to learn more about this powerful text and document analysis solution and its potential business applications, feel free to contact us.

Nazeer Basha Shaik (MSc. Data Analytics) is a Data Scientist at Genie Enterprise.
He is deeply involved in the development of the GenieReader technology and the implementation in several custom solutions for our clients.