At the outset, the approach taken was a rule-based data-mapping activity. This method required customisations for each website to adapt to their unique terminologies, classifications, and free-text fields. Unfortunately, POC 1 didn’t yield results as expected due to the vast disparities in data presentation among different websites.

POC 2 introduced Natural Language Processing (NLP) and Named Entity Recognition (NER) techniques to extract key entities from the free-text fields. This approach demonstrated potential by pinpointing data locations, but it lacked the accuracy required to confidently detect and obtain accurate data. Efforts were made to combine NLP and NER with rule-based logic, but the complexity and cost of this approach became apparent.

In the final phase, a successful solution was created, blending rule-based processing and machine learning. The AI Engineering team precisely identified all the fields that the data entry team needed to populate, along with the associated validation rules. For each field, a decision was made to employ either rule-based processing or machine learning. The machine learning models, although time-consuming to build, provided the capability to generate data for fields that had previously necessitated human intervention.