On the low-code platform Meltemee, we have developed an application that supports organizations processing large volumes of data, usually in multiple IT systems, in organizing personal data. In practice, data deduplication is a technique for eliminating duplicated data. It doesn't sound complicated, but if a bank or insurance company has multiple systems that process personal data, and the organization has grown through acquisitions, the task of organizing data and eliminating duplicates can be quite challenging.
The low-code application on the Meltemee platform supports customers in data deduplication and building a so-called “golden record.” The process begins with data standardization and cleaning, removal of unwanted characters, correction of names using a name dictionary, expansion of abbreviations, filling in missing fields if they can be inferred from other fields, standardizing and tokenizing selected fields, checking data correctness and compliance with rules, comparing records and classifying them, and finally creating so-called “golden records.” The application uses advanced data comparison algorithms based on internationally recognized and mathematically proven methods, and the result of the work is made available to the customer's source systems in the form of a golden record – the most up-to-date source of truth about the personal data being processed.
