Data deduplication with AI: Entity Resolution – Automated data cleansing for greater efficiency

Entity Resolution: The problem

Multiple entries can occur in a data record where the same person or object is listed several times. A common example is a list of employees in which the same person is listed under different variants.

Let’s take the head of department “Ms. Erika Mustermann” as an example. She may be listed under different entries such as “Erika”, “Ms. Mustermann”, “e.mustermann@firma.de” or “Head of Sample Department”. Although all these entries refer to the same person, they appear individually.

In order to filter these multiple entries, the data records must be clearly assignable. To do this, a list must be created in which the duplications are filtered into individual, unique data records.

Multiple entries lead to higher operating costs due to redundant data processing, impair data analysis and cause unnecessary work time for cleansing. They can also lead to confusion, for example when different departments access inconsistent data or when wrong decisions are made based on incomplete information.

Example from everyday life

We assume a customer who needs to classify products from different e-commerce websites into the correct categories in the course of a purchase or for integration into a central product database. For example, a product could be listed in the category “Devices > Pneumatics > Control devices” on one website and under “Pneumatic control devices” on another. These different categorizations must be correctly merged to ensure that the products are displayed consistently and correctly everywhere.