Duplicate Identification Algorithms in SaaS Platforms

Dac Nguyen, Quy H. Nguyen, Minh-Son Dao, Duc-Tien Dang-Nguyen, Cathal Gurrin, Binh T. Nguyen
2020 Proceedings of the 2020 Intelligent on Intelligent Cross-Data Analysis and Retrieval Workshop  
Existing duplicate records is one of the most common issues in many Software-as-as-Service (SaaS) platforms. In this paper, we study the duplicate identification problem in one specific SaaS platform related to quality and compliance management by using the address information. We interpret all typical mistakes from users that can generate the existent duplicated organizations in a given dataset, collected from the SaaS platform. Also, we create another set by crawling location data from Open
more » ... dress (US Zone). We compare different methods, including Bag-of-words (using Cosine Distance), Record Linkage Toolkits, and Siamese Neural Networks using the triplet loss, in terms of precision, recall, and F1-score. The experimental results show that using Siamese Neural Networks can achieve a better performance in comparison with other techniques. We plan to publish our Open Address dataset and all implementation codes to facilitate further research in the related fields.
doi:10.1145/3379174.3392319 fatcat:gbvt4urt3nft5mb3xqgxowdf2q