AIDOS: System for analyzing unstructured information, opinions, and sentiment using natural language processing and machine learning

The project responded to companies' need to extract useful knowledge from the large amount of unstructured information generated by their customers and users (opinions, comments, conversations), in order to understand the perception, assessment, and social reaction toward their products and brands.

The system was structured around two main blocks: a text analyzer, responsible for performing semantic analysis of free text, removing noise, and detecting mentions of commercial entities (companies, products, and brands); and a profile identifier, responsible for establishing opinion profiles (or clusters) of communities and users, generating information that marketing teams can use to study social perception of a given topic.

The development was organized around the following capabilities:

• Acquisition and integration of unstructured information: from a wide variety of sources, such as forms and surveys, recordings of conversations, or the opinions users share on social media.
• Automatic audio-to-text transcription: through an in-house speech to text module developed from the study and evaluation of different deep learning techniques, which makes it possible to use content available only in voice format as a source for analysis.
• Polarity and topic classification: based on sentiment and entity corpora developed specifically for the project, to determine whether an opinion is positive, neutral, or negative and what type of product it refers to.
• Automatic identification of opinion profiles: using clustering algorithms that group users according to their assessments, making it possible to design communication and marketing strategies aimed at each profile.

The system was validated in the food sector, automatically analyzing the opinions expressed on the social network Twitter about several major national supermarket chains, processing more than 938,000 messages from 4,437 different user accounts.

CTIC developed AIDOS drawing on its experience in natural language processing, machine learning, and the analysis of unstructured data from digital sources.