Knowledge of semantic information about users’ content and interests is essential for providing selection and recommendation services that filter data according to each individual’s intentions and preferences. On the other hand, the lack of efficient and generic techniques for extracting this high-level information makes the semantic gap problem persist to this day. Using user-produced annotations can reduce this problem as semantically richer metadata can be obtained from user interaction activity. However, to enable this strategy, some issues need to be investigated, such as the presence of noise and irrelevant data, the way information is constructed and represented, the lack of mechanisms to transform annotations into semantically structured data, and the application of this information to appropriate services. This research plan aims to investigate these challenges to enable the development of multimedia selection applications. It is hoped that semantic metadata about content and personal preferences can be extracted collaboratively by users unrestrictedly to the data domain, requiring no costly and error-prone efforts as in current approaches in the literature.