In his research project at Software Campus SuGraBo, which he realizes with Software AG, Sven Hertling from DFKI is researching a search engine for a graphic user interface. Therein he develops a framework to develop models from any, particularly already existing software. Based on this, various assistances are offered to support learning and dealing with the software. The main application is a semantic search on the elements of the user interface (buttons, menu etc). Once a result is selected, so-called “in application tutorials“ are offered and moreover, textual assistance systems can be maintained and used across application boundaries. Next to context-specific auto-completion, for instance, background information can be shown in text. He presented a prototype and textual assistance systems at the Summit 2017.
For a paper based on this research, he and his colleagues received the Best Paper Award in February at the 9thInternational Conference on Agents and Artificial Intelligence in Porto, Portugal.
The paper “Where is that Button Again?! – Towards a Universal GUI Search Engine” provides an overview on similar approaches and describes the three main challenges the team had to solve. hese were the analysis of the graphic user interface, understanding user requests and finding a suitable solution on how to find the searched element or function in the user interface.
Based on a user study, the team assessed their method and showed that it is perceived as particularly useful when a feature, which is not used regularly, is searched. The method shows considerable potential concerning other application possibilities.