This is the extended abstract for a seminar on «Genre innovation» at the University of Oslo, 6 December 2013.

We have developed a web site for mobile devices, presenting church music to tourists in Rome (Fagerjord, 2011). Using the phones’ GPS, it gives directions to six churches where tourists can listen to music written for that church centuries ago together with spoken commentaries. Blending genres from radio and tourist guides we have created a genre prototype. We also hoped to learn more about what a good location-based texts is. Is this design research, or just design?

According to Hevner et.al. (2004), design science builds a new artifacts from a “knowledge base” of foundations and methodologies, and the resulting design adds to this knowledge base (80). When we create new experiences, services, and genres, we draw on humanist knowledge of genres, storytelling, rhetoric, visual culture, and and much more. Can we give back to these disciplines, using design as a research method in the humanities?

We tested our application in Rome, using observation, “think-aloud” methods, semi-structured interviews and a simple survey, methods adapted from the Human-Computer Interaction (HCI) field in computer science. We were able to make the application easier to use, but observational methods could only be used to evaluate the user interface. They hardly gave any insight into how participants experienced the churches together with the music and the commentaries.

To find this out, we interviewed the users and gave them a simple survey. They all said they liked our application, and the survey scores were all  positive. We were encouraged, but what could we add to the knowledge base? We took, for example, care to point out synaesthetic parallels when we made the commentaries, drawing attention to structural similarities in the church’s architecture and music written in the same period. Can we now conclude that this is a general principle that works for situated sound? No. We can’t even be certain that it worked in this case: The users may have liked other parts of the application. As Popper argued, only falsifications are reliable.

Surveys more sophisticated than ours exist. Psychologists have in recent years investigated what they call hedonistic quality, such as how appealing the user finds a product’s look and feel. AttrakDiff is one questionnaire created to measure hedonistic quality (Hassenzahl 2001). Its authors have tested it statistically and found it valid, but remind us that while the questionnaire measures how pleasurable a product is, it cannot say what about the product that creates pleasure or indifference.

Disciplines like engineering, computer science, information systems, pharmaceutics, or medicine create artifacts that are based in theories from (observational) science, and scientists test the artifacts with the same methods that were used to create the theories, mainly observations and statistics, Krippendorff observes (2006. p. 260-61). When we design genre prototypes using humanist theories, we should evaluate their success against the same theories with the methods that created them. I will outline four such methods:

Much of humanist scholarship concerns itself with canonical masterworks, and may not explain why a work is high quality. Its greatness is taken for granted. This will not do for design research. Presenting a new genre prototype as a demonstration of excellence hardly adds to the knowledge base. Some scholars have discussed why works should be in the canon, however, and these criteria may be used as heuristics, as check lists we can evaluate a new designs with.

Krippendorff (2006 p.264;) suggests another method of validation: Before the design, he asks stakeholders what a successful product or artifact should be like. When evaluating the finished artifact, he asks stakeholders to describe it, and compares theire descriptions with the earlier accounts of desirable futures. Similar descriptions indicate success. A variant of this method could be used for the Rome project: We could interview users and ask them to describe how they experienced the service. If we have clearly stated goals for the project, we can evaluate whether the users’ descriptions match these goals using semantic analysis.

We can also do textual experiments, creating texts that are as similar as possible, but differ on the important detail we want to evaluate. For the next version of the church music project we want, for example, to create some audio commentaries that do not use synaesthetic parallels at all, and see if the testers notice any difference. We have hypothesized that this is an important technique for situated sound, and will suffer a severe blow if users don’t notice a difference.

Last, we can use the oldest validation method of them all, peer review, a long an honored tradition in humanist scholarship. We should let other researchers analyze and critique our genre designs.