Some thoughts after SIRTEL07

Last week the SIRTEL workshop took place. The papers are found here and the slides, well, most of them, at the EC-TEL07 conference wiki. I have pretty good feeling about the workshop, it was one day long, we had about 20 people participating, some of whom chose to stay with us for the whole time, and some who were hopping between workshops. For me that is totally fine, we all are responsible for our own learning! Especially in conferences where many parallel sessions are running, I would encourage people to try to get best out of them.

For those who could not make it at all, you can soon find recording on the SIRTEL site.

The workshop had four main sessions:

• We started with a keynote address from people who work with music recommenders. MyStrands people talked about applying social recommender systems to technology enhanced learning. It was an interesting talk that challenged all of us to think what are recommenders for learning purposes in the first place (goal) and what kind of data do we want to use to do that.
  
  As any good keynote, this one gave more ideas to think than answers. It nicely set the base for the further discussions during the workshop that focused on the need to define the field of Social Information Retrieval for Technology Enhanced Learning, and to establish a baseline so that we know what are we really set to do.
  
  
• The second session was about Tagging and Visualisation. We had my presentation about the user behaviour on tagging in multiple languages; then there was a presentation from COSL that talked about "Activities of Daily Living on the Web", Brandon also showed a few demos of the widgets that can be used to rate or recommend related content. That was followed by a talk on reward structures to encourage teachers to share open educational material. Finally, we listened about Visualisation of social bookmarks, a work that leads into visualising bookmarks in an educational repository.
  
  
• The 3rd session was on Recommender Systems. Here we first heard about some R&D work that OU NL is carrying out using the idea of learning paths to better support learning activities of students. Then, there was a study about using affiliation networks as a mechanism for collaborative filtering (understood largely). This was followed by a study on simulating recommendations based on multi-attribute ratings on learning resources by teachers. Finally, we had a system demo of Daffodil that supports collaborative information seeking.
  
  
• The final session was what we called "Enablers and Challenges". It was a discussion session, and as we advanced, it was clear that people had a lot to say. It might even have been better to allow more time for this, but hey, you live and you learn.

I try to sum-up, but basically it illustrates the main topics that we talked about. If you look at the left side, there are the fundamental questions:

• How to define and chart out the area of Social Information Retrieval (SIR) for learning?
• Is this application domain different from other SIR, on micro and macro level?
• What do we recommend?
• In what context?
• and based on what?

On the right hand, there are the issues related to implementation and evaluation of it. These are:

• What are the best SIR methods for TEL?
• And what is the data that we should use? The "data issue" was something that was heavily emphasised by the MyStrands folks, who obviously speak of experience.
  
• The questions rouse also: when do we start implementing these for real or are we just over-engineering and never ready to launch?
• Evaluation and empirical data for real evidences was on the focus a lot.

More will follow. This is quick and dirty now, hopefully I will get more input from people participating in order to get more depth on our summary.

Massively multiplayer object sharing by R.Sinha

I've followed some stuff from Rashmi Sinha, and I think every once in a while she comes up with good ideas. Like I liked the stuff early on that she did on the recommenders and the focus on user-centric design. Sometimes I just don't like her stuff, it sounds very popularistic and her references are, well, not very academic. But then again, maybe she does not need to be either..

Anyway, this slideshow has cool ingredients. I like the idea of object/artefacts in the center of the social networks, that's why I'm a big fan of social bookmarking, for example. I really don't care that much about connecting to people that I don't know (mySpace) or even using LinkedIn (what's the point, you get a list of people, but no substance..), but when I can connect through items and tags to people's stuff that I find interesting, I find it useful.

In the slideshow Sinha talks about models of 2nd generation networks (the 1st g was only about people):

• Model 1: Watercooler conversations
  (around objects e.g., Flickr, Yahoo answers)
• Model 2: Viral sharing (passing on interesting stuff, e.g., YouTube videos)
• Model 3: Tag-based social sharing (linked by concepts. e.g., del.icio.us)
• Model 4: Social news creation (rating news stories, e.g., digg, Newsvine)
  

Then, further on, she talks about Cognitive Diversity, which I also find really important. It's related to the continuum of wisdom of crowds vs. stupidity of mobs. What I got out of the slide 29:

• Good answers need many perspectives, thus many perspectives are needed otherwise groups become too homogenous, which might have its dangers also (stupidity of mobs, see Digg for that ;). If all the new members are too similar and like-minded, they don't bring anything new to the group (that's why we want serendipity from recommenders!). Diversity reduces groupthink (think of Digg again and how fast not favourable stuff gets buried), groupthink is bad and only way to fight that is diversity.
  

Moreover, she also talks about the importance of social influence condition and about Watt's study.

Lastly, some design principles:

• Make system personally useful: For end-user system should have strong personal use; Self-expression (e.g., Newsvine);Social status: Digg
• Don’t count on altruism: System should thrive on people’s selfishness
  

The LibraryThing Recommender

I knew that LibraryThing.com had plans to work on a recommender for books, and seems like its out now. It's called LibrarySuggester; you can type a name of any book that you own or have read and the systems spills out suggestions in different categories:

• People with this book also have...(v 1)
• Special sauce recommendations!
• Books with similar tags
• Books with similar library subjects and classifications..
• Amazon recommendations
• People with this book also have...(v2)
  

LibraryThing Suggester analyses the more than thirteen million books and sixteen million tags LibraryThing members have added, and comes back with reading suggestions. Amazon suggestions come from Amazon.com, not LibraryThing.

Crowdsourcing

13 million books and 16 million tags, holy cow! That's some serious amount of data that people have free-willingly entered into the system! Just imagine trying to do the same before the day when the Web was crowdsourced. It would have taken an enormous amount of man-hours to enter people's likes and dislikes in books into a recommeder system as input to compute a list of recommendations, let alone the ratings, evaluations and discussions people have added too.

This is exactly the same way we want to go down with learning resources; first create a tool for teachers to create their favourite collections of learning resources and then use those to better serve them in terms of recommendations.

Transparency

When I look at the recommendations from LibrarySuggester, what I like is that they are clearly classified in different classes of recommendations and on what those are based on. It is nice, as a user, to get the reasoning behind, e.g. ah, I was recommended this book because other "people with this book also have.." or I know that it is based on similar tags, etc.

This kind of practice of being transparent about the recommendations has also been argued about in previous research in the field, and it seems to be something that people appreciate, as opposed to a "black box" recommendations where the user has no idea on what the recommendations are based upon (Swearingen, 2001; Rafaeli 2005).

List of recommendations

Also, what I like is that LibrarySuggester offers a list of recommendations, as opposed to one or a few to choose from. However, in my list there were 74 recommendations all together, which I find way too much!

There are also some really evident ones, like books from the same author, which is not really a salient recommendation. McNee, et al. (2006) talk about a "similarity hole" that item-item collaborative filtering algorithm can trap users into by only giving similar recommendations. They argue that the old-skool accuracy metrics should be taken with a caution, as they only are designed to judge the accuracy of individual items and not the list of items. Thus, "the recommendation list should be judged for its usefulness as a complete entity, not just as a collection of individual items."

Moreover, within the same framework, which is called Human-Recommender-Interaction, these folks talk about three aspects that should be improved in recommendations. They are similarity (discussed above), recommendation serendipity, and the importance of user needs and expectations in a recommender.

Serendipity

Take the list of "Special sauce recommendations" for Dune by F.Herbert. On the list of 20 books you can find on the top 2 of his other books, and 2 by B.Herbert, his son. This sounds rather dull and not really anything surprising, you could find that easily from a bookstore too. By serendipity, the authors mean how unexpected the recommendation is for the user and how novel it is. For me personally this is a very important factor and why I like the idea of recommenders, as opposed to just content-based retrieval of resources.

I won't discuss the importance of user needs and expectations in a recommender, as in this case it is pretty clear. In some other cases, though, like for learning resources, this comes very important, as teachers do have different tasks at hand when they are looking for learning resources. This is something I've blogged before about and will keep exploring in my context of research.

McNee, S.M. , Riedl, J. , and Konstan, J.A. (2006) "Being Accurate is Not Enough: How Accuracy Metrics have hurt Recommender Systems". In the Extended Abstracts of the 2006 ACM Conference on Human Factors in Computing Systems (CHI 2006) [to appear], Montreal, Canada, April 2006

Rafaeli S., Dan-Gur Y., Barak M. (2005), “Social Recommender Systems: Recommendations in
Support of E-Learning”, Journal of Distance Education Technologies, 3(2), 29-45,
April - June 2005.

Swearingen K., Sinha R. (2001). , “Beyond algorithms: An HCI perspective on recommender
systems”, ACM SIGIR 2001 Workshop on Recommender Systems, 2001.

Questions and notes on Making Recommendations Better: An Analytic Model for Human-Recommender Interaction

S.M. McNee, J. Riedl, and J.A. Konstan. "Making Recommendations Better: An Analytic Model for Human-Recommender Interaction". In the Extended Abstracts of the 2006 ACM Conference on Human Factors in Computing Systems (CHI 2006) [to appear], Montreal, Canada, April 2006.

The paper start from an healthy self-assertion that recommenders do not always generate good recommendations for users. Authors propose a Human-Recommender Interaction (HRI) as a framework and a methodology to understand users, their tasks and how do they relate to recommender algorithms. They propose that HRI can be a bridge between user information seeking tasks and recommender algorithms, when applied in the HRI Analytic Process Model, it can become a constructive model to help the process to design a recommender.

As the information density grows, users have more specific needs for their information seeking. HRI can be used to describe these needs, thus firstly, thinking about myself and my research area, I have to describe user types (probably I could use the LRE logs to deduce this) and typical domain tasks (this would have to be some guestimates that I test with a focus group). The authors suggest Hackos 1998 for this, but looks pretty old. A detailed analysis of these tasks will allow us to link task to specific HRI Aspects.


HRI Aspects, the three pillars:

• the Recommendation Dialog, the act of giving information and recieving one recommendation list from a recommender. This contains aspects like Correctness, Transparency, Saliency, Serendipity Quantity, Usefulness, Spread and Usability. The authors argues that recommender's purpose is to generate salient recommendations that strike an emotional response (the awe factor!)
• the Recommender Personality (uh, I don't like that term), the user's perception of the recommender over a period of time. Aspects are such like personalisation, boldness, adaptability, trust/first impression, risk taking/aversion, affirmation, pigeonholing and freshness.
• User Information Seeking Tasks, the reason the user came to the recommender system. Aspects such as Concreteness of task, task compromising, recommender appropriateness, expectations of recommender usefulness, recommender importance in meeting needs. Check out Case 2002.
  

The authors claim (still without any proof of concept, as the paper is pretty recent) that a user's needs and expectations from a recommender can be described by selecting the most relevant aspects from each pillar.

The HRI Analytic Process Model can help to analyse and redesign recommenders to better meet user information needs. (Can it also help to design them in the first place?) For example it could help to understand whether a user would be contented with risky recommendations or more like the ones that affirm her information seeking needs.

Moreover, the authors say that by looking at which HRI aspects are important to which task, some metrics can be designed (I would be very interested in those metrics!) to categorise the differences between tasks. These metrics could be used to benchmark the known algorithms, and thus help to choose the proper one for the task. Rather, as the authors state, a recommender should have a set of algorithms to use instead of being “one for all users”-type of set.

Questions for Mr. Riedl

• What are the metrics, any exaples?
• What are the outcomes of the simulations against the well-known algorithms, did the mapping between the tasks and algorithms materialise, and if not, how well? More information available? In the paper it mentiones that they are submitted, under review. Whom to contact?

More on HRI, a PhD thesis by McNee: http://www-users.cs.umn.edu/~mcnee/mcnee-thesis-preprint.pdf
Research statement by the above: http://www-users.cs.umn.edu/~mcnee/mcnee-research-statement.pdf