what every PhD should know: dinner discussion with a google guy

Just barely hanging out there. Today was lots of serious fun and intellectual challenges at the RecSys 2007 Doctoral Consortium . Interestingly, all the participants came from lots of different backgrounds from computer science, information retrieval to me from education. I think the diversity of backgrounds and focuses of studies represent the growth of the Field of Recommenders, it's not only about the best algorithm anymore, but a plethora of questions around.

Anyway, being somewhat a newbie here (yeah, I do not know all the people or study areas here, very eye opening!), it makes me think that all the PhD students should be exposed to the question " If you were to have dinner next to a main researcher in Google/Yahoo/or any other big name, what would you want to talk about?".

Well, as it happens to be, I never thought of that before. Neither was I prepped for that by my study programme. Nevertheless, I just spent my dinner next to Krishna Bharat, you know, the guy who greated Google News, nothing less, nothing more. Probably tomorrow I'll have like ten things I want to ask from him with no chance to get his attention anymore.

Bottom line: it is not only about the 1 minute elevator pitch, but about the life and such in general.

Is rating broken?

Yahoo!, You Tube and Netflix all use ratings on their services to better gear towards users' needs. We are talking about huge numbers here; Yahoo! gets some 5 million ratings a day for artists, albums, songs and videos; Netflix has currently 2 million ratings per day from its 5M customers; and within a day after the launch of lonelygirl15's My First Kiss- clip on YouTube she's got over 5000 ratings. Clearly, rating is something not to neglect, it is an easy way for users to input their opinion, as well as rather a straight way to compute affinities in relation to some other type of data such as search history, demographies, etc..

However, it seems like there is more to rating than meets the eye, and it becomes increasingly complicated for services to make the best use out of it. In the Recommenders06 conference issues with ratings were mentioned, but not discussed thoroughly. To me this seems a highly important issue, as current services are using rating as a primary input for their recommenders and many of the algorithms work based on ratings.


The following issues came up with ratings (unordered list):

• Semantics of rating are pretty unclear; what does a user actually mean with 3.5?
• Meaning of ratings is very subjective; does my 2.5 mean the same as your 2.5?
• Ratings are straight out unclear; on the scale 1 to 5, does one (1) mean that I really don't want to ever see it again or does it mean that I just quite don't like it?
• What does a single attribute actually mean when rating for example one movie; is it about the plot, the actor, soundtrack? What if I like the plot but hate the main actor, how to express that?
• Love/Hate-ratings: many services are getting more and more ratings only on the far dimentions of the scale; rating value distribution is large.
  
• Binaries like thumbs up and down have issues too; How do I interpret something that has 10 thumbs-ups and 10 downs? Am I going to take the risk to either really like it or really hate it?
• Rating variance; how does 10 up and 10 down rating effect on people's choice? Do people go for the middle way? Apparently not, see Jolie's presentation.
• Only a few have rated many, many have rated a few – distribution of ratings is very sparse. It is hard to recommend something for those many with few ratings.
• Rating distribution between genres: some genres are more predictable than others thanks to user ratings. How to recommend the ones not so ofter rated? In Netflix comedies and drama are more predictable than musicals, for example. The presentage of 4-5 start movies rented has increased, as prediction accuracy becomes better.
• Do users understand what ratings are for? Whether users really understand what ratings can do for them when using Yahoo! Music stuff or Netflix? It's about the trade-off between control gained over the service but yielding to users' convenience that they give up when taking time to rate.
  
• Feedback loop between rating and recommendations can become self-promoting. If I rate something good, the recommender keeps recommending that or similar items to me (also known as similarity-trap). There is the popularity bias: at the end, everything is related to Britney Spears.
• Knowing the users' intentions: wanna buy or listen?
• Ratings depend on when the item was rated (Netflix found out that ratings done immediately after watching the movie vary from the ones made at the later stage!).
• Ratings are vulnerable to chilling, intentionally bad ratings, want to lift some music up on the list by rating it high, etc. (influencing the vote is relatively easy using some algorithms whereas hybrids might be more robust against manipulation, see Mr Mobasher’s persentation).
• Does the user feel home with the other raters? Am I sure that I belong to this group of users and tastes? For example Last.fm started as a rather geek service, thus lots of users have rated items that match geeky music taste! This becomes really important when we think about internationalisation of recommender services, can my taste match with white-male-middle-class American taste?
• Computing affinities with userprofile, editorial rankings, etc can take a long time, for example some Yahoo! services are only updated weekly since it's so computing intensive.
  

Ok, I think it's broken, but the question is can it be fixed? Well, that looks like a long list of issues to deal with, but I'm sure nothing has gone beyond repairing.

However, there are many remaining questions: How to help people who don't rate? How to better understand users behaviour, what do they like and what not, and get that information in a more implicit way? The following remedies were mentioned in the conference:

• Going beyond rating for data input for recommenders by monitoring the play events in an online radio.
• Uploaded playlists by users can yield important information about sequencing music, moods that they are played in, etc.
• Netflix talked about encoding traits of movies that predict emotional responses, for example. Maria, one of the students, talked about combining personality traits and mood settings to further personalise and contextualise recommendations.
• Prof.Riedl talked about letting users know the value of their rating to the community, e.g. how important rating one given item is to make better recommendations for this given group. It seems like people care about others, they are willing to make ratings to help other people similar to them finding better items.
  
• Using social networks to better make and find recommendations.
• Imporving ROI for users; with fewer inputs get more valuable outputs like playlists, concerts, videos, music news, etc.
  

Some more ways that I could think of:

• If binary types of ratings are something that people do, let's just use thumbs up and thumbs down.
• If scales are used, be explicit about them. No one really knows what the stars mean in iPod! Say clearly: O means “never play again”.
• Multi-attribute ratings: if you allow people to rate, give them also options to be more clear about it; I think the plot is good, but acting sucks. There are people who love to do ratings and evaluations (just look at Amazon.com with their reviewers lists!) and many times they are good in doing it.
  
• Leverage on re-using ratings from other services: Netflix, Yahoo!, MovieLense and God knows how many other services rate the movies. Think about webservices or harvesting those ratings and get rid of the sparsity problem! There should be some interoperability between user ratings and other evaluations between services.
• I want a meta-recommender! Would be good to know if my music taste matches with other people's taste in a given service or whether I should hang out somewhere else to get favourable recommendations.
• Anyway, those services are too focused only getting people to use that one and only service, by pooling up and letting users to take advantage of their profile in place a in place b would be convenient for me! Maybe Attention metadata could become to help here. Attention XML and Attention Metadata: Collecting, Managing and Exploiting of Rich Usage Information at International ACM Workshop
  

A meta recommender of movies and music

Recommenders06
Originally uploaded by alvy.

Would be fun to play around with a meta-recommender that compares recommendations from different services, for example, for a movie. So, I would be in one service, say Netflix and had hard time choosing a film between the big choice of movies. Instead of knowing what other Netflix users thought about the movie, I could also see how users in Yahoo!, MovieLens, etc rated it.

As we found out today, ratings, that are widely used in recommenders, are quite tricky things. There is a lot of sparsity problems, variations in ratings, interpretations of a rating scale, not enough criteria or too much of them, etc. So, it might be useful (or then totally not), to see how MovieLens and Amazon users rated the movie, to compare whether there is a deep variation between them, and eventually find a community whose tase is similar to yours.

Of course the datamodels are not the same and there are different rating scales, but some normalisation could be done or thumbs up/down, or so. Besides, comparing the evaluation data across different applications would probably yeld very interesting results!

Recommenders06 - round table and attention

IMG_2840
Originally uploaded by jjdonald.

As the round table discussion drifted from one thing to another, some things penetrated my attention - specially when the discussion was about business models and future trends. It was mentioned that convergences between things are found, lilke the pact with Apple and Nike.

It made me think further about the labour intensiveness of the social applications on the web; you have to add your user profile, friends, music taste, etc to make it fun to work with (some automatisation exist, but not quite..). The same thing with recommenders, first I spend time to rate movies in MovieLens to get some recommendations. Then I go to Amazon.com to buy some movies or books, and they don't know shit about my taste as I haven't been shopping exclusively there.

Someone mentioned "Universal profile", which apparently has been thought of at some point in the past. To me that sounds like a really bad idea. That should be a mother big profile, a real monster to keep all info about me and my intentions and attentions.

The idea of using Attention metadata (AttentionTrust.org) kind of solution would be better. I own my attention, and I choose how much of it I want to share with a commercial entity to get better recommendations.

Some current work going on Attention XML and Attention Metadata here: Collecting, Managing and Exploiting of Rich Usage Information at International ACM Workshop http://ariadne.cs.kuleuven.be/cama2006/

Recommenders 06

Yesterday after arrival to Bilbao, Spain, some people from the conference got together to go around the town. There was Claudio, Nikos, Jolie and Iilja. So I already had a chance to talk with Claudio Baccigalupo (IIIA-CSIC, Spain) about his research already.

(btw; Blog postings with presentations are found at http://blog.recommenders06.com/)

It's about automatically generating playlists based on Case-based Reasoning (CBR). You can find a demo at http://labs.mystrands.com. As a case base for his recommender he uses uploaded playlists by MyStrands users who have decided to upload them. Then the system analyses them for regarding the order, and when a user says that she wants a playlist with F.Sinatra, the recommender creates one based on previous playlists. Of course, for the system to work well it is desirable to assume that the case base is perfect, i.e. hopefully the playlists are created by semi-professional deejays, and not just with people without any understanding of how songs actually best fit together to have a nice flow.

Fun, will play around with it!