[jaffathecake]

I know this isn't quite what you mean, but when you first hit the page, the list is in a random order, but it's then stable across reloads.

I considered the 'vs' approach, but I worried that there might be a lot of iterations where one or two of the options would be things that the person didn't understand, or didn't care about.

How do you feel about something like this: The user goes through the long list, picking what they understand and don't dislike, then the 'vs' system is there for helping determine the order of those items. Then the user gets the ranking which they can tweak.

[MzHN]

> I worried that there might be a lot of iterations where one or two of the options would be things that the person didn't understand, or didn't care about.

I won't disagree here, it would be tedious.

> picking what they understand and don't dislike, then the 'vs' system is there for helping determine the order of those items

Sounds promising to me.