[Update: Moved to front from 13/3. I've created a wiki to continue the planning and discussion. Click here to sign up and begin editing.]
I want to say a little more about how academics might hope to take advantage of the shift to a 'publish, then filter' world.
The first thing to note is that we already solicit and offer each other informal feedback (i.e. outside of the journal peer review process) in a number of ways -- sending drafts to friends, presenting at conferences, etc. But these opportunities are predictably and unfortunately limited by the constraints of face-to-face social networking (e.g. geography: I never could have spoken with all the wonderful people I'm now surrounded by, back when I lived in New Zealand).
So it would seem desirable to take this informal process, and expand and enhance it by means of appropriate online infrastructure. We could create a website - a global database - to which philosophers could submit their draft papers in exchange for reviewing and rating others' submissions. (Authors could respond to these reviews in turn, either by revising their paper or explaining why they consider the criticisms misguided; the reviewers may subsequently revise their ratings. It's a dynamic process.)
Such feedback could be valuable to the authors. Insightful reviews could be reputation-enhancing for the reviewers. And, emerging from of all this public give-and-take, readers end up with a searchable database of cutting-edge philosophical research, complete with a (rough) metric of quality, to help us find especially interesting and important new papers. (One might variously browse by rating, download popularity, number of reviews/comments, etc.)
As you can probably tell, I like this idea a lot. Unfortunately, I don't have the technical know-how to set up such a thing. (Oisin suggests Drupal?) But I'd be willing to look into it further -- and see if I can get institutional support, etc. -- if there's sufficient interest out there to make such a project worthwhile. Any takers?