Open Sourced: Newscombinator Front End

[tomw1808]

What happens when you are on the HN top site (beside you open a beer and enjoy that little tiny moment after months of hard work):

a live snapshot http://i.imgur.com/0KbPjcx.png

[tomw1808]

Thank you everyone. Hard to describe the feelings, somewhere between moonwalk and break-dance.

another snapshot: http://i.imgur.com/CBYRvV3.png

server load peaking @2.00 seems to be stable

[PeterWhittaker]

Cool, I like it. One request: Start all stories collapsed. Right now, the Ellen Pao story is at the top, with about two dozen related headlines beneath it - that takes up a lot of screen space (every time I refresh I have to collapse that space).

It would also be good to be able to collapse/expand by clicking on the headline count.

I'm not sure how I feel about the continuous scroll. Often I like that, but here I am unsure, but it may be because of my particular use case: I use LinkVisitor to mark all links as visited, so that next time I know which I've already read or decided to ignore. Having a fixed number of headlines makes it easier for me to recognize that it is time to hit cmd-fn-f2. If others get a lot of value from the continuous scroll, don't worry about it, mine is likely quite the edge case.

[tomw1808]

Hey there, thanks for the feedback.

Its kind of a funny thing here with that Ellen Pao case. I didn't see similar stories for quite a while on the very top and exactly now there is one. Shouldn't be an excuse, rather something I literally just learned about my app. And if everything is collapsed at the very beginning, it also looks kind of odd (for me at least).

Thanks for the great feedback. Considering its an angular app, it shouldn't be that hard to have a switch to choose between endless scroll / normal pagination. But its a long way to go. I'll put it on the list :).

[kriro]

Seems really cool, it gets the great honour of being a pinned tab in my browser (ńext to HN).

I think a little icon that indicates the source next to the title (on the left) would be really useful to quickly scan what media ranks where and maybe only select the top stuff from your most trusted source etc.

[tomw1808]

That is, indeed, a great honor. I didn't expect my Saturday ending at home, monitoring the servers - in a good way.

The icon is a great idea, now the problem: What happens if a link was on 8 sites (2 x reddit, 4 x ycombinator, datatau, makernews). What would you expect next to the title?

[karbon]

I also have been exploring what it may mean to automagically categorize news content.. I took an approach on http://www.rivyr.com of following RSS feeds.. then reverse searching the titles against my product database to attempt to autoclassify news by product. Wish I could see your backend as that's where the secret sauce is but definitely understand not doing that ;)

[tomw1808]

That is super interesting. I'm actually working more closely with Apache OpenNLP for feature extraction and LibLINEAR to classify the news according to their category. Its not too far from coming out of alpha to an open beta version. Why not subscribe to the mailinglist and I'll shoot you more infos over in the upcoming weeks?

[serkanh]

This was something i was planning to build soon for myself. Great job on the UI.One thing kinda bugged me was when you select the source from the sidepanel it collapses for each selection not allowing to select another source without collapsing. I am on OSX Yosemite/Chrome-Version 41.0.2272.104. Also is the backend Elasticsearch? Is the backend elasticsearch?

[tomw1808]

Hi. First: thank you for the kind words. Second: I'm going to fix that, I know, it already annoyed me today and I just implemented the collapsed sources last week for better readability.

The backend is Solr in combination with a custom scraper in java and a lamp stack. The REST interface is wrapped around a php for better auth handling - and mainly because I'm a big fan of the Zend Framework and Pimcore [1] - one of the best CMS I've come across in my life (I'm not working for them).

Although I cannot give you access to the full source of the scraper/backend (at the moment), why not get in touch and/or follow on twitter/mailinglist. I'm going to send out updates what problems I was struggling on and how I solved it from time to time.

[1] https://www.pimcore.org/

[charlieegan3]

The related stories feature's cool. I made my own tool for reading HN (and some other sites) it's kind of different as it pulls the stories into a linear feed. I'll need to have a go at adding related stories. My site's http://www.serializer.io if you're interested.

[tomw1808]

I saw it today in the HN Notifications Thread (I think) and was already impressed, its great work, imho.

[zo1]

Btw, your "Btw, JSON anyone?" link is broken:

    {
      "error": {
        "msg": "undefined field: \"log\"",
        "code": 400
      }
    }

Otherwise, great work! Looks very nice!

[tomw1808]

Interesting, as it works normally for me. Which browser you're on, if I may ask?

It should just open a new tab with the same request as sent to the REST interface anyways in the background...

Like that one http://goo.gl/eLiUT1

[sandGorgon]

this is interesting - could you talk a bit about your solr schema ? In a very naive way I would be simply storing it all as a indexable text blob. Did you do anything more, just so that you can avoid a database ?

[tomw1808]

Sure. Well, basically Solr is just a flat index, although you can store now mutliple-values in one field and even make joins. Here is the clou: Everything is preprocessed and stored in a mysql-db and solr just "caches" everything. Its some kind of the flattened mysql schema. So I have a combination of a relational db and all the benefits with the huge bonus of the speed and reliability of solr in between. I started of with just a MySQL db, but real-time scoring is really a pain in the "back" with mysql and I couldn't figure a nice way to do it. On the other hand, if you expand a story, you see that the comments/upvotes from the sources are fetched, because joins like that are no possible in solr (read: I couldn't find a way to do it).

So, all the information is preprocessed at index time and solr is basically a text-blob spread across multiple fields, yes.

[gabrielgiordano]

Cool, thanks a lot :)

[_RPM]

why is it written in angular?

[enrmarc]

I'm really intrigued. Why not Angular?

[tomw1808]

why not?