Hacker News officially released their API this October, giving access to a vast amount of news articles, comments, polls, job postings, etc and via JSON, perfect to put it into Elasticsearch.
Elasticsearch is currently the most popular Open-Source search engine, used for a wide variety of use cases. It natively works with JSON documents so this sounds like a perfect fit.
It runs on a DigitalOcean 512MB droplet droplet and hosts the Elasticsearch node and a simple Tornado app for the frontend. Crontab runs the update every 5 minutes.
Set up Elasticsearch and make sure it's running at http://localhost:9200
See here if you need more information on how to install Elasticsearch.
I use Python and Tornado for the scripts to import and query the data.
We'll start with loading the Top 100 HN stories IDs, retrieve detailed information about each item and then index them in Elasticsearch.
Top 100 Stories:
curl https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty
the result looking something like this:
[ 8605204, 8604814, 8602936, 8604489, 8604533, 8604626, 8605207, 8605186,
...
8603147, 8602037 ]
We can now loop through the IDs and retrieve more detailed information:
curl https://hacker-news.firebaseio.com/v0/item/8605204.json?print=pretty
yields this:
{
"by" : "davecheney",
"id" : 8605204,
"kids" : [ 8605567, 8605461, 8605280, 8605824, 8605404, 8605601, 8605246, 8605323, 8605712, 8605346, 8605743, 8605242, 8605321, 8605268 ],
"score" : 260,
"text" : "",
"time" : 1415926359,
"title" : "Go is moving to GitHub",
"type" : "story",
"url" : "https://groups.google.com/forum/#!topic/golang-dev/sckirqOWepg"
}
And store the JSON document in Elasticsearch:
curl -XPUT http://localhost:9200/hn/story/***item['id']*** -d @doc.json
where ***item['id']*** is the ID of the document we just retrieved and @doc.json is the body of the document we just downloaded.
Check out the full Python code here: src/update.py
This is the loop over the top 100 IDs:
response = yield http_client.fetch('https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
top100_ids = json.loads(response.body)
for item_id in top100_ids:
yield download_and_index_item(item_id)
print "Done"
and this (shortened) piece downloads the individual items:
def download_and_index_item(item_id):
url = "https://hacker-news.firebaseio.com/v0/item/%s.json?print=pretty" % item_id
response = yield http_client.fetch(url)
item = json.loads(response.body)
# all sorts of clean-up of "item"
es_url = "http://localhost:9200/hn/%s/%s" % (item['type'], item['id'])
request = HTTPRequest(es_url, method="PUT", body=json.dumps(item), request_timeout=10)
response = yield http_client.fetch(request)
if not response.code in [200, 201]:
print "\nfailed to add item %s" % item['id']
else:
sys.stdout.write('.')
Once we have a batch of HN articles in ES we can run queries
curl "http://localhost:9200/hn/story/_search?pretty"
gives us all the stories (the first 10 really as ES defaults to 10 results by default).
All stories for a given user:
curl "http://localhost:9200/hn/story/_search?q=by:davecheney&pretty"
We can also run aggregations and for see who posted the most stories and what the most popular domains are:
curl -XGET 'http://localhost:9200/hn/story/_search?search_type=count' -d '
{ "aggs" : { "domains" : { "terms" : { "field" : "domain", "size": 11 } }, "by" : { "terms" : { "field" : "by", "size": 5 } } } }'
returning something like this:
{ "aggregations": {
"by": {
"buckets": [
{ "doc_count": 5,
"key": "luu" "},
{ "doc_count": 3,
"key": "benbreen" },
{ "doc_count": 3,
"key": "dnetesn" "},
...
]
},
"domains": {
"buckets": [
{ "doc_count": 6,
"key": "github.com" },
{ "doc_count": 4,
"key": "medium.com" },
...
]
}
}
}
Elasticsearch is doing a pretty good job at figuring out what type a field is but sometimes it can use a little help.
Run this query to see how ES maps each field of the story type:
curl -XGET 'http://localhost:9200/hn/_mapping/story'
Looks all pretty straight forward but one mapping sticks out:
"time": {
"type": "long"
},
The type long is ok but what we really want is the type date so we can take advantage of the built-in date operators and aggregations.
Let's set up a index mapping for time:
curl -XPUT "http://localhost:9200/hn/" -d '{
"mappings" : {
"story" : {
"properties" : {
"time" : { "type" : "date" }
}
}
}
}'
That should do the trick so now we can run a query to see how many stories are being posted to the HN Top 100 per week:
curl -XGET 'http://localhost:9200/hn/story/_search?search_type=count' -d '
{
"aggs" : {
"articles_over_time" : {
"date_histogram" : {
"field" : "time",
"interval" : "1w"
}
}
}
}
'
Result:
{ "aggregations": {
"articles_over_time": {
"buckets": [
{ "doc_count": 1609,
"key": 1413158400000,
"key_as_string": "2014-10-13T00:00:00.000Z"
},
{ "doc_count": 1195,
"key": 1413763200000,
"key_as_string": "2014-10-20T00:00:00.000Z"
},
{ "doc_count": 1236,
"key": 1414368000000,
"key_as_string": "2014-10-27T00:00:00.000Z"
},
{ "doc_count": 1304,
"key": 1414972800000,
"key_as_string": "2014-11-03T00:00:00.000Z"
}
] } },
}
- use bulk API
- more interesting queries
- simple web interface to query ES
Open pull requests, issues or email me at o@21zoo.com