Saturday, May 04, 2013

Play python with twitter streaming

I was interested in how to do twitter streaming. I tried to find out on the internet and got many examples. I tested it with python code(tweepy). I got data in JSON format. I thought it's a good idea, if I was able insert this data into mongodb.
So. I created my application on twitter and tested some a little bit code.
[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
> db.twittertest.find()
Used some code on the internet and changed a bit to insert data into database.
[surachart@centos ~]$ cat ./test-tweeter.py
#!/usr/bin/python
import tweepy
import pymongo
import json

consumer_key = "..."
consumer_secret = "..."
access_token_key = "..."
access_token_secret = "..."


auth1 = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth1.set_access_token(access_token_key, access_token_secret)

class StreamListener(tweepy.StreamListener):
    def on_status(self, tweet):
        print tweet.text

    def on_error(self, status_code):
        print 'Error: ' + repr(status_code)
        return False

    def on_data(self, data):
        print 'Ok! Inserting Data.'
        from pymongo import MongoClient
        client = MongoClient()
        client = MongoClient('localhost', 27017)
        db = client.test
        test_id = db.twittertest.insert(json.loads(data))

l = StreamListener()
streamer = tweepy.Stream(auth=auth1, listener=l)
streamer.filter(track=['oracle'])
I filtered by using word 'oracle'.
[surachart@centos ~]$ ./test-tweeter.py
Ok! Inserting Data.

[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
{ "_id" : ObjectId("51851511cdc8722152c28171"), "contributors" : null, "truncated" : false, "text" : "Job Opportunity for Oracle PL/SQL Developers with HTML/Web development experience for 6+ Months(Contract) for Las...: http://t.co/p6p8cfV6qF", "in_reply_to_status_id" : null, "id" : NumberLong("330684010961448962"), "favorite_count" : 0, "source" : "<a href=\"http://www.linkedin.com/\" rel=\"nofollow\">LinkedIn</a>", "retweeted" : false, "coordinates" : null, "entities" : { "symbols" : [ ], "user_mentions" : [ ], "hashtags" : [ ], "urls" : [     {       "url" : "http://t.co/p6p8cfV6qF",       "indices" : [   118,    140 ],  "expanded_url" : "http://lnkd.in/G-hhiX",       "display_url" : "lnkd.in/G-hhiX" } ] }, "in_reply_to_screen_name" : null, "id_str" : "330684010961448962", "retweet_count" : 0, "in_reply_to_user_id" : null, "favorited" : false, "user" : { "follow_request_sent" : null, "profile_use_background_image" : true, "default_profile_image" : true, "id" : 360409067, "verified" : false, "profile_image_url_https" : "https://si0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "profile_sidebar_fill_color" : "DDEEF6", "profile_text_color" : "333333", "followers_count" : 22, "profile_sidebar_border_color" : "C0DEED", "id_str" : "360409067", "profile_background_color" : "C0DEED", "listed_count" : 1, "profile_background_image_url_https" : "https://si0.twimg.com/images/themes/theme1/bg.png", "utc_offset" : null, "statuses_count" : 1777, "description" : null, "friends_count" : 0, "location" : "", "profile_link_color" : "0084B4", "profile_image_url" : "http://a0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "following" : null, "geo_enabled" : false, "profile_background_image_url" : "http://a0.twimg.com/images/themes/theme1/bg.png", "name" : "Anand P. Sharma", "lang" : "en", "profile_background_tile" : false, "favourites_count" : 0, "screen_name" : "anandph123", "notifications" : null, "url" : null, "created_at" : "Tue Aug 23 04:52:03 +0000 2011", "contributors_enabled" : false, "time_zone" : null, "protected" : false, "default_profile" : true, "is_translator" : false }, "geo" : null, "in_reply_to_user_id_str" : null, "possibly_sensitive" : false, "lang" : "en", "created_at" : "Sat May 04 14:03:06 +0000 2013", "filter_level" : "medium", "in_reply_to_status_id_str" : null, "place" : null }
>
I was able to insert data...  and I will play with it later.


No comments: