This is a quick update to my post about a year ago, with details on how to mine Twitter streams in real-time using PHP. This new code includes updates for the v1.1 API, including authentication using OAuth.
The first thing you need to do is sign in to the Twitter developer portal with your Twitter account here: https://dev.twitter.com/user/login
Once you’ve logged in, click on your profile icon in the top right hand corner, select
“My applications”, and create a new application if you don’t already have one.
Select the option to create the access token as well, as the requests need to be signed by a Twitter account.
The Code
ctwitter_stream.php
class ctwitter_stream
{
private $m_oauth_consumer_key;
private $m_oauth_consumer_secret;
private $m_oauth_token;
private $m_oauth_token_secret;
private $m_oauth_nonce;
private $m_oauth_signature;
private $m_oauth_signature_method = 'HMAC-SHA1';
private $m_oauth_timestamp;
private $m_oauth_version = '1.0';
public function __construct()
{
//
// set a time limit to unlimited
//
set_time_limit(0);
}
//
// set the login details
//
public function login($_consumer_key, $_consumer_secret, $_token, $_token_secret)
{
$this->m_oauth_consumer_key = $_consumer_key;
$this->m_oauth_consumer_secret = $_consumer_secret;
$this->m_oauth_token = $_token;
$this->m_oauth_token_secret = $_token_secret;
//
// generate a nonce; we're just using a random md5() hash here.
//
$this->m_oauth_nonce = md5(mt_rand());
return true;
}
//
// process a tweet object from the stream
//
private function process_tweet(array $_data)
{
print_r($_data);
return true;
}
//
// the main stream manager
//
public function start(array $_keywords)
{
while(1)
{
$fp = fsockopen("ssl://stream.twitter.com", 443, $errno, $errstr, 30);
if (!$fp)
{
echo "ERROR: Twitter Stream Error: failed to open socket";
} else
{
//
// build the data and store it so we can get a length
//
$data = 'track=' . rawurlencode(implode($_keywords, ','));
//
// store the current timestamp
//
$this->m_oauth_timestamp = time();
//
// generate the base string based on all the data
//
$base_string = 'POST&' .
rawurlencode('https://stream.twitter.com/1.1/statuses/filter.json') . '&' .
rawurlencode('oauth_consumer_key=' . $this->m_oauth_consumer_key . '&' .
'oauth_nonce=' . $this->m_oauth_nonce . '&' .
'oauth_signature_method=' . $this->m_oauth_signature_method . '&' .
'oauth_timestamp=' . $this->m_oauth_timestamp . '&' .
'oauth_token=' . $this->m_oauth_token . '&' .
'oauth_version=' . $this->m_oauth_version . '&' .
$data);
//
// generate the secret key to use to hash
//
$secret = rawurlencode($this->m_oauth_consumer_secret) . '&' .
rawurlencode($this->m_oauth_token_secret);
//
// generate the signature using HMAC-SHA1
//
// hash_hmac() requires PHP >= 5.1.2 or PECL hash >= 1.1
//
$raw_hash = hash_hmac('sha1', $base_string, $secret, true);
//
// base64 then urlencode the raw hash
//
$this->m_oauth_signature = rawurlencode(base64_encode($raw_hash));
//
// build the OAuth Authorization header
//
$oauth = 'OAuth oauth_consumer_key="' . $this->m_oauth_consumer_key . '", ' .
'oauth_nonce="' . $this->m_oauth_nonce . '", ' .
'oauth_signature="' . $this->m_oauth_signature . '", ' .
'oauth_signature_method="' . $this->m_oauth_signature_method . '", ' .
'oauth_timestamp="' . $this->m_oauth_timestamp . '", ' .
'oauth_token="' . $this->m_oauth_token . '", ' .
'oauth_version="' . $this->m_oauth_version . '"';
//
// build the request
//
$request = "POST /1.1/statuses/filter.json HTTP/1.1\r\n";
$request .= "Host: stream.twitter.com\r\n";
$request .= "Authorization: " . $oauth . "\r\n";
$request .= "Content-Length: " . strlen($data) . "\r\n";
$request .= "Content-Type: application/x-www-form-urlencoded\r\n\r\n";
$request .= $data;
//
// write the request
//
fwrite($fp, $request);
//
// set it to non-blocking
//
stream_set_blocking($fp, 0);
while(!feof($fp))
{
$read = array($fp);
$write = null;
$except = null;
//
// select, waiting up to 10 minutes for a tweet; if we don't get one, then
// then reconnect, because it's possible something went wrong.
//
$res = stream_select($read, $write, $except, 600, 0);
if ( ($res == false) || ($res == 0) )
{
break;
}
//
// read the JSON object from the socket
//
$json = fgets($fp);
//
// look for a HTTP response code
//
if (strncmp($json, 'HTTP/1.1', 8) == 0)
{
$json = trim($json);
if ($json != 'HTTP/1.1 200 OK')
{
echo 'ERROR: ' . $json . "\n";
return false;
}
}
//
// if there is some data, then process it
//
if ( ($json !== false) && (strlen($json) > 0) )
{
//
// decode the socket to a PHP array
//
$data = json_decode($json, true);
if ($data)
{
//
// process it
//
$this->process_tweet($data);
}
}
}
}
fclose($fp);
sleep(10);
}
return;
}
};
The “process_tweet()” method will be called for each matching tweet- just modify that method to process the tweet however you want (load it into a database, print it to screen, email it, etc). The keyword matching isn’t perfect- if you search for a string of words, it won’t necessarily match the words in that exact order, but you can check that yourself from the process_tweet() method.
Then create a simple PHP application to run the collector:
require 'ctwitter_stream.php';
$t = new ctwitter_stream();
$t->login('consumer_key', 'consumer secret', 'access token', 'access secret');
$t->start(array('facebook', 'fbook', 'fb'));
You’ll need to provide the Consumer Key, Consumer Secret, Access Token, and the Access Secret, all of which are available from the Details section of your Application.
This new class uses the PHP hash_hmac() function for OAuth, which is available only in PHP 5.2.1 and up, and in the PECL hash extension 1.1 and up.
You can also Download the file here: http://mikepultz.com/uploads/ctwitter_stream.php.zip