46
Get over 100 tweets using Twitter API v2
This is an example of implementing a tweet search using the Twitter API v2 using PHP.
There are no PHP implementation examples for retrieving more than 100 tweets at a time, so I'll give a brief introduction along with the specifications of the API itself. (Python and other languages are available)
Twitter API v2 is a new API that was released in August 2020.
Also, just last month, in mid-November, v2 became the default API, and it is recommended to migrate from v1.1.
In this article, we will focus mainly on PHP implementations.
The most famous Twitter API library, "TwittterOAuth", released version 3.0.0 in July of this year to support v2.
You will not be able to access with v1.1 credentials and will need to create a new v2 application, but if you have already passed the Developer application, you can issue it as soon as possible.
You can register an application for v2 from the left column of the Developer Portal.
By the way, if you make a request using the authentication key of the v1.1 app, it will kindly return with the registration URL, telling you that you need to register for v2, as shown below.
["detail"]=>
string(201) "When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal."
["registration_url"]=>
string(55) "https://developer.twitter.com/en/docs/projects/overview"
// ︙Endpoint Types
In v1.1, there were categories such as Premium plan, but in v2, it depends on the usage.
- /2/tweets/search/recent ... Get tweets from the past week.
- Full-archive search ... retrieve tweets from all time periods. Only for academic research users.
API request limits
The rate limit changes for the application and user authentication used, which is almost the same as v1.1.
| User | Application | |
|---|---|---|
| Request Limit | 180 / 15min | 450 / 15min |
| Tweets per once | 10-100 | 10-100 |
There is also a monthly limit of 2 million tweets per month for the entire v2 app. You can check the status of the limit from the Twitter Deleloper app page.
See Repo
Sample code is shown below, using PHP 7.x or 8.x with the Composer library installed.
The rate limit restriction process is usually a good way to get and check the limit status, but this time we will introduce a method to specify the number of times to access in advance. A bearer token with a loose limit is used.
If you are doing personal development or just want to get the information, you can record the number of requests made on the client side and stop the access when it reaches 150. If you want to do it properly, do it yourself as it is the same as v1.1.
<?php
require_once "vendor/autoload.php";
use Abraham\TwitterOAuth\TwitterOAuth;
$connection = new TwitterOAuth(
"************************", // API Key
"************************", // API Secret
null,
"************************" // Bearer Token
);
$query = "#keyword";
// Set API version
$connection->setApiVersion("2");
$params = [
/*
- Reference:https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent
- query and max_result are required
*/
"query" => $query . " -is:retweet", // if you exclute retweets
"start_time" => "2021-12-12T00:00:00", // UTC
"end_time" => "2021-12-03T00:00:00",
"tweet.fields" => "created_at", // additional posting date and time
"max_results" => 100 // set between 10-100
];
// Set the number of requests to $req_count
// set the number of requests to $loop_count
$req_count = 0;
$loop_count = 400;
for ($i = 0; $i < $loop_count; $i++) {
/**
* When request is limited
* We have set the time longer than 15 minutes and shorter than 450 times to allow for this.
* Please take care of these issues on your own.
*/
if($req_count > 400){
echo "Wait for 15 minutes";
$loop_count = 0;
sleep(1000000);
}
// Request using v2 API
$request = $connection->get("tweets/search/recent", $params);
// Process the acquired information
foreach ($request->data as $data) {
// Your code here
}
echo "Success Request";
// If you have more than 100 cases
if (isset($request->meta->next_token)) {
echo $req_count;
$req_count++;
// Add next_token to the query parameter array
$params["next_token"] = $request->meta->next_token;
} else {
// If not, quit
break;
}
}- Save the query parameters as an associative array $params with the conditions you want to retrieve.
- Make a request using twitteroauth
- Next_token is included in the JSON returned from the request, so acquire it.
- Put the next_token into the associative array
- Repeat the request using for or while
I think it's okay to sleep in the for as needed so as not to bother the Twitter side. It's working!
Please refer to the documentation for the parameters and information you can get. It is especially nice to be able to put the start and end time directly into the query.
Have a great New Year!
Twitter @ichii731 | GitHub
Code to get more than 100 items in the previous v1.1
Finally, a bonus. Here is a sample code to get more than 100 items with v1.1.
<?php
require "vendor/autoload.php";
use Abraham\TwitterOAuth\TwitterOAuth;
function tweet_loop($keyword, $loop_count)
{
global $CK, $CS, $AT, $ATS;
$connection = new TwitterOAuth($CK, $CS, $AT, $ATS);
$params = [
'q' => $keyword,
'count' => 100
];
for ($i = 0; $i < $loop_count; $i++) {
$results = $connection->get("search/tweets", $params);
foreach ($results->statuses as $val) {
// Write the post-acquisition process here
$tweet_results[] = $val->text;
}
//Conditional branching to see if there are more tweets that can be retrieved
if (isset($results->search_metadata->next_results)) {
// Get max_id
$max_id = preg_replace('/.*?max_id=([\d]+)&.*/', '$1', $results->search_metadata->next_results);
// Add max_id to params
$options['max_id'] = $max_id;
} else {
break;
}
}
return $tweet_results;
}46