Skip to content

Support for the maxlag parameter (with retries) in API requests (fixes #92) #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
waldyrious merged 5 commits into from
Aug 23, 2021
Merged

Support for the maxlag parameter (with retries) in API requests (fixes #92) #112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ca4b09f
Add maxlag & retries properties and get/set methods
Xymph Jul 7, 2021
8fc61be
Add Wikimate request() method with maxlag mechanism
Xymph Jul 8, 2021
e8046f8
Switch API calls to new request() method
Xymph Jul 8, 2021
6cf7ce2
Add Wikimate note about maxlag exception; add changelog entry
Xymph Jul 8, 2021
3a7bbd0
Apply code and comment fixes per review suggestions
Xymph Aug 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Since v0.10.0 this project adheres to [Semantic Versioning](http://semver.org/)

#### Added

* Support for the maxlag parameter (with retries) in API requests ([#112])
* Support for getting/setting user agent for API requests ([#107])
* Added missing PHPDoc comments for properties, constants, and more ([#109])

Expand Down Expand Up @@ -129,4 +130,5 @@ Since v0.10.0 this project adheres to [Semantic Versioning](http://semver.org/)
[#108]: https://github.com/hamstar/Wikimate/pull/108
[#109]: https://github.com/hamstar/Wikimate/pull/109
[#111]: https://github.com/hamstar/Wikimate/pull/111
[#112]: https://github.com/hamstar/Wikimate/pull/112
[#114]: https://github.com/hamstar/Wikimate/pull/114
166 changes: 153 additions & 13 deletions Wikimate.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@
/**
* Provides an interface over wiki API objects such as pages and files.
*
* All requests to the API can throw an exception if the server is lagged
* and a finite number of retries is exhausted. By default requests are
* retried indefinitely. See {@see Wikimate::request()} for more information.
*
* @author Robert McLeod & Frans P. de Vries
* @since 0.2 December 2010
*/
Expand Down Expand Up @@ -39,6 +43,14 @@ class Wikimate
*/
const TOKEN_LOGIN = 'login';

/**
* Default lag value in seconds
*
* @var integer
* @link https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Maxlag_parameter
*/
const MAXLAG_DEFAULT = 5;

/**
* Base URL for API requests
*
Expand Down Expand Up @@ -93,6 +105,21 @@ class Wikimate
*/
protected $debugMode = false;

/**
* Maximum lag in seconds to accept in requests
*
* @var integer
* @link https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Maxlag_parameter
*/
protected $maxlag = self::MAXLAG_DEFAULT;

/**
* Maximum number of retries for lagged requests (-1 = retry indefinitely)
*
* @var integer
*/
protected $maxretries = -1;

/**
* Creates a new Wikimate object.
*
Expand Down Expand Up @@ -125,6 +152,78 @@ protected function initRequests()
$this->session->useragent = $this->useragent;
}

/**
* Sends a GET or POST request in JSON format to the API.
*
* This method handles maxlag errors as advised at:
* {@see https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Maxlag_parameter)
* The request is sent with the current maxlag value
* (default: 5 seconds, per MAXLAG_DEFAULT).
* If a lag error is received, the method waits (sleeps) for the
* recommended time (per the Retry-After header), then tries again.
* It will do this indefinitely unless the number of retries is limited,
* in which case an exception is thrown once the limit is reached.
*
* The string type for $data is used only for upload POST requests,
* and must contain the complete multipart body, including maxlag.
*
* @param array|string $data Data for the request
* @param array $headers Optional extra headers to send with the request
* @param boolean $post True to send a POST request, otherwise GET
* @return Requests_Response The API response
* @throw Exception If lagged and ran out of retries
*/
private function request($data, $headers = array(), $post = false)
{
$retries = 0;

// Add format & maxlag parameter to request
if (is_array($data)) {
$data['format'] = 'json';
$data['maxlag'] = $this->getMaxlag();
}

// Send appropriate type of request, once or multiple times
do {
if ($post) {
$response = $this->session->post($this->api, $headers, $data);
} else {
$response = $this->session->get($this->api.'?'.http_build_query($data), $headers);
}

// Check for replication lag error
$server_lagged = ($response->headers->offsetGet('X-Database-Lag') !== null);
if ($server_lagged) {
// Determine recommended or default delay
if ($response->headers->offsetGet('Retry-After') !== null) {
$sleep = (int)$response->headers->offsetGet('Retry-After');
} else {
$sleep = $this->getMaxlag();
}

if ($this->debugMode) {
preg_match('/Waiting for [^ ]*: ([0-9.-]+) seconds? lagged/', $response->body, $match);
Comment thread
waldyrious marked this conversation as resolved.
echo "Server lagged for {$match[1]} seconds; will retry in {$sleep} seconds\n";
}
sleep($sleep);

// Check retries limit
if ($this->getMaxretries() >= 0) {
$retries++;
} else {
$retries = -1; // continue indefinitely
}
}
} while ($server_lagged && $retries <= $this->getMaxretries());

// Throw exception if we ran out of retries
if ($server_lagged) {
throw new Exception("Server lagged ($retries consecutive maxlag responses)");
} else {
return $response;
}
}

/**
* Obtains a wiki token for logging in or data-modifying actions.
* For now this method, in Wikimate tradition, is kept simple and supports
Expand All @@ -149,11 +248,11 @@ protected function token($type = self::TOKEN_DEFAULT)
'action' => 'query',
'meta' => 'tokens',
'type' => $type,
'format' => 'json'
);

// Send the token request
$response = $this->session->post($this->api, array(), $details);
$response = $this->request($details, array(), true);

// Check if we got an API result or the API doc page (invalid request)
if (strpos($response->body, "This is an auto-generated MediaWiki API documentation page") !== false) {
$this->error = array();
Expand Down Expand Up @@ -204,7 +303,6 @@ public function login($username, $password, $domain = null)
'lgname' => $username,
'lgpassword' => $password,
'lgtoken' => $logintoken,
'format' => 'json'
);

// If $domain is provided, set the corresponding detail in the request information array
Expand All @@ -213,7 +311,8 @@ public function login($username, $password, $domain = null)
}

// Send the login request
$response = $this->session->post($this->api, array(), $details);
$response = $this->request($details, array(), true);

// Check if we got an API result or the API doc page (invalid request)
if (strpos($response->body, "This is an auto-generated MediaWiki API documentation page") !== false) {
$this->error = array();
Expand Down Expand Up @@ -253,6 +352,50 @@ public function login($username, $password, $domain = null)
return true;
}

/**
* Gets the current value of the maxlag parameter.
*
* @return integer The maxlag value in seconds
*/
public function getMaxlag()
{
return $this->maxlag;
}

/**
* Sets the new value of the maxlag parameter.
*
* @param integer $ml The new maxlag value in seconds
* @return Wikimate This object
*/
public function setMaxlag($ml)
{
$this->maxlag = (int)$ml;
return $this;
}

/**
* Gets the current value of the max retries limit.
*
* @return integer The max retries limit
*/
public function getMaxretries()
{
return $this->maxretries;
}

/**
* Sets the new value of the max retries limit.
*
* @param integer $mr The new max retries limit
* @return Wikimate This object
*/
public function setMaxretries($mr)
{
$this->maxretries = (int)$mr;
return $this;
}

/**
* Gets the user agent for API requests.
*
Expand Down Expand Up @@ -361,13 +504,12 @@ public function getFile($filename)
public function query($array)
{
$array['action'] = 'query';
$array['format'] = 'json';

if ($this->debugMode) {
echo "query GET parameters:\n";
echo http_build_query($array) . "\n";
}
$apiResult = $this->session->get($this->api.'?'.http_build_query($array));
$apiResult = $this->request($array);

if ($this->debugMode) {
echo "query GET response:\n";
Expand All @@ -386,13 +528,12 @@ public function query($array)
public function parse($array)
{
$array['action'] = 'parse';
$array['format'] = 'json';

if ($this->debugMode) {
echo "parse GET parameters:\n";
echo http_build_query($array) . "\n";
}
$apiResult = $this->session->get($this->api.'?'.http_build_query($array));
$apiResult = $this->request($array);

if ($this->debugMode) {
echo "parse GET response:\n";
Expand Down Expand Up @@ -420,14 +561,13 @@ public function edit($array)
);

$array['action'] = 'edit';
$array['format'] = 'json';
$array['token'] = $edittoken;

if ($this->debugMode) {
echo "edit POST parameters:\n";
print_r($array);
}
$apiResult = $this->session->post($this->api, $headers, $array);
$apiResult = $this->request($array, $headers, true);

if ($this->debugMode) {
echo "edit POST response:\n";
Expand Down Expand Up @@ -455,14 +595,13 @@ public function delete($array)
);

$array['action'] = 'delete';
$array['format'] = 'json';
$array['token'] = $deletetoken;

if ($this->debugMode) {
echo "delete POST parameters:\n";
print_r($array);
}
$apiResult = $this->session->post($this->api, $headers, $array);
$apiResult = $this->request($array, $headers, true);

if ($this->debugMode) {
echo "delete POST response:\n";
Expand Down Expand Up @@ -510,6 +649,7 @@ public function upload($array)

$array['action'] = 'upload';
$array['format'] = 'json';
$array['maxlag'] = $this->getMaxlag();
$array['token'] = $uploadtoken;

// Construct multipart body:
Expand Down Expand Up @@ -541,7 +681,7 @@ public function upload($array)
'Content-Length' => strlen($body),
);

$apiResult = $this->session->post($this->api, $headers, $body);
$apiResult = $this->request($body, $headers, true);

if ($this->debugMode) {
echo "upload POST response:\n";
Expand Down