I'm not going to go into details of why sharding is important for App Engine apps except for this quote from Google's article...
When developing an efficient application on Google App Engine, you need to pay attention to how often an entity is updated. While App Engine's datastore scales to support a huge number of entities, it is important to note that you can only expect to update any single entity or entity group about five times a second. That is an estimate and the actual update rate for an entity is dependent on several attributes of the entity, including how many properties it has, how large it is, and how many indexes need updating. While a single entity or entity group has a limit on how quickly it can be updated, App Engine excels at handling many parallel requests distributed across distinct entities, and we can take advantage of this by using sharding.
I'll get to the implementation of the counter shorty, but first lets have a look at the kind of data it will end up storing in the data store. There are two properties that store data - count and target. The count property is obvious, it stores the shard's counter value, the target property stores the entity that we're counting. The way I approached my implementation, the target is made up of the entity type concatenated with its ID. In reality the target can be any value you want to have a counter for.
As can be seen in the image above, the shards all have the same target. Their individual counter values differ because the total sharded counter value is made up of the sum of all of the shards. Another important point is the key name for the shards is made up of the target and the shard number. This makes it trivial to look up a shard directly and increment its value.
Let start looking at the code. First is the CounterShardEntity class. This class encapsulates the data that is going to be stored in the Data Store. There's not too much logic here. The constructor is used to set up the initial value of the entity - this is used the very first time the shard is created in the Data Store. It expects an array with keys (target, shardNum and count) as input. These are self explanatory.
The kind() function is used to get the Data Store entity kind based on the class name. It also prepends a prefix, KIND_PFX which is set to an empty string but can be customised to any string as desired.
The schema() function defines the data schema for this entity, which is a good practice when working with php-gds. This ensures that the correct properties and their types are set up and also defines the PHP class to be used with this entity.
CounterShardEntity.php
<?php
namespace IK;
class CounterShardEntity extends \GDS\Entity {
const KIND_PFX = '';
public function __construct(array $params = null) {
if ($params != null) {
$this->setKeyName($params['target'] . '_' . $params['shardNum']);
$this->target = $params['target'];
$this->count = $params['count'];
}
}
public static function kind() {
return self::KIND_PFX . str_replace('\\', '_', __CLASS__);
}
public static function schema() {
return (new \GDS\Schema(self::kind()))
->setEntityClass('\\' . __CLASS__)
->addString ('target', true)
->addInteger ('count', false);
;
}
}
?>
The next class is CounterShardDS. This is what is going to be used to interact with the counter. This class serves as the Data Store interface to the CounterShardEntity instances. The NUM_SHARDS constant defines how many shards the counter will have. Once in use this constant can be increased but never decreased in value. The constructor doesn't do much apart from instantiating a php-gds DataStore class for the CounterShardEntity and recording the target for later use.
The getCount() function queries all of the counter shards for the same target and adds up their counters to return the total count. It's actually quite straight forward.
The increment() function is a little more complex but not by much. It generates a random shard number and tries to look that shard up. At the same time it gets the overall count. The shard is looked up by key name in the data store - if it's found, great, otherwise a new CounterShardEntity is created with 0 initial count. The shard counter (existing or new) is then incremented and the shard is stored in the Data Store via the upsert() function. The return value is the current count plus 1.
CounterShardDS.php
<?php
namespace IK;
class CounterShardDS {
const NUM_SHARDS = 20;
private $data;
private $target;
public function __construct($targetType, $targetId) {
$this->data = new \GDS\Store(CounterShardEntity::schema());
$this->target = $targetType . '_' . $targetId;
}
public function getCount() {
$gql = 'SELECT * FROM ' . CounterShardEntity::kind() . ' WHERE target = @target';
$this->data->query($gql, ['target' => $this->target]);
$shards = $this->data->fetchAll();
$count = 0;
if ($shards != null) {
foreach ($shards as $shard) {
$count += $shard->count;
}
}
return $count;
}
public function increment() {
$shardNum = rand(1, self::NUM_SHARDS);
$curCount = $this->getCount();
$shard = $this->data->fetchByName($this->target . '_' . $shardNum);
if ($shard == null) {
$shard = new CounterShardEntity([
'target' => $this->target,
'shardNum' => $shardNum,
'count' => 0
]);
}
$shard->count++;
$this->data->upsert($shard);
return $curCount + 1;
}
}
?>
Now that that is out of the way it's time to look at how the counter is used. This code demonstrates looking up the current value of the counter and then incrementing it...
counter_test.php
<?php
require_once __DIR__ . '/vendor/autoload.php';
require_once 'CounterShardEntity.php';
require_once 'CounterShardDS.php';
$counter = new \IK\CounterShardDS('Counter Test', 1);
error_log('Current counter value: ' . $counter->getCount());
error_log('Next counter value: ' . $counter->increment());
?>
It should be easy to follow. The counter is an instance of the CounterShardDS class. When creating it, the entity type and ID are passed in and that's all that's required. The rest is self explanatory.
So there you go, this is quite a versatile sharded counter implementation for the Google Data Store in PHP. It can be used to count on pretty much anything as long as its type and ID can be specified.
-i
