Efficiently Supporting Frequent Content Updates in Games

The Need for Updates

Good games don’t stand still. Keeping players engaged requires there always to be more dungeons to explore, achievements to unlock, hats to purchase, boxes to be looted, etc…

New content may be carefully designed and curated, or it may be user-generated and infinitely plentiful. It is common to need to distribute large pack files containing multiple assets, as well as access a large number of small asset files on demand. In both cases, new content could be added anywhere from every few minutes to a few days.

Getting this new content to players after the initial installation of the game can be both technically challenging and costly. Where a streaming installation method has been used (as is common on mobile platforms due to app store restrictions), the same challenges can be encountered during the initial installation.

Making players wait longer than expected for large downloads can only lead to a sharp decline in engagement and, ultimately, a loss of revenue. So, the technical solutions employed must operate as efficiently as possible. Greater efficiency will not only improve player satisfaction but also reduce costs. Ultimately, games that can provide their core gameplay loop while assets are retrieved on demand, install quicker, start faster, and generally perform better.

Making it Possible, Making it Work

The key to making content delivery scale efficiently is designing the game and workflow to revolve around static immutable objects. This means that all new content should function in addition to the existing older content. If for any reason this is not possible, then at minimum within the distribution network a newer version of an object should be able to coexist with the version it is replacing.

When an asset, or asset bundle, has a globally unique identifier, it can be considered a single object for the purposes of distribution. It is uploaded to a bucket on the CacheFly Managed Object Storage platform (MOS).

The concept of a bucket is used to group objects together arbitrarily. Each game uses multiple buckets to separate different kinds of assets from each other. This is not only tidy but also allows for different delivery configurations per asset type to be easily implemented. This is vital for ensuring that a sudden spike of downloads for the game’s newly viral music track doesn’t make the rest of the game unplayable due to noisy neighbor issues.

Once game content objects are uploaded, downloading them from the CacheFly gaming optimized content delivery network (CDN) is as simple as a HTTP request.

How and when this request is made is specific to how assets are managed within the game engine. Such management is, of course, unique for each engine. With all engines, it is common to be able to give assets globally unique identifiers and to be able to load/stream them into the scene asynchronously. Delivering content at runtime is not tied to any specific game engine but can be implemented in all the major engines.

Unreal Engine and Godot both provide all the building blocks necessary for game developers to be able to write code to stream assets from a CDN dynamically. By requiring custom-written implementations, these engines provide the ultimate flexibility and fine-tuning potential. The Unity game engine, while allowing for custom implementations, provides a built-in no-code configurable solution with their Addressable Asset System (a.k.a. Addressables). Many other engines are similarly capable.

Content Protection

Content should be protected at the distribution level, even if this is just to reduce distribution costs. Although it doesn’t seem possible to completely eliminate game piracy, using secure channels to download content on demand raises the barrier for pirates in several ways.

There are several possible strategies, and the one chosen for a specific game will depend on the type of game, the game engine being used, and, ultimately, the needs of the game developer.

Unprotected Yet Obfuscated

Some games, especially those which are free to play, may intentionally not bother with any direct form of content protection.

It is noted that storing assets and asset bundles in objects on a CDN provides a level of obfuscation for accessing the content. This hurdle, combined with regular additional content releases, can encourage some of the users of a pirated game copy to switch to a licensed copy.

Protected with Encryption

Encryption-based protection works by compressing and encrypting each object before uploading it. The game client downloads the objects over a standard unencrypted HTTP connection and then decrypts them before loading them into the game.

A downside to this approach is that it does not prevent unauthorized downloads, which still incur a cost.

A significant benefit of this approach is that it allows local area network caching of game content. Internet Cafes, LAN Parties, and other similar institutions often do this using the popular open-source software Lancache. This improves the gaming experience for their customers and can significantly boost the game’s popularity, leading to increased revenue.

Protected with Signatures

A token authentication protection approach requires the game client to present a signature with each download request. Download requests must be made over an encrypted HTTPS connection to ensure the token/signature is not stolen. The CDN then refuses any request not accompanied by a valid signature.

As this limits downloads to only authorized players, this approach can help prevent piracy and reduce the ongoing costs associated with content distribution.

Management of Secrets

With both of these approaches, the game client needs a secret. This is often controlled through some form of digital rights management licensing service, often requiring the player to be logged in before the downloads can function. Although authentication of this kind provides unwanted friction, this can often be piggybacked off an existing integration which exists to provide achievements, leader boards, and other such functionality.

In some cases, games may choose to simply embed the secret in the game executable; although this is not recommended, it is arguably a step up from no protection at all. One major disadvantage is that changing the secret requires the game clients to be updated; however, doing so also breaks their ability to make downloads.

Storing User Generated Content

Games that support user-generated content (UGC) require a server to store and distribute the content. Although there is much more technical depth to UGC than simply storing it, without great storage and fast distribution, the whole player experience becomes lackluster and frustrating.

With asynchronous content downloading already implemented in a game, the ability to download user-generated content becomes primarily a matter of knowing how to locate and address it.

There are many ways to handle uploading UGC.

The CacheFly recommended approach is to have a game server issue an upload token to the game client.

This token is valid for uploading to a single object key within a bucket dedicated to UGC.

The game client then uploads the new content directly to object storage. Upon upload completion, the game client will notify the game server.

The server can then retrieve, validate, index, and process the content. The server will make the content available to other game clients only if the content is permitted. As the content is already present in the distribution network, clients can download and use it immediately.

The Infrastructure Behind the Delivery

The CacheFly gaming-optimized content delivery network (CDN) brings the content for the game geographically close to the players who need it anywhere in the world. The network is built from hundreds of servers, which collectively store copies of the content in many geographically diverse locations.

Many of these locations are large data centers in large cities, but many of them are located directly inside the networks of residential and mobile internet providers. For players using these providers, their requests for content never need to flow through the open internet. This significantly improves delivery times by eliminating uncertainty and avoiding congestion.

A multi-tiered object caching architecture allows for content to be placed in different storage mediums depending on how frequently it needs to be accessed. The most popular and essential objects are always the fastest to request, as they are stored in the quickest memory available.

A set of smart load balancers quickly reacts to the ever-changing environment of the internet to ensure resiliency and eliminate bottlenecks. The same systems also dynamically detect and mitigate DDoS attacks, keeping content online despite the intentions of bad actors.

The object storage platform and content delivery network are fully integrated at both the network and application levels. Using both together provides the best possible performance at the lowest possible cost. These qualities arise because requests never need to leave the CacheFly network.

With time until completion being the only metric players care about, CacheFly famously focuses on maximizing throughput and minimizing latency simultaneously. Latency is minimized by ensuring content is close to players. Throughput is maximized through intelligent storage, an abundance of capacity, and a proactive team ensuring maximum efficiency at every level of the service.

To learn more visit https://www.cachefly.com/gaming/.