Storage ( PR #93 )

1. Introduction

Over the years the web has grown various APIs that can be used for storage, e.g., IndexedDB, localStorage , and showNotification() . The Storage Standard consolidates these APIs by defining:

• A bucket, the primitive these APIs store their data in
• A way of making that bucket persistent
• A way of getting usage and quota estimates for an origin

Traditionally, as the user runs out of storage space on their device, the data stored with these APIs gets lost without the user being able to intervene. However, persistent buckets cannot be cleared without consent by the user. This thus brings data guarantees users have enjoyed on native platforms to the web.

navigator.storage.persist().then(persisted => {
  if (persisted) {
    /* … */
  }
});

Promise.all([
  navigator.storage.persisted(),
  navigator.permissions.query({name: "persistent-storage"})
]).then(([persisted, permission]) => {
  if (!persisted && permission.status == "granted") {
    navigator.storage.persist().then( /* … */ );
  } else if (!persisted && permission.status == "prompt") {
    showPersistentStorageExplanation();
  }
});

function retrieveNextChunk(nextChunkInfo) {
  return navigator.storage.estimate().then(info => {
    if (info.quota - info.usage > nextChunkInfo.size) {
      return fetch(nextChunkInfo.url);
    } else {
      throw new Error("insufficient space to store next chunk");
    }
  }).then( /* … */ );
}

2. Terminology

This specification depends on the Infra Standard. [INFRA]

This specification uses terminology from the DOM, HTML, IDL, Permissions API, and URL Permissions Standards. [DOM] [HTML] [WEBIDL] [PERMISSIONS] [URL]

3. Lay of the land

A user agent has various kinds of semi-persistent state:

Credentials

End-user credentials, such as username and passwords submitted through HTML forms

Permissions

Permissions for various features, such as geolocation

Network

HTTP cache, cookies, authentication entries, TLS client certificates

Storage

Indexed DB, Cache API, service worker registrations, localStorage , sessionStorage , application caches, notifications, etc.

This standard primarily concerns itself with storage.

4. Model

Standards defining local or session storage APIs will define a storage endpoint and register it by changing this standard. They will use either obtain a local storage bottle map or obtain a session storage bottle map , which will give them:

• Failure, which might mean the API has to throw or otherwise indicate there is no storage available for that environment settings object .

• A storage proxy map that operates analogously to a map , which can be used to store data in a manner that suits the API. This standard takes care of isolating that data from other APIs, storage keys , and storage types .

If you are defining a standard for such an API, consider filing an issue agains this standard for assistance and review.

To isolate this data this standard defines a storage shed which segments storage shelves by a storage key . A storage shelf in turn consists of a storage bucket and will likely consist of multiple storage buckets in the future to allow for different storage policies. And lastly, a storage bucket consists of storage bottles , one for each storage endpoint .

4.1. Storage endpoints

A storage endpoint is a local or session storage API that uses the infrastructure defined by this standard to keep track of its storage needs.

A storage endpoint has an identifier , which is a storage identifier .

A storage endpoint also has types , which is a set of storage types .

A storage identifier is an ASCII string .

A storage type is " local " or " session ".

The registered storage endpoints are a set of storage endpoints defined by the following table:

As mentioned, standards can use these storage identifiers with obtain a local storage bottle map and obtain a session storage bottle map . It is anticipated that some APIs will be applicable to both storage types going forward.

4.2. Storage sheds keys

A storage key is an origin . [HTML]

This is expected to change, change; see Client-Side Storage Partitioning .

To obtain a storage key , given an environment settings object environment , run these steps:

1. Let key be environment ’s origin .

2. If key is an opaque origin , then return failure.

3. If the user has disabled storage, then return failure.

4. Return key .

4.3. Storage sheds

A storage shed is a map of storage keys to storage shelves . It is initially empty.

A user agent holds a storage shed , which is a storage shed . A user agent’s storage shed holds all local storage data.

A browsing session holds a storage shed , which is a storage shed . A browsing session’s session ’s storage shed holds all session storage data.

To legacy-clone a browsing session storage shed for defining , given a browsing session. It is roughly analogous to top-level session A and a browsing context session except that it cannot B , run these steps:

1. For each key → shelf of A ’s storage shed :

   1. Let newShelf be replaced due the result of running create a storage shelf with " session ".

   2. Set newShelf ’s bucket map [" default "]'s bottle map [" sessionStorage "]'s map to a clone of shelf ’s bucket map [" Cross-Origin-Opener-Policy default "]'s bottle map [" sessionStorage "]'s map .

   3. Set B ’s storage shed [ key ] to newShelf .

This is considered legacy as the benefits, if any, do not outweigh the implementation complexity. And therefore it will not be expanded or navigation. used outside of HTML . [HTML]

4.3. 4.4. Storage shelves

A storage shelf exists for each storage key within a storage shed . It holds a bucket map , which is a map of strings to storage buckets .

For now " default " is the only key that exists in a bucket map . See issue #2 . It is given a value when a storage shelf is obtained for the first time.

To obtain a storage shelf , given a storage shed shed , an environment settings object environment , and a storage type type , run these steps:

1. Let key be the result of running obtain a storage key with environment ’s origin . .

2. If key is an opaque origin , then return failure. If the user has disabled storage, failure, then return failure.

3. If shed [ key ] does not exist , then: Let then set shelf shed be a new storage shelf . Set [ shelf key ’s bucket map [" default "] ] to the result of running create a storage bucket shelf with type .

4. Set shed [ key ] to shelf . Return shed [ key ].

To obtain a local storage shelf , given an environment settings object environment , return the result of running obtain a storage shelf with the user agent’s storage shed , environment , and " local ".

To create a storage shelf , given a storage type type , run these steps:

1. Let shelf be a new storage shelf .

2. Set shelf ’s bucket map [" default "] to the result of running create a storage bucket with type .

3. Return shelf .

4.4. 4.5. Storage buckets

A storage bucket is a place for storage endpoints to store data. Whenever a storage bucket is cleared by the user agent, it must be cleared in its entirety.

A storage bucket has a bottle map of storage identifiers to storage bottles .

A local storage bucket is a storage bucket for local storage APIs.

A local storage bucket has a mode , which is " best-effort " or " persistent ". It is initially " best-effort ".

A session storage bucket is a storage bucket for session storage APIs.

To create a storage bucket , given a storage type type , run these steps:

1. Let bucket be null.

2. If type is " local ", then set bucket to a new local storage bucket .

3. Otherwise:

   1. Assert: type is " session ".

   2. Set bucket to a new session storage bucket .

4. For each endpoint of registered storage endpoints whose types contain type , set bucket ’s bottle map [ endpoint ’s identifier ] to a new storage bottle .

5. Return bucket .

4.5. 4.6. Storage bottles

A storage bottle is a part of a storage bucket carved out for a single storage endpoint . A storage bottle has a map , which is initially an empty map . A storage bottle also has a proxy map reference set , which is initially an empty set .

A storage bottle ’s map is where the actual data meant to be stored lives. User agents are expected to store this data, and make it available across agent and even agent cluster boundaries, in an implementation-defined manner, so that this standard and standards using this standard can access the contents.

To obtain a storage bottle map , given a storage type type , environment settings object environment , and storage identifier identifier , run these steps:

1. Let shed be null.

2. If type is " local ", then set shed to the user agent’s storage shed .

3. Otherwise:

   1. Assert: type is " session ".

   2. Set shed to environment ’s browsing session ’s storage shed .

4. Let shelf be the result of running obtain a storage shelf , with shed , environment , and type .

5. If shelf is failure, then return failure.

6. Let bucket be shelf ’s bucket map [" default "].

7. Let bottle be bucket ’s bottle map [ identifier ].

8. Let proxyMap be a new storage proxy map whose backing map is bottle ’s map .

9. Append proxyMap to bottle ’s proxy map reference set .

10. Return proxyMap .

To obtain a local storage bottle map , given an environment settings object environment and storage identifier identifier , return the result of running obtain a storage bottle map with " local ", environment , and identifier .

To obtain a session storage bottle map , given an environment settings object environment and storage identifier identifier , return the result of running obtain a storage bottle map with " session ", environment , and identifier .

4.6. 4.7. Storage proxy maps

A storage proxy map is equivalent to a map , except that all operations are instead performed on its backing map .

This allows for the backing map to be replaced. This is needed for issue #4 and potentially the Storage Access API .

5. Persistence permission

A local storage bucket can only have its mode change to " persistent " if the user (or user agent on behalf of the user) has granted permission to use the "persistent-storage" feature.

When granted to an origin , the persistence permission can be used to protect storage from the user agent’s clearing policies. The user agent cannot clear storage marked as persistent without involvement from the origin or user. This makes it particularly useful for resources the user needs to have available while offline or resources the user creates locally.

The " persistent-storage " powerful feature ’s permission-related flags, algorithms, and types are defaulted, except for:

permission state

"persistent-storage" 's permission state must have the same value for all environment settings objects with a given origin .

permission revocation algorithm

1. If "persistent-storage" 's permission state is "granted" , then return.

2. Let shelf be the result of running obtain a local storage shelf with current settings object .

3. Set shelf ’s bucket map [" default "]'s mode to " best-effort ".

6. Usage and quota

The storage usage of a storage shelf is a rough estimate of the amount of bytes used by it.

This cannot be an exact amount as user agents might, and are encouraged to, use deduplication, compression, and other techniques that obscure exactly how much bytes a storage shelf uses.

The storage quota of a storage shelf is a conservative estimate of the amount of bytes available to it. This amount should be less than the total available storage space on the device to give users some wiggle room.

User agents are strongly encouraged to consider navigation frequency, recency of visits, bookmarking, and permission for "persistent-storage" when evaluating quotas.

7. Management

Whenever a storage bucket is cleared by the user agent, it must be cleared in its entirety. User Interface Guidelines agents should avoid clearing storage buckets while script that is able to access them is running, unless instructed otherwise by the user.

If removal of storage buckets leaves the encompassing storage shelf ’s bucket map empty , then remove that storage shelf and corresponding storage key from the encompassing storage shed .

7.1. Storage pressure

A user agent that comes under storage pressure should clear network state and local storage buckets whose mode is " best-effort ", ideally prioritizing removal in a manner that least impacts the user.

If a user agent continues to be under storage pressure, then the user agent should inform the user and offer a way to clear the remaining local storage buckets , i.e., those whose mode is " persistent ".

Session storage buckets must be cleared as browsing sessions are closed.

If the user agent allows for revival of browsing sessions , e.g., through reopening browsing sessions or continued use of them after restarting the user agent, then clearing necessarily involves a more complex set of heuristics.

7.2. User interface guidelines

User agents should not distinguish between network state and storage in their user interface. Instead user agents should offer users the ability to remove clear all storage for each group of schemelessly same site origins . This ensures to some extent that network state cannot be used to revive storage. This also reduces the amount users need to know about the different ways in which websites can store data.

Credentials should be separated as they contain data the user might not be able to revive, such as an autogenerated password. Permissions are best separated too to avoid inconveniencing the user. Credentials and permissions are also somewhat easier to understand and differentiate for users from network state and storage. 7.1. Storage Pressure When the user agent notices it comes under storage pressure and it cannot free up sufficient space by clearing network state, session storage buckets , and local storage buckets whose mode is " best-effort ", then the user agent should inform the user and offer a way to clear the remaining local storage buckets , i.e., those whose mode is " persistent ".

8. API

[SecureContext]
interface mixin NavigatorStorage {
  [SameObject] readonly attribute StorageManager storage;
};
Navigator includes NavigatorStorage;
WorkerNavigator includes NavigatorStorage;

Each environment settings object has an associated StorageManager object. [HTML]

The storage getter steps are to return this ’s relevant settings object ’s StorageManager object.

[SecureContext,
 Exposed=(Window,Worker)]
interface StorageManager {
  Promise<boolean> persisted();
  [Exposed=Window] Promise<boolean> persist();
  Promise<StorageEstimate> estimate();
};
dictionary StorageEstimate {
  unsigned long long usage;
  unsigned long long quota;
};

The persisted() method steps are:

1. Let promise be a new promise.

2. Let shelf be the result of running obtain a local storage shelf with this ’s relevant settings object .

3. If shelf is failure, then reject promise with a TypeError .

4. Otherwise, run these steps in parallel :

   1. Let persisted be true if shelf ’s bucket map [" default "]'s mode is " persistent "; otherwise false.

      It will be false when there’s an internal error.

   2. Queue a task to resolve promise with persisted .

5. Return promise .

The persist() method steps are:

1. Let promise be a new promise.

2. Let shelf be the result of running obtain a local storage shelf with this ’s relevant settings object .

3. If shelf is failure, then reject promise with a TypeError .

4. Otherwise, run these steps in parallel :

   1. Let permission be the result of requesting permission to use "persistent-storage" .

      User agents are encouraged to not let the user answer this question twice for the same origin around the same time and this algorithm is not equipped to handle such a scenario.

   2. Let bucket be shelf ’s bucket map [" default "].

   3. Let persisted be true if bucket ’s mode is " persistent "; otherwise false.

      It will be false when there’s an internal error.

   4. If persisted is false and permission is "granted" , then:

      1. Set bucket ’s mode to " persistent ".

      2. If there was no internal error, then set persisted to true.

   5. Queue a task to resolve promise with persisted .

5. Return promise .

The estimate() method steps are:

1. Let promise be a new promise.

2. Let shelf be the result of running obtain a local storage shelf with this ’s relevant settings object .

3. If shelf is failure, then reject promise with a TypeError .

4. Otherwise, run these steps in parallel :

   1. Let usage be storage usage for shelf .

   2. Let quota be storage quota for shelf .

   3. Let dictionary be a new StorageEstimate dictionary whose usage member is usage and quota member is quota .

   4. If there was an internal error while obtaining usage and quota , then queue a task to reject promise with a TypeError .

      Internal errors are supposed to be extremely rare and indicate some kind of low-level platform or hardware fault. However, at the scale of the web with the diversity of implementation and platforms, the unexpected does occur.

   5. Otherwise, queue a task to resolve promise with dictionary .

5. Return promise .

Acknowledgments

With that, many thanks to Adrian Bateman, Aislinn Grigas, Alex Russell, Ali Alabbas, Andrew Sutherland, Ben Kelly, Ben Turner, Dale Harvey, David Grogan, Domenic Denicola, fantasai, Jake Archibald, Jeffrey Yasskin, Jinho Bang, Jonas Sicking, Joshua Bell, Kenji Baheux, Kinuko Yasuda, Luke Wagner, Michael Nordman, Mounir Lamouri, Shachar Zohar, 黃強 (Shawn Huang), 簡冠庭 (Timothy Guan-tin Chien), and Victor Costan for being awesome!

This standard is written by Anne van Kesteren ( Mozilla , annevk@annevk.nl ).

Copyright © 2020 WHATWG (Apple, Google, Mozilla, Microsoft). This work is licensed under a Creative Commons Attribution 4.0 International License .