uncaught QuotaExceededError for local storage or Compress support

[ron-liu]

When I set mode to use local storage and try to request and get a response more than 5M, it will cause the error: Uncaught QuotaExceededError. That is because it exceed the build-in localstorage quota.

I will suggest the following two ways to fix:

• have a quota settings first by default 5M, then when putting, if the coming response is great than the quota, it should ignore and output a warning in console.
• have a compress settings, when putting, if the coming response is great than the quota, it should use the passing compress interface to compress before put it into local storage.

Cheers,
Ron

[jmdobry]

How to do you propose measuring how much is in localStorage?

[ron-liu]

Looks like have to have settings for different browsers based on the detected browser type?

[jmdobry]

I'll just come out and say it: There is no way to measure how much data is already stored in localStorage. localStorage could have already have been filled up by something other than angular-cache, in which case there is absolutely nothing angular-cache can do to free up space. Never ever will angular-cache do anything to any data in localStorage that wasn't put there by angular-cache.

[brphelps]

We just ran into this on a project I'm working on -- the lack of maximum capacity basically means we can't trust angular-cache to cache sets of data that are potentially larger than 5MB (unfortunate).

Instead of worrying about how you can tell "everything" that's in localStorage, it's much more interesting to be able to tell if angular-cache is staying within a certain boundary: One example implementation would be for me ( a consumer) to set the maximum capacity to 4MB, and the only expectation would be that angular-cache would be responsible for maintaining its data within that 4MB quota. This allows frameworks which use local storage in very limited ways to not be put at risk by also using localStorage w/ angular-cache, which is actually fairly elegant.

I would also expect a default maximum to be something like 4.6MB (or some other logical "almost maximum" value) to prevent completely starving localStorage capacity for a domain.

[dmcass]

localStorage capacity varies by browser, so setting an arbitrary maximum is probably not a great idea. See this article on working with quota on mobile browsers from the beginning of last year, which is still fairly accurate and includes information about desktop browsers as well. If you need more space to store your data and you have no options other than localStorage, you're better off wrapping localStorage with something like lz-string in the storageImpl option of your angular-cache, which sounds like what the OP was asking for. Below is an example of what that configuration would look like.

storageImpl: {
  getItem: function (key) {
    return LZString.decompressFromUTF16(localStorage.getItem(key));
  },
  setItem: function (key, value) {
    localStorage.setItem(key, LZString.compressToUTF16(value));
  },
  removeItem: function (key) {
    localStorage.removeItem(key);
  }
}

Compression isn't free, and often comes at the expense of performance. Beware that if your codebase is liberal with storing and retrieving data, something like compression will slow everything down until you optimize when and where your application stores and retrieves data. Using a combination of memory and localStorage is how I've approached it thus far. This allows for fast access when needed (memory) and long term storage when it doesn't interfere with responsiveness of the application (update localStorage when the user is idle).

If instead you'd like angular-cache to respond in a different way to failed storage attempts with something like an LRU algorithm, I'd argue that you're better off figuring out a way to segment your data into smaller chunks so that the already existing capacity option can manage that overflow for you. While I see the value in something like a byteCapacity option, angular-cache would never touch anything that it doesn't own, so that wouldn't be a very reliable solution for a plugin like this. Additionally, if you're storing large amounts of data in a single key, you could end up overwriting your cache often enough that it might not be worth having a cache in the first place.

[jmdobry]

@dmcass Awesome thoughts, thanks!

[vincentsels]

@dmcass: I don't see how using the capacity option replaces having angular-cache handle failed storage attempts due to exceeded quota by removing the LRU entry/entries and retrying. The latter is a fail-proof implementation. Using the capacity option with small chunks, you still need to pick an arbitrary (small enough) number, which will never be ideal. If it's not too much work and doesn't add too much complexity, I'd really like to see this plugin properly handle errors due to exceeded quota by removing as many LRU entries as needed in order to be able to store the new entry, and only throw the DOMException when this fails even after all entries were removed (i.e. when attempting to store an entry larger than the browser's limit).

See this article for a way to detect the error across several browsers (not sure how other browsers like mobile browsers handle them).

@jmdobry: your opinion ?

[vincentsels]

Hm. Of course, having the plugin consume all remaining storage space wouldn't be very tidy either - the next storage attempt on the domain that's not coming from the plugin (if there are any, that is) would throw the exception instead, just postponing the problem a bit.
In that regard, it seems using small chunks of data and the capacity property does seem like the only sensible solution :/.

[brphelps]

@vincentsels You captured my original intention quite succinctly :). The point of the capacity limit is to allow a developer to ensure that angular-cache doesn't cause significant problems for other localStorage consumers. If the end answer is that "There's nothing you can do, because we won't add a limit to angular-cache you have to make sure all other localStorage consumers are able to 'just deal with it' ". Either that, or I have to implement my own capacity manager on top of angular-cache, which feels very wrong.

[jmdobry]

Handling the quota exceeded error by attempting to remove the least recently used item (if any), then retrying (once) seems like it shouldn't be much work. But, what if removing one item wasn't enough? Does it give up at that point? Or does it keep removing items until either it succeeds or the cache is empty?

[vincentsels]

@jmdobry: nevermind that: as was already pointed out, there's no use having angular-cache taking up all available space because that would suffocate anything else that tries to store anything. Sorry - should've read the other replies more carefully before posting!

As brphelps suggested, being able to set a fraction that angular-cache is allowed to consume would be very nice, but as noted by dmcass, browsers' implementations don't even guarantee 5MB and you can't 'query' the limit (without trying it out by storing dummy data). You could maybe try to guess it based on the userAgent or let the user pick a value, or assume the lowest value of all common browsers (2 MB for Android browser, according to the site referenced by dmcass). Since data is stored as 16 bit characters, you could then calculate the portion that angular-cache is consuming.

For now, I think I'll go with using the capacity + the lz-string compression library suggested by dmcass, and aim for a total size of ~1.5 MB.

[3choBoomer]

FWIW, when one does encounter the QuotaExceededError, the issue is compounded the next time a page load happens and you attempt to create a new CacheFactory item and this line is hit:
https://github.com/jmdobry/angular-cache/blob/master/dist/angular-cache.js#L939

That throws the QuotaExceededError and then defaults your cache back to the memory cache (essentially invalidating the entire local storage cache).

IMO, this should be handled the same way the LRU algorithm works.

[jmendiara]

@jmdobry et all, please take a look to httpu.caches to see an implementation of a $cacheFactory with size (not exact but very accurate) support. It also incorporates the same compress approach @dmcass described.

[scottopolis]

Hey @jmdobry thanks for this project, it works great. I'm also running into this issue, and I think this library needs a way to handle it. No matter how you handle this issue, anything would be better than my app completely freezing due to the quota exceeded error ;)

What about this approach, used by Modernizr? https://github.com/Modernizr/Modernizr/blob/master/feature-detects/storage/localstorage.js

I may try this in my own project.

[jmdobry]

Recommended solution

Make liberal use of the capacity option or provide a custom storageImpl in your cache config:

With lz-string compression:

storageImpl: {
  getItem: function (key) {
    return LZString.decompressFromUTF16(localStorage.getItem(key));
  },
  setItem: function (key, value) {
    try {
      localStorage.setItem(key, LZString.compressToUTF16(value));
    } catch (err) {
      // quota exceeded or some other error
      // handle this however you want
    }
  },
  removeItem: function (key) {
    localStorage.removeItem(key);
  }
}

Without compression:

storageImpl: {
  getItem: localStorage.getItem,
  setItem: function (key, value) {
    try {
      localStorage.setItem(key, LZString.compressToUTF16(value));
    } catch (err) {
      // quota exceeded or some other error
      // handle this however you want
    }
  },
  removeItem: localStorage.removeItem
}

or

try {
  cache.put('foo', 'bar')
} catch (err) {
  // quota exceeded or some other error
  // handle this however you want
}

Links:

• Angular-cache's capacity option
• Angular-cache's storageImpl option
• Working with quota on mobile browsers
• Catching localStorage quota exceeded errors

Thanks @dmcass!