[core] introduce Blob for compressed and uncompressed dataupstream/blob

- Blob is a wrapper type for a shared_ptr<const string> that has accessor functions for getting compressed and uncompressed data
- Moved util::writeFile, util::readFile, util::compress, util::uncompress, decodeImage, and encodePNG to the Blob interface
- Added Blob support to Request and file sources
- Added Blob support to VectorTile objects
- Added support for gzip decoding to util::uncompress
- We're no longer compressing WebP, PNG, and JPEG data when storing in the OfflineDatabase
- Android's HTTPRequest returns compressed Blobs by default

One caveat is that our previous decompress function didn't support gzip, so once users upgrade to this version, their offline cache may contain both zlib-compressed data and gzip-compressed data, but older versions won't be able to decompress gzip data. On the other hand, we don't support downgrading SDKs anyway, so this shouldn't be a problem. To be on the safe side, we could bump the user_version of the SQLite DB.

1 files changed, 36 insertions, 1 deletions

diff --git a/src/mbgl/util/compression.cpp b/src/mbgl/util/compression.cpp
index 30e813cbb8..4d4e094f06 100644
--- a/src/mbgl/util/compression.cpp
+++ b/src/mbgl/util/compression.cpp
@@ -71,7 +71,8 @@ std::string decompress(const std::string &raw) {
     memset(&inflate_stream, 0, sizeof(inflate_stream));
 
     // TODO: reuse z_streams
-    if (inflateInit(&inflate_stream) != Z_OK) {
+    // MAX_WBITS + allows decoding gzip in addition to zlib
+    if (inflateInit2(&inflate_stream, MAX_WBITS + 32) != Z_OK) {
         throw std::runtime_error("failed to initialize inflate");
     }
 
@@ -100,5 +101,39 @@ std::string decompress(const std::string &raw) {
 
     return result;
 }
+
+bool isCompressible(const std::string& raw) {
+    // WebP
+    if (raw.size() >= 12 && static_cast<uint8_t>(raw[0]) == 'R' &&
+        static_cast<uint8_t>(raw[1]) == 'I' && static_cast<uint8_t>(raw[2]) == 'F' &&
+        static_cast<uint8_t>(raw[3]) == 'F' && static_cast<uint8_t>(raw[8]) == 'W' &&
+        static_cast<uint8_t>(raw[9]) == 'E' && static_cast<uint8_t>(raw[10]) == 'B' &&
+        static_cast<uint8_t>(raw[11]) == 'P') {
+        // Note: the WebP container format allows uncompressed data as well, but we just assume that
+        // all WebP files are already compressed.
+        return false;
+    }
+
+    // PNG
+    if (raw.size() >= 8 && static_cast<uint8_t>(raw[0]) == 0x89 &&
+        static_cast<uint8_t>(raw[1]) == 'P' && static_cast<uint8_t>(raw[2]) == 'N' &&
+        static_cast<uint8_t>(raw[3]) == 'G' && static_cast<uint8_t>(raw[4]) == '\r' &&
+        static_cast<uint8_t>(raw[5]) == '\n' && static_cast<uint8_t>(raw[6]) == 0x1a &&
+        static_cast<uint8_t>(raw[7]) == '\n') {
+        // Note: this assumes the PNG file itself is compressed. However, it is possible to create
+        // PNG files with uncompressed data in it (zlib compression 0), but they are exceedingly
+        // rare, so we don't care about them.
+        return false;
+    }
+
+    // JPEG
+    if (raw.size() >= 3 && static_cast<uint8_t>(raw[0]) == 0xff &&
+        static_cast<uint8_t>(raw[1]) == 0xd8 && static_cast<uint8_t>(raw[2]) == 0xff) {
+        return false;
+    }
+
+    return true;
+}
+
 } // namespace util
 } // namespace mbgl