Name: sqlite
URL: https://sqlite.org/
Version: 3.22.0
Included In Release: Yes
Security Critical: Yes
License: Public domain

1) Managing differences between SQLite core and Chromium's version.
2) Making changes to Chromium SQLite.
3) Import new release of SQLite.
4) Running SQLite's test suite within the Chromium checkout.

---

1) Managing differences between SQLite core and Chromium's version.

Chromium maintains some differences WRT SQLite, for reasons beyond this
document's remit.  Some differences are bugs we have found and fixed (and
hopefully upstreamed), some are fixes we've backported from a later version of
SQLite, and some our local changes unlikely to ever be upstreamed.  New versions
of SQLite are imported every year or two, at which point the changes need to be
reviewed for continued applicability, and sometimes adjusted to reflect upstream
code changes.

To this end, the repository contains a reference copy of the SQLite source code
as of the last import, plus a series of patches which can be applied to
re-create the current trunk code.  These patches are generated and processed by
git, with the intention of re-creating a commit series so that importers can use
their regular revision-control knowledge to manage import merges.

The directory structure is as follows. Files common to all third_party projects
(BUILD.GN, OWNERS, LICENSE) are omitted.

* sqlite-src-*/ - Upstream source code, without any modifications. The number
                  after src- is a release version or a snapshot number (which is
                  a commit time). See https://www.sqlite.org/cgi/src/timeline
* patches/ - Our patches to the currently used release, formatted by git using
             the UNIX mailbox format. The patches can be applied with git am,
             and created with git format-patch.
* src/ - The currently used source code, with our patches applied.
* amalgamation/ - The supported method of using SQLite is via an amalgamation
                  build, which merges all the code in one .c file and one .h
                  file. See https://www.sqlite.org/amalgamation.html
* amalgamation/config.h - Linux build configuration
* scripts/ - Scripts that generate the files in the amalgamation
* sqlite.h - The header used by the rest of Chromium to include SQLite. This
             forwards to amalgamation/sqlite3.h
* fuzz/ - Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build

---

2) Making changes to Chromium SQLite.

third_party/sqlite/src is the patched source from SQLite.  This is used to
generate the amalgamation, a concatenation of all of the files into a giant
sqlite3.c.  To prototype, edit in src/, then call
  ./scripts/generate_amalgamation.sh
to regenerate sqlite3.c.  The code in src/ is much easier to edit, and the
SQLite test framework can easily be run.  During development it may be
convenient to modify BUILD.gn based on src/main.mk to just pull in the src/
files rather than sqlite3.c.

Once your patch is complete, squash it down into a reasonable CL, then
re-generate the patches.  This is a truncated version of the import flow.  The
following is written like a shell script to allow copy/paste to a shell, ignore
comments and change the obvious lines.  These instructions should work on Linux
or OSX.  They may assume a modern version of git (I'm using 2.2.1).

# Everything based in sqlite subdir.
cd third_party/sqlite

BASE=3080704

#### Create a reference branch.
git checkout -b sqlite_${BASE} master
git rm -rf src
rm -rf src  # In case there are things git doesn't know in there.
cp -a sqlite-src-${BASE} src
# -f includes ignored files, of which there are a couple.
git add -f src/
git commit -m "Reset to sqlite-src-${BASE}"
# This branch is unlikely to build.

#### Create a reference branch with patches applied.
git checkout -b sqlite_${BASE}_patched master
git rebase sqlite_${BASE}
git am --keep-non-patch patches/*.patch
git diff master
# This branch should be identical to master, unless someone forgot to export
# their changes into a patch.  If so, do that as a separate CL and start over.

#### Cherry-pick your change.
git cherry-pick <your change>
# This branch should be identical to your development branch, except
# amalgamation.

# Rebuild the patch set.
git rm patches/*
git format-patch --output-directory=patches sqlite_${BASE}..HEAD
git add patches/*.patch
git commit -m "Rebuild patches for sqlite_${BASE}"

# Re-generate the amalgamation.
./scripts/generate_amalgamation.sh
git commit -m './scripts/generate_amalgamation.sh' amalgamation/
# At this point everything should build and work.

# Do a squash upload.  This should add your single patch to patches/, and apply
# the changes your patch represents to src/ and amalgamation/.  Other patches
# will have hash changes.  A sensible check-in comment would be something like
# the patch's checkin comment, plus "regenerate amalgamation and generate patch
# file."
# TODO(pwnall): Should hash changes be checked in, or backed out?

# Find unsuspecting victim and send review.

---

3) Import a new SQLite release.

Importing a new SQLite involves merging our local changes with SQLite's changes.
Like any other merge, this may involve modifying some commits and dropping
others. The basic idea below is to generate git branches to work with:

* sqlite-new-upstream - new release code archived in a separate directory
* sqlite-old-base     - current release without patches
* sqlite-old          - current release with patches mapped to git commits
* sqlite-new-base     - new release without patches
* sqlite-new          - new release with patches mapped to git commits
* sqlite-new-cl       - new release in one git commit, for git cl upload

We will upload sqlite-new-upstream as a massive (800k LOC+) CL that cannot
possibly be reviewed, but is generated in an automated fashion. We will then
squash sqlite-new to sqlite-new-upstream and obtain one CL that only contains
diffs. The second CL is still large, but it's a fraction of the first
(automated) CL.

# The steps below are easier if done in the SQLite directory.
cd third_party/sqlite

export OLD=3220000
export NEW=3230000
export GNU_SED=sed  # OSX: "brew install gnu-sed", then use "gsed" here.

#### Download and unpack the new SQLite release.
git new-branch sqlite-new-upstream
# URL from "Alternative Source Code Formats" at https://sqlite.org/download.html
curl https://sqlite.org/2018/sqlite-src-3230000.zip > upstream.zip
mkdir sqlite-src-$NEW
unzip ./upstream.zip -d sqlite-src-$NEW
rm ./upstream.zip
mv sqlite-src-$NEW/sqlite-*/* sqlite-src-$NEW/
rmdir sqlite-src-$NEW/sqlite-*/
xdg-open sqlite-src-$NEW  # Make sure everything looks right.

#### Add the new release code in a separate CL, for code review sanity.
git add sqlite-src-$NEW  # Committing the code as downloaded, on purpose.
git clean -i -d -x sqlite-src-$NEW  # Make sure no file is git-ignored.
git commit -m "sqlite: Add code for release 3.23.0"
git cl upload  # Have the new code in a separate (impossible to review) CL.

#### Create a branch for the old SQLite release's upstream version.
git new-branch --upstream-current sqlite-old-base
git rm -rf src
cp -r sqlite-src-${OLD}/ src
# Clean up trailing whitespace and CRLF so any patches look clean.
find src/ -type f -not -iname "*.db" -not -iname "*.eps" -not -iname "*.ico" \
    -not -iname "*.jpg" -not -iname "*.pfx" -not -iname "*.png" \
    -not -iname "*.tiff" -not -iname "*.vsix" \
    -exec $GNU_SED --in-place 's/[[:space:]]\+$//' {} \+
git add src/
git clean -i -d -x sqlite-src-$NEW  # Make sure no file is git-ignored.
git commit -m "Squash: Reset SQLite src/ to sqlite-src-${OLD}."
# This branch will not build. It will be used for rebasing, then deleted.

#### Create a branch for our old SQLite code, with patches mapped to commits.
git new-branch --upstream-current sqlite-old
git am --keep-non-patch --ignore-space-change patches/*.patch
git diff --ignore-space-change origin/master src/
# This branch should be identical to master.

#### Create a branch for the new SQLite release's upstream version.
git checkout sqlite-old-base
git new-branch --upstream-current sqlite-new-base
git rm -rf src
cp -r sqlite-src-${NEW}/ src
# Clean up trailing whitespace and CRLF so any patches look clean.
find src/ -type f -not -iname "*.db" -not -iname "*.eps" -not -iname "*.ico" \
    -not -iname "*.jpg" -not -iname "*.pfx" -not -iname "*.png" \
    -not -iname "*.tiff" -not -iname "*.vsix" \
    -exec $GNU_SED --in-place 's/[[:space:]]\+$//' {} \+
git add src/
git clean -i -d -x sqlite-src-$NEW  # Make sure no file is git-ignored.
git commit -m "Squash: Reset SQLite src/ to sqlite-src-${NEW}."
# This branch will not build. It will be used for rebasing, then deleted.

#### Create a branch for updating our patches.
git checkout sqlite-old
git new-branch --upstream-current sqlite-new
# Rebase our patches, which are mapped to separate commits, onto the new
# release. There will be merge conflicts that must be fixed. This is the
# interesting part of the work.
git rebase sqlite-new-base

#### Finally, create the branch that we'll upload.
git new-branch --upstream-current sqlite-new-cl
./scripts/generate_amalgamation.sh

#### Validate the upgrade.
# The goal is to have a set of reasonably-independent CLs which can be
# understood separately, so that future importers can sensibly determine how to
# handle conflicts.  So use git-rebase and slipstream fixups back into their
# original CL until everything builds and works.
cd ../..
ninja -C out/Default
# Check that extract_sqlite_api.py added chrome_ to all exported symbols.
# Only "_fini" and "_init" should be unprefixed.
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'
out/Default/sql_unittests
third_party/WebKit/Tools/Scripts/run-webkit-tests -t Default storage/websql/*
cd third_party/sqlite

#### Create the review.
# Rebuild the patch set.
git rm patches/*
git format-patch --output-directory=patches --ignore-space-change \
    sqlite-new-base..sqlite-new
git add amalgamation/
git add patches/*.patch
git commit -m "Squash: regenerate amalgamation and patches."
git branch --set-upstream-to=sqlite-new-upstream
git cl upload --squash
# Edit the commit message appropriately to reflect anything from
# <https://www.sqlite.org/changes.html> which might be deemed important. Don't
# enumerate all of the patch messages, those are assumed, but do reference any
# material changes made.
# TODO(pwnall) Describe an appropriate comment style.  Seems like it should at
# least include the SQLite version number.  Meanwhile, look in the logs for past
# commits to model things on.

#### Drop the old version of SQLite.
git new-branch sqlite-rm-old
git rm -r sqlite_${BASE}
git commit -m "sqlite: Remove source code for old release ${OLD}."

Note that things can be broken down differently, if you prefer.  For instance,
adding the new version of the SQLite distro and removing the old one can be
distinct CLs.

--------------------------------------------

4) Running SQLite's test suite within the Chromium checkout.

TODO(pwnall): This hasn't been tried out for at least a year.

Prerequisites: The test suite requires tcl-dev and libicu-dev.  Install those on
Ubuntu like:
  sudo apt-get install tcl8.6-dev libicu-dev
On OSX, I use MacPorts:
  sudo port install tcl

cd third_party/sqlite/src
mkdir build
cd build
# This sometimes gives integer-size warnings on the differences between
# Tcl_WideInt and sqlite3_int64 and int64.  Usually this is easily fixed by
# changing a variable to Tcl_WideInt.
make -j -f ../Makefile.linux-gcc testfixture sqlite3 sqlite3_analyzer sqldiff
make -f ../Makefile.linux-gcc test > /tmp/test.log
egrep 'errors out of' /tmp/test.log
# Show broken tests:
egrep 'Failures on these tests:' /tmp/test.log
# Broken tests will also show lines ending in "..." instead of "... Ok".

In version 3.10.2 on OSX 10.11.2, I see:
   6 errors out of 139819 tests
The failed tests are:
   pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11
This is due to the change in os_unix.c fileHasMoved() to support WebDatabase.
Commenting out the early return allows them to succeed.

In version 3.10.2 on Ubuntu 14.04.3 I see:
   9 errors out of 140309 tests
The failed tests are:
   oserror-1.1.1 oserror-1.1.2 oserror-1.1.3
   pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11
The oserror tests fail because there are too many fds available, and can be
fixed by running "ulimit -n 1024" before the test.  The pager4 tests are failing
for the same reason as for OSX.

--

NOTE(pwnall): On Ubuntu it is possible to run the tests in a tmpfs something
like:

TMPFS=/dev/shm/sqlite_build
BUILD=$PWD
mkdir $TMPFS
(cd $TMPFS ; $BUILD/testfixture $BUILD/../test/veryquick.test >/tmp/test.log)

This is faster, but it is plausible that different things are being tested than
real-world use.