diff options
| author | Alexander Shorin <kxepal@apache.org> | 2013-09-02 00:53:46 +0400 |
|---|---|---|
| committer | Alexander Shorin <kxepal@apache.org> | 2013-09-27 22:01:47 +0400 |
| commit | 7c4a8c4e3975a1410f21f338caf12926f75f1639 (patch) | |
| tree | 61eee3502252c99ad4c30e029bf49cf401949a46 | |
| parent | 627806e25628861481f760acddfd6eb2ded0b13e (diff) | |
| download | couchdb-7c4a8c4e3975a1410f21f338caf12926f75f1639.tar.gz | |
Better introduction into CouchDB.
Import articles from Guide to CouchDB:
- Why CouchDB?
- Eventual Consistency
- Getting Started
+ Technical Overview from Wiki
30 files changed, 1937 insertions, 158 deletions
diff --git a/share/doc/build/Makefile.am b/share/doc/build/Makefile.am index 188714c5c..dcc11507b 100644 --- a/share/doc/build/Makefile.am +++ b/share/doc/build/Makefile.am @@ -109,6 +109,13 @@ html_files = \ html/_sources/install/mac.txt \ html/_sources/install/unix.txt \ html/_sources/install/windows.txt \ + html/_sources/intro/consistency.txt \ + html/_sources/intro/curl.txt \ + html/_sources/intro/futon.txt \ + html/_sources/intro/index.txt \ + html/_sources/intro/overview.txt \ + html/_sources/intro/tour.txt \ + html/_sources/intro/why.txt \ html/_sources/query-server/index.txt \ html/_sources/query-server/erlang.txt \ html/_sources/query-server/javascript.txt \ @@ -131,7 +138,6 @@ html_files = \ html/_sources/contributing.txt \ html/_sources/externals.txt \ html/_sources/index.txt \ - html/_sources/intro.txt \ html/_sources/json-structure.txt \ html/_static/ajax-loader.gif \ html/_static/basic.css \ @@ -216,6 +222,13 @@ html_files = \ html/install/mac.html \ html/install/unix.html \ html/install/windows.html \ + html/intro/consistency.html \ + html/intro/curl.html \ + html/intro/futon.html \ + html/intro/index.html \ + html/intro/overview.html \ + html/intro/tour.html \ + html/intro/why.html \ html/query-server/index.html \ html/query-server/erlang.html \ html/query-server/javascript.html \ @@ -237,7 +250,6 @@ html_files = \ html/whatsnew/index.html \ html/externals.html \ html/index.html \ - html/intro.html \ html/json-structure.html \ html/objects.inv \ html/genindex.html \ @@ -258,7 +270,31 @@ image_files = \ ../images/futon-editeddoc.png \ ../images/futon-overview.png \ ../images/futon-replform.png \ - ../images/logo.png + ../images/intro-consistency-01.png \ + ../images/intro-consistency-02.png \ + ../images/intro-consistency-03.png \ + ../images/intro-consistency-04.png \ + ../images/intro-consistency-05.png \ + ../images/intro-consistency-06.png \ + ../images/intro-consistency-07.png \ + ../images/intro-tour-01.png \ + ../images/intro-tour-02.png \ + ../images/intro-tour-03.png \ + ../images/intro-tour-04.png \ + ../images/intro-tour-05.png \ + ../images/intro-tour-06.png \ + ../images/intro-tour-07.png \ + ../images/intro-tour-08.png \ + ../images/intro-tour-09.png \ + ../images/intro-tour-10.png \ + ../images/intro-why-01.png \ + ../images/intro-why-02.png \ + ../images/intro-why-03.png \ + ../images/logo.png \ + ../images/views-intro-01.png \ + ../images/views-intro-02.png \ + ../images/views-intro-03.png \ + ../images/views-intro-04.png src_files = \ ../src/api/basics.rst \ @@ -321,6 +357,13 @@ src_files = \ ../src/install/mac.rst \ ../src/install/unix.rst \ ../src/install/windows.rst \ + ../src/intro/consistency.rst \ + ../src/intro/curl.rst \ + ../src/intro/futon.rst \ + ../src/intro/index.rst \ + ../src/intro/overview.rst \ + ../src/intro/tour.rst \ + ../src/intro/why.rst \ ../src/query-server/index.rst \ ../src/query-server/erlang.rst \ ../src/query-server/javascript.rst \ @@ -343,7 +386,6 @@ src_files = \ ../src/contributing.rst \ ../src/externals.rst \ ../src/index.rst \ - ../src/intro.rst \ ../src/json-structure.rst \ ../src/conf.py diff --git a/share/doc/images/intro-consistency-01.png b/share/doc/images/intro-consistency-01.png Binary files differnew file mode 100644 index 000000000..9ae691214 --- /dev/null +++ b/share/doc/images/intro-consistency-01.png diff --git a/share/doc/images/intro-consistency-02.png b/share/doc/images/intro-consistency-02.png Binary files differnew file mode 100644 index 000000000..06c23ea71 --- /dev/null +++ b/share/doc/images/intro-consistency-02.png diff --git a/share/doc/images/intro-consistency-03.png b/share/doc/images/intro-consistency-03.png Binary files differnew file mode 100644 index 000000000..2164c6c9a --- /dev/null +++ b/share/doc/images/intro-consistency-03.png diff --git a/share/doc/images/intro-consistency-04.png b/share/doc/images/intro-consistency-04.png Binary files differnew file mode 100644 index 000000000..068fa775c --- /dev/null +++ b/share/doc/images/intro-consistency-04.png diff --git a/share/doc/images/intro-consistency-05.png b/share/doc/images/intro-consistency-05.png Binary files differnew file mode 100644 index 000000000..a94f9c35d --- /dev/null +++ b/share/doc/images/intro-consistency-05.png diff --git a/share/doc/images/intro-consistency-06.png b/share/doc/images/intro-consistency-06.png Binary files differnew file mode 100644 index 000000000..af316d4df --- /dev/null +++ b/share/doc/images/intro-consistency-06.png diff --git a/share/doc/images/intro-consistency-07.png b/share/doc/images/intro-consistency-07.png Binary files differnew file mode 100644 index 000000000..7fb5027e5 --- /dev/null +++ b/share/doc/images/intro-consistency-07.png diff --git a/share/doc/images/intro-tour-01.png b/share/doc/images/intro-tour-01.png Binary files differnew file mode 100644 index 000000000..41b6420e8 --- /dev/null +++ b/share/doc/images/intro-tour-01.png diff --git a/share/doc/images/intro-tour-02.png b/share/doc/images/intro-tour-02.png Binary files differnew file mode 100644 index 000000000..f2636645c --- /dev/null +++ b/share/doc/images/intro-tour-02.png diff --git a/share/doc/images/intro-tour-03.png b/share/doc/images/intro-tour-03.png Binary files differnew file mode 100644 index 000000000..47f013b4d --- /dev/null +++ b/share/doc/images/intro-tour-03.png diff --git a/share/doc/images/intro-tour-04.png b/share/doc/images/intro-tour-04.png Binary files differnew file mode 100644 index 000000000..33cecccf9 --- /dev/null +++ b/share/doc/images/intro-tour-04.png diff --git a/share/doc/images/intro-tour-05.png b/share/doc/images/intro-tour-05.png Binary files differnew file mode 100644 index 000000000..08042d781 --- /dev/null +++ b/share/doc/images/intro-tour-05.png diff --git a/share/doc/images/intro-tour-06.png b/share/doc/images/intro-tour-06.png Binary files differnew file mode 100644 index 000000000..bb9c7c8de --- /dev/null +++ b/share/doc/images/intro-tour-06.png diff --git a/share/doc/images/intro-tour-07.png b/share/doc/images/intro-tour-07.png Binary files differnew file mode 100644 index 000000000..dc009a89f --- /dev/null +++ b/share/doc/images/intro-tour-07.png diff --git a/share/doc/images/intro-tour-08.png b/share/doc/images/intro-tour-08.png Binary files differnew file mode 100644 index 000000000..6976f57e5 --- /dev/null +++ b/share/doc/images/intro-tour-08.png diff --git a/share/doc/images/intro-tour-09.png b/share/doc/images/intro-tour-09.png Binary files differnew file mode 100644 index 000000000..f41b3062d --- /dev/null +++ b/share/doc/images/intro-tour-09.png diff --git a/share/doc/images/intro-tour-10.png b/share/doc/images/intro-tour-10.png Binary files differnew file mode 100644 index 000000000..e95bc668e --- /dev/null +++ b/share/doc/images/intro-tour-10.png diff --git a/share/doc/images/intro-why-01.png b/share/doc/images/intro-why-01.png Binary files differnew file mode 100644 index 000000000..c9274506f --- /dev/null +++ b/share/doc/images/intro-why-01.png diff --git a/share/doc/images/intro-why-02.png b/share/doc/images/intro-why-02.png Binary files differnew file mode 100644 index 000000000..a5bb4ce49 --- /dev/null +++ b/share/doc/images/intro-why-02.png diff --git a/share/doc/images/intro-why-03.png b/share/doc/images/intro-why-03.png Binary files differnew file mode 100644 index 000000000..1f5e536dc --- /dev/null +++ b/share/doc/images/intro-why-03.png diff --git a/share/doc/src/couchapp/index.rst b/share/doc/src/couchapp/index.rst index d01b91e13..0985d4924 100644 --- a/share/doc/src/couchapp/index.rst +++ b/share/doc/src/couchapp/index.rst @@ -10,6 +10,8 @@ .. License for the specific language governing permissions and limitations under .. the License. +.. _couchapp: + ======== CouchApp ======== diff --git a/share/doc/src/index.rst b/share/doc/src/index.rst index 047088e4b..c41cc9f83 100644 --- a/share/doc/src/index.rst +++ b/share/doc/src/index.rst @@ -10,24 +10,15 @@ .. License for the specific language governing permissions and limitations under .. the License. -Introduction -============ -|Apache CouchDB(TM)|_ is a document database built for the web. +|Apache CouchDB(TM)|_ |release| Documentation +============================================= -If you would like to help document the project, please send a note to the -`developer mailing list <http://couchdb.apache.org/#mailing-list>`_. - -This is a work in progress. - -Contents -======== - .. toctree:: :maxdepth: 3 :numbered: - intro + intro/index install/index config/index replication/index diff --git a/share/doc/src/intro/consistency.rst b/share/doc/src/intro/consistency.rst new file mode 100644 index 000000000..d10040b9b --- /dev/null +++ b/share/doc/src/intro/consistency.rst @@ -0,0 +1,465 @@ +.. Licensed under the Apache License, Version 2.0 (the "License"); you may not +.. use this file except in compliance with the License. You may obtain a copy of +.. the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +.. License for the specific language governing permissions and limitations under +.. the License. + + +.. _intro/consistency: + +==================== +Eventual Consistency +==================== + +In the previous chapter, we saw that CouchDB's flexibility allows us to +evolve our data as our applications grow and change. In this chapter, +we'll explore how working "with the grain" of CouchDB promotes simplicity in +our applications and helps us naturally build scalable, distributed systems. + + +Working with the Grain +====================== + +A *distributed system* is a system that operates robustly over a wide network. +A particular feature of network computing is that network links can +potentially disappear, and there are plenty of strategies for managing this +type of network segmentation. CouchDB differs from others by accepting +eventual consistency, as opposed to putting absolute consistency ahead of raw +availability, like `RDBMS`_ or `Paxos`_. What these systems have in common is +an awareness that data acts differently when many people are accessing it +simultaneously. Their approaches differ when it comes to which aspects of +*consistency*, *availability*, or *partition* tolerance they prioritize. + +Engineering distributed systems is tricky. Many of the caveats and "gotchas" +you will face over time aren't immediately obvious. We don't have all the +solutions, and CouchDB isn't a panacea, but when you work with CouchDB's +grain rather than against it, the path of least resistance leads you to +naturally scalable applications. + +Of course, building a distributed system is only the beginning. A website +with a database that is available only half the time is next to worthless. +Unfortunately, the traditional relational database approach to consistency +makes it very easy for application programmers to rely on global state, +global clocks, and other high availability no-nos, without even realizing +that they're doing so. Before examining how CouchDB promotes scalability, +we'll look at the constraints faced by a distributed system. After we've seen +the problems that arise when parts of your application can't rely on being +in constant contact with each other, we'll see that CouchDB provides an +intuitive and useful way for modeling applications around high availability. + +.. _RDBMS: http://en.wikipedia.org/wiki/Relational_database_management_system +.. _Paxos: http://en.wikipedia.org/wiki/Paxos_%28computer_science%29 + + +.. _cap: + +The CAP Theorem +=============== + +The CAP theorem describes a few different strategies for distributing +application logic across networks. CouchDB's solution uses replication to +propagate application changes across participating nodes. This is a +fundamentally different approach from consensus algorithms and relational +databases, which operate at different intersections of consistency, +availability, and partition tolerance. + +The CAP theorem, shown in :ref:`intro/consistency-01`, +identifies three distinct concerns: + +- **Consistency**: + All database clients see the same data, even with concurrent updates. +- **Availability**: + All database clients are able to access some version of the data. +- **Partition tolerance**: + The database can be split over multiple servers. + +Pick two. + +.. _intro/consistency-01: + +.. figure:: ../../images/intro-consistency-01.png + :align: center + :alt: The CAP theorem + + Figure 1. The CAP theorem + +When a system grows large enough that a single database node is unable to +handle the load placed on it, a sensible solution is to add more servers. +When we add nodes, we have to start thinking about how to partition data +between them. Do we have a few databases that share exactly the same data? +Do we put different sets of data on different database servers? +Do we let only certain database servers write data and let others handle +the reads? + +Regardless of which approach we take, the one problem we'll keep bumping into +is that of keeping all these database servers in sync. If you write some +information to one node, how are you going to make sure that a read request +to another database server reflects this newest information? These events +might be milliseconds apart. Even with a modest collection of database +servers, this problem can become extremely complex. + +When it's absolutely critical that all clients see a consistent view of the +database, the users of one node will have to wait for any other nodes to come +into agreement before being able to read or write to the database. +In this instance, we see that availability takes a backseat to consistency. +However, there are situations where availability trumps consistency: + + Each node in a system should be able to make decisions purely based on + local state. If you need to do something under high load with failures + occurring and you need to reach agreement, you're lost. If you're + concerned about scalability, any algorithm that forces you to run + agreement will eventually become your bottleneck. Take that as a given. + + -- Werner Vogels, Amazon CTO and Vice President + +If availability is a priority, we can let clients write data to one node of +the database without waiting for other nodes to come into agreement. +If the database knows how to take care of reconciling these operations between +nodes, we achieve a sort of "eventual consistency" in exchange for high +availability. This is a surprisingly applicable trade-off for many applications. + +Unlike traditional relational databases, where each action performed is +necessarily subject to database-wide consistency checks, +CouchDB makes it really simple to build applications that sacrifice immediate +consistency for the huge performance improvements that come with simple +distribution. + + +Local Consistency +================= + +Before we attempt to understand how CouchDB operates in a cluster, +it's important that we understand the inner workings of a single CouchDB node. +The CouchDB API is designed to provide a convenient but thin wrapper around +the database core. By taking a closer look at the structure of the database +core, we'll have a better understanding of the API that surrounds it. + + +The Key to Your Data +-------------------- + +At the heart of CouchDB is a powerful *B-tree* storage engine. +A B-tree is a sorted data structure that allows for searches, insertions, +and deletions in logarithmic time. As :ref:`intro/consistency-02` +illustrates, CouchDB uses this B-tree storage engine for all internal data, +documents, and views. If we understand one, we will understand them all. + + +.. _intro/consistency-02: + +.. figure:: ../../images/intro-consistency-02.png + :align: center + :alt: Anatomy of a view request + + Figure 2. Anatomy of a view request + + +CouchDB uses MapReduce to compute the results of a view. MapReduce makes use +of two functions, "map" and "reduce", which are applied to each document in +isolation. Being able to isolate these operations means that view computation +lends itself to parallel and incremental computation. More important, +because these functions produce key/value pairs, CouchDB is able to insert +them into the B-tree storage engine, sorted by key. Lookups by key, +or key range, are extremely efficient operations with a B-tree, +described in `big O` notation as ``O(log N)`` and ``O(log N + K)``, +respectively. + +In CouchDB, we access documents and view results by key or key range. +This is a direct mapping to the underlying operations performed on CouchDB's +B-tree storage engine. Along with document inserts and updates, +this direct mapping is the reason we describe CouchDB's API as being a thin +wrapper around the database core. + +Being able to access results by key alone is a very important restriction +because it allows us to make huge performance gains. As well as the massive +speed improvements, we can partition our data over multiple nodes, +without affecting our ability to query each node in isolation. +`BigTable`_, `Hadoop`_, `SimpleDB`_, and `memcached`_ restrict object lookups +by key for exactly these reasons. + +.. _BigTable: http://en.wikipedia.org/wiki/BigTable +.. _Hadoop: http://hadoop.apache.org +.. _SimpleDB: http://aws.amazon.com/simpledb/ +.. _memcached: http://memcached.org + + +No Locking +---------- + +A table in a relational database is a single data structure. If you want to +modify a table -- say, update a row -- the database system must ensure +that nobody else is trying to update that row and that nobody can read from +that row while it is being updated. The common way to handle this uses what's +known as a lock. If multiple clients want to access a table, the first client +gets the lock, making everybody else wait. When the first client's request is +processed, the next client is given access while everybody else waits, +and so on. This serial execution of requests, even when they arrived in +parallel, wastes a significant amount of your server's processing power. +Under high load, a relational database can spend more time figuring out who +is allowed to do what, and in which order, than it does doing any actual work. + +.. note:: + Modern relational databases avoid locks by implementing MVCC under + the hood, but hide it from the end user, requiring them to coordinate + concurrent changes of single rows or fields. + +Instead of locks, CouchDB uses `Multi-Version Concurrency Control` (MVCC) to +manage concurrent access to the database. :ref:`intro/consistency-03` +illustrates the differences between MVCC and traditional locking mechanisms. +MVCC means that CouchDB can run at full speed, all the time, +even under high load. Requests are run in parallel, making excellent use of +every last drop of processing power your server has to offer. + + +.. _intro/consistency-03: + +.. figure:: ../../images/intro-consistency-03.png + :align: center + :alt: MVCC means no locking + + Figure 3. MVCC means no locking + + +Documents in CouchDB are versioned, much like they would be in a regular +version control system such as `Subversion`_. If you want to change +a value in a document, you create an entire new version of that document +and save it over the old one. After doing this, you end up with two versions +of the same document, one old and one new. + +How does this offer an improvement over locks? Consider a set of requests +wanting to access a document. The first request reads the document. +While this is being processed, a second request changes the document. +Since the second request includes a completely new version of the document, +CouchDB can simply append it to the database without having to wait for the +read request to finish. + +When a third request wants to read the same document, CouchDB will point it +to the new version that has just been written. During this whole process, +the first request could still be reading the original version. + +A read request will always see the most recent snapshot of your database at +the time of the beginning of the request. + +.. _Subversion: http://subversion.apache.org/ + + +Validation +========== + +As application developers, we have to think about what sort of input we +should accept and what we should reject. The expressive power to do this type +of validation over complex data within a traditional relational database +leaves a lot to be desired. Fortunately, CouchDB provides a powerful way to +perform per-document validation from within the database. + +CouchDB can validate documents using JavaScript functions similar to those +used for MapReduce. Each time you try to modify a document, +CouchDB will pass the validation function a copy of the existing document, +a copy of the new document, and a collection of additional information, +such as user authentication details. The validation function now has the +opportunity to approve or deny the update. + +By working with the grain and letting CouchDB do this for us, +we save ourselves a tremendous amount of CPU cycles that would otherwise have +been spent serializing object graphs from SQL, converting them into domain +objects, and using those objects to do application-level validation. + + +Distributed Consistency +======================= + +Maintaining consistency within a single database node is relatively easy for +most databases. The real problems start to surface when you try to maintain +consistency between multiple database servers. If a client makes a write +operation on server `A`, how do we make sure that this is consistent with +server `B`, or `C`, or `D`? For relational databases, this is a very complex +problem with entire books devoted to its solution. You could use +multi-master, master/slave, partitioning, sharding, write-through caches, +and all sorts of other complex techniques. + + +Incremental Replication +======================= + +CouchDB's operations take place within the context of a single document. +As CouchDB achieves eventual consistency between multiple databases by using +incremental replication you no longer have to worry about your database +servers being able to stay in constant communication. Incremental replication +is a process where document changes are periodically copied between servers. +We are able to build what's known as a *shared nothing* cluster of databases +where each node is independent and self-sufficient, leaving no single point +of contention across the system. + +Need to scale out your CouchDB database cluster? Just throw in another server. + +As illustrated in :ref:`intro/consistency-04`, with CouchDB's incremental +replication, you can synchronize your data between any two databases however +you like and whenever you like. After replication, each database is able +to work independently. + +You could use this feature to synchronize database servers within a cluster +or between data centers using a job scheduler such as cron, +or you could use it to synchronize data with your laptop for offline work as +you travel. Each database can be used in the usual fashion, +and changes between databases can be synchronized later in both directions. + + +.. _intro/consistency-04: + +.. figure:: ../../images/intro-consistency-04.png + :align: center + :alt: Incremental replication between CouchDB nodes + + Figure 4. Incremental replication between CouchDB nodes + + +What happens when you change the same document in two different databases and +want to synchronize these with each other? CouchDB's replication system +comes with automatic conflict detection and resolution. When CouchDB detects +that a document has been changed in both databases, it flags this document +as being in conflict, much like they would be in a regular version control +system. + +This isn't as troublesome as it might first sound. When two versions of a +document conflict during replication, the winning version is saved as the +most recent version in the document's history. Instead of throwing the losing +version away, as you might expect, CouchDB saves this as a previous version +in the document's history, so that you can access it if you need to. This +happens automatically and consistently, so both databases will make exactly +the same choice. + +It is up to you to handle conflicts in a way that makes sense for your +application. You can leave the chosen document versions in place, +revert to the older version, or try to merge the two versions and save the +result. + + +Case Study +========== + +Greg Borenstein, a friend and coworker, built a small library for converting +Songbird playlists to JSON objects and decided to store these in CouchDB as +part of a backup application. The completed software uses CouchDB's MVCC and +document revisions to ensure that Songbird playlists are backed up robustly +between nodes. + +.. note:: + `Songbird`_ is a free software media player with an integrated web browser, + based on the Mozilla XULRunner platform. Songbird is available for Microsoft + Windows, Apple Mac OS X, Solaris, and Linux. + + .. _Songbird: http://en.wikipedia.org/wiki/Songbird_%28software%29 + +Let's examine the workflow of the Songbird backup application, +first as a user backing up from a single computer, and then using Songbird to +synchronize playlists between multiple computers. We'll see how document +revisions turn what could have been a hairy problem into something that *just +works*. + +The first time we use this backup application, we feed our playlists to the +application and initiate a backup. Each playlist is converted to a JSON +object and handed to a CouchDB database. As illustrated in +:ref:`intro/consistency-05`, CouchDB hands back the document ID and +revision of each playlist as it's saved to the database. + + +.. _intro/consistency-05: + +.. figure:: ../../images/intro-consistency-05.png + :align: center + :alt: Backing up to a single database + + Figure 5. Backing up to a single database + + +After a few days, we find that our playlists have been updated and we want to +back up our changes. After we have fed our playlists to the backup +application, it fetches the latest versions from CouchDB, +along with the corresponding document revisions. When the application hands +back the new playlist document, CouchDB requires that the document revision +is included in the request. + +CouchDB then makes sure that the document revision handed to it in the +request matches the current revision held in the database. Because CouchDB +updates the revision with every modification, if these two are out of sync it +suggests that someone else has made changes to the document between the time +we requested it from the database and the time we sent our updates. Making +changes to a document after someone else has modified it without first +inspecting those changes is usually a bad idea. + +Forcing clients to hand back the correct document revision is the heart of +CouchDB's optimistic concurrency. + +We have a laptop we want to keep synchronized with our desktop computer. +With all our playlists on our desktop, the first step is to +"restore from backup" onto our laptop. This is the first time we've done this, +so afterward our laptop should hold an exact replica of our desktop playlist +collection. + +After editing our Argentine Tango playlist on our laptop to add a few new +songs we've purchased, we want to save our changes. The backup application +replaces the playlist document in our laptop CouchDB database and a new +document revision is generated. A few days later, we remember our new songs +and want to copy the playlist across to our desktop computer. As illustrated +in :ref:`intro/consistency-06`, the backup application copies the new document +and the new revision to the desktop CouchDB database. Both CouchDB databases +now have the same document revision. + + +.. _intro/consistency-06: + +.. figure:: ../../images/intro-consistency-06.png + :align: center + :alt: Synchronizing between two databases + + Figure 6. Synchronizing between two databases + + +Because CouchDB tracks document revisions, it ensures that updates like these +will work only if they are based on current information. If we had made +modifications to the playlist backups between synchronization, +things wouldn't go as smoothly. + +We back up some changes on our laptop and forget to synchronize. A few days +later, we're editing playlists on our desktop computer, make a backup, +and want to synchronize this to our laptop. As illustrated in +:ref:`intro/consistency-07`, when our backup application tries to replicate +between the two databases, CouchDB sees that the changes being sent from our +desktop computer are modifications of out-of-date documents and helpfully +informs us that there has been a conflict. + +Recovering from this error is easy to accomplish from an application +perspective. Just download CouchDB's version of the playlist and provide an +opportunity to merge the changes or save local modifications into a new +playlist. + + +.. _intro/consistency-07: + +.. figure:: ../../images/intro-consistency-07.png + :align: center + :alt: Synchronization conflicts between two databases + + Figure 7. Synchronization conflicts between two databases + + +Wrapping Up +=========== + +CouchDB's design borrows heavily from web architecture and the lessons +learned deploying massively distributed systems on that architecture. +By understanding why this architecture works the way it does, +and by learning to spot which parts of your application can be easily +distributed and which parts cannot, you'll enhance your ability to design +distributed and scalable applications, with CouchDB or without it. + +We've covered the main issues surrounding CouchDB's consistency model and +hinted at some of the benefits to be had when you work *with* CouchDB and not +against it. But enough theory -- let's get up and running and see what all the +fuss is about! diff --git a/share/doc/src/intro/curl.rst b/share/doc/src/intro/curl.rst new file mode 100644 index 000000000..06b62b597 --- /dev/null +++ b/share/doc/src/intro/curl.rst @@ -0,0 +1,122 @@ +.. Licensed under the Apache License, Version 2.0 (the "License"); you may not +.. use this file except in compliance with the License. You may obtain a copy of +.. the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +.. License for the specific language governing permissions and limitations under +.. the License. + +.. _intro/curl: + +============================== +cURL: Your Command Line Friend +============================== + +The ``curl`` utility is a command line tool available on Unix, Linux, +Mac OS X and Windows and many other platforms. ``curl`` provides easy +access to the HTTP protocol (among others) directly from the +command-line and is therefore an ideal way of interacting with CouchDB +over the HTTP REST API. + +For simple ``GET`` requests you can supply the URL of the request. For +example, to get the database information: + +.. code-block:: bash + + shell> curl http://127.0.0.1:5984 + +This returns the database information (formatted in the output below for +clarity): + +.. code-block:: json + + { + "couchdb": "Welcome", + "uuid": "85fb71bf700c17267fef77535820e371", + "vendor": { + "name": "The Apache Software Foundation", + "version": "1.4.0" + }, + "version": "1.4.0" + } + + +.. note:: For some URLs, especially those that include special characters such + as ampersand, exclamation mark, or question mark, you should quote + the URL you are specifying on the command line. For example: + + .. code-block:: bash + + shell> curl 'http://couchdb:5984/_uuids?count=5' + +You can explicitly set the HTTP command using the ``-X`` command line +option. For example, when creating a database, you set the name of the +database in the URL you send using a PUT request: + +.. code-block:: bash + + shell> curl -X PUT http://127.0.0.1:5984/demo + {"ok":true} + +But to obtain the database information you use a ``GET`` request (with +the return information formatted for clarity): + +.. code-block:: bash + + shell> curl -X GET http://127.0.0.1:5984/demo + { + "compact_running" : false, + "doc_count" : 0, + "db_name" : "demo", + "purge_seq" : 0, + "committed_update_seq" : 0, + "doc_del_count" : 0, + "disk_format_version" : 5, + "update_seq" : 0, + "instance_start_time" : "1306421773496000", + "disk_size" : 79 + } + +For certain operations, you must specify the content type of request, +which you do by specifying the ``Content-Type`` header using the ``-H`` +command-line option: + +.. code-block:: bash + + shell> curl -H 'Content-Type: application/json' http://127.0.0.1:5984/_uuids + +You can also submit 'payload' data, that is, data in the body of the +HTTP request using the ``-d`` option. This is useful if you need to +submit JSON structures, for example document data, as part of the +request. For example, to submit a simple document to the ``demo`` +database: + +.. code-block:: bash + + shell> curl -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:5984/demo \ + -d '{"company": "Example, Inc."}' + {"ok":true,"id":"8843faaf0b831d364278331bc3001bd8", + "rev":"1-33b9fbce46930280dab37d672bbc8bb9"} + +In the above example, the argument after the ``-d`` option is the JSON +of the document we want to submit. + +The document can be accessed by using the automatically generated +document ID that was returned: + +.. code-block:: bash + + shell> curl -X GET http://127.0.0.1:5984/demo/8843faaf0b831d364278331bc3001bd8 + {"_id":"8843faaf0b831d364278331bc3001bd8", + "_rev":"1-33b9fbce46930280dab37d672bbc8bb9", + "company":"Example, Inc."} + +The API samples in the :ref:`api/basics` show the HTTP command, URL and any +payload information that needs to be submitted (and the expected return +value). All of these examples can be reproduced using ``curl`` with the +command-line examples shown above. diff --git a/share/doc/src/intro.rst b/share/doc/src/intro/futon.rst index 57eb2358c..3f93f5ee9 100644 --- a/share/doc/src/intro.rst +++ b/share/doc/src/intro/futon.rst @@ -10,32 +10,11 @@ .. License for the specific language governing permissions and limitations under .. the License. -============ -Introduction -============ +.. _intro/futon: -There are two interfaces to CouchDB, the built-in Futon web-based -interface and the CouchDB API accessed through the HTTP REST interface. -The former is the simplest way to view and monitor your CouchDB -installation and perform a number of basic database and system -operations. More information on using the Futon interface can be found -in :ref:`using-futon`. - -The primary way to interact with the CouchDB API is to use a client -library or other interface that provides access to the underlying -functionality through your chosen language or platform. However, since -the API is supported through HTTP REST, you can interact with your -CouchDB with any solution that supports the HTTP protocol. - -There are a number of different tools that talk the HTTP protocol and -allow you to set and configure the necessary information. One tool for -this that allows for access from the command-line is ``curl``. See -:ref:`using-curl`. - -.. _using-futon: - -Using Futon -=========== +=================================== +Futon: Web GUI Administration Panel +=================================== Futon is a native web-based interface built into CouchDB. It provides a basic interface to the majority of the functionality, including the @@ -50,7 +29,7 @@ interface to the databases, configuration or replication systems. The side panel on the right provides navigation to the main areas of Futon interface: -.. figure:: ../images/futon-overview.png +.. figure:: ../../images/futon-overview.png :align: center :alt: Futon Overview @@ -96,10 +75,11 @@ The main sections are: you select this page, you can run the tests by using the Run All button. This will execute all the tests, which may take some time. + .. _futon-management: Managing Databases and Documents --------------------------------- +================================ You can manage databases and documents within Futon using the main Overview section of the Futon interface. @@ -107,7 +87,7 @@ Overview section of the Futon interface. To create a new database, click the Create Database ELLIPSIS button. You will be prompted for the database name, as shown in the figure below. -.. figure:: ../images/futon-createdb.png +.. figure:: ../../images/futon-createdb.png :align: center :alt: Creating a Database @@ -125,7 +105,7 @@ saving the document back into the database. For example, the figure below shows the editor for a single document, a newly created document with a single ID, the document ``_id`` field. -.. figure:: ../images/futon-editdoc.png +.. figure:: ../../images/futon-editdoc.png :align: center :alt: Editing a Document @@ -153,7 +133,7 @@ To add a field to the document: the page. This will save the document, and then display the new document with the saved revision information (the ``_rev`` field). - .. figure:: ../images/futon-editeddoc.png + .. figure:: ../../images/futon-editeddoc.png :align: center :alt: Edited Document @@ -163,17 +143,18 @@ The same basic interface is used for all editing operations within Futon. You *must* remember to save the individual element (fieldname, value) using the green tick button, before then saving the document. + .. _futon-replication: Configuring Replication ------------------------ +======================= When you click the Replicator option within the Tools menu you are presented with the Replicator screen. This allows you to start replication between two databases by filling in or select the appropriate options within the form provided. -.. figure:: ../images/futon-replform.png +.. figure:: ../../images/futon-replform.png :align: center :alt: Replication Form @@ -203,113 +184,3 @@ returned when the replication process completes by the API. The Replicator tool is an interface to the underlying replication API. For more information, see :ref:`api/server/replicate`. For more information on replication, see :ref:`replication`. - -.. _using-curl: - -Using ``curl`` -============== - -The ``curl`` utility is a command line tool available on Unix, Linux, -Mac OS X and Windows and many other platforms. ``curl`` provides easy -access to the HTTP protocol (among others) directly from the -command-line and is therefore an ideal way of interacting with CouchDB -over the HTTP REST API. - -For simple ``GET`` requests you can supply the URL of the request. For -example, to get the database information: - -.. code-block:: bash - - shell> curl http://127.0.0.1:5984 - -This returns the database information (formatted in the output below for -clarity): - -.. code-block:: json - - { - "couchdb": "Welcome", - "uuid": "85fb71bf700c17267fef77535820e371", - "vendor": { - "name": "The Apache Software Foundation", - "version": "1.4.0" - }, - "version": "1.4.0" - } - - -.. note:: For some URLs, especially those that include special characters such - as ampersand, exclamation mark, or question mark, you should quote - the URL you are specifying on the command line. For example: - - .. code-block:: bash - - shell> curl 'http://couchdb:5984/_uuids?count=5' - -You can explicitly set the HTTP command using the ``-X`` command line -option. For example, when creating a database, you set the name of the -database in the URL you send using a PUT request: - -.. code-block:: bash - - shell> curl -X PUT http://127.0.0.1:5984/demo - {"ok":true} - -But to obtain the database information you use a ``GET`` request (with -the return information formatted for clarity): - -.. code-block:: bash - - shell> curl -X GET http://127.0.0.1:5984/demo - { - "compact_running" : false, - "doc_count" : 0, - "db_name" : "demo", - "purge_seq" : 0, - "committed_update_seq" : 0, - "doc_del_count" : 0, - "disk_format_version" : 5, - "update_seq" : 0, - "instance_start_time" : "1306421773496000", - "disk_size" : 79 - } - -For certain operations, you must specify the content type of request, -which you do by specifying the ``Content-Type`` header using the ``-H`` -command-line option: - -.. code-block:: bash - - shell> curl -H 'Content-Type: application/json' http://127.0.0.1:5984/_uuids - -You can also submit 'payload' data, that is, data in the body of the -HTTP request using the ``-d`` option. This is useful if you need to -submit JSON structures, for example document data, as part of the -request. For example, to submit a simple document to the ``demo`` -database: - -.. code-block:: bash - - shell> curl -H 'Content-Type: application/json' \ - -X POST http://127.0.0.1:5984/demo \ - -d '{"company": "Example, Inc."}' - {"ok":true,"id":"8843faaf0b831d364278331bc3001bd8", - "rev":"1-33b9fbce46930280dab37d672bbc8bb9"} - -In the above example, the argument after the ``-d`` option is the JSON -of the document we want to submit. - -The document can be accessed by using the automatically generated -document ID that was returned: - -.. code-block:: bash - - shell> curl -X GET http://127.0.0.1:5984/demo/8843faaf0b831d364278331bc3001bd8 - {"_id":"8843faaf0b831d364278331bc3001bd8", - "_rev":"1-33b9fbce46930280dab37d672bbc8bb9", - "company":"Example, Inc."} - -The API samples in the :ref:`api/basics` show the HTTP command, URL and any -payload information that needs to be submitted (and the expected return -value). All of these examples can be reproduced using ``curl`` with the -command-line examples shown above. diff --git a/share/doc/src/intro/index.rst b/share/doc/src/intro/index.rst new file mode 100644 index 000000000..900dca2eb --- /dev/null +++ b/share/doc/src/intro/index.rst @@ -0,0 +1,50 @@ +.. Licensed under the Apache License, Version 2.0 (the "License"); you may not +.. use this file except in compliance with the License. You may obtain a copy of +.. the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +.. License for the specific language governing permissions and limitations under +.. the License. + + +.. _intro: + +============ +Introduction +============ + +CouchDB is a database that completely embraces the web. Store your data with +JSON documents. Access your documents with your web browser, :ref:`via HTTP +<api/basics>`. :ref:`Query <api/doc>`, :ref:`combine <views>`, +and :ref:`transform <listfun>` your documents with :ref:`JavaScript +<query-server/js>`. CouchDB works well with modern web and mobile apps. You can +even serve web apps directly out of CouchDB. And you can distribute your data, +or your apps, efficiently using CouchDB’s :ref:`incremental replication +<replication/intro>`. CouchDB supports master-master setups with +:ref:`automatic conflict <replication/conflicts>` detection. + +CouchDB comes with a suite of features, such as on-the-fly document +transformation and real-time :ref:`change notifications <changes>`, that makes +:ref:`web app <couchapp>` development a breeze. It even comes with an easy +to use :ref:`web administration console <intro/futon>`. You guessed it, +served up directly out of CouchDB! We care a lot about `distributed scaling`_. +CouchDB is highly available and partition tolerant, but is also :ref:`eventually +consistent <intro/consistency>`. And we care *a lot* about your data. +CouchDB has a fault-tolerant storage engine that puts the safety of your data +first. + +.. _distributed scaling: http://en.wikipedia.org/wiki/CAP_theorem + +.. toctree:: + :maxdepth: 2 + + overview + why + consistency + tour + futon + curl diff --git a/share/doc/src/intro/overview.rst b/share/doc/src/intro/overview.rst new file mode 100644 index 000000000..b36f8f1be --- /dev/null +++ b/share/doc/src/intro/overview.rst @@ -0,0 +1,387 @@ +.. Licensed under the Apache License, Version 2.0 (the "License"); you may not +.. use this file except in compliance with the License. You may obtain a copy of +.. the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +.. License for the specific language governing permissions and limitations under +.. the License. + + +.. _intro/overview: + +================== +Technical Overview +================== + +Document Storage +================ + +A CouchDB server hosts named databases, which store **documents**. +Each document is uniquely named in the database, and CouchDB provides +a `RESTful`_ :ref:`HTTP API <api/basics>` for reading and updating (add, edit, +delete) database documents. + +Documents are the primary unit of data in CouchDB and consist of any number +of fields and attachments. Documents also include metadata that’s maintained +by the database system. Document fields are uniquely named and contain values +of :ref:`varying types <json>` (text, number, boolean, lists, etc), +and there is no set limit to text size or element count. + +The CouchDB document update model is lockless and optimistic. +Document edits are made by client applications loading documents, +applying changes, and saving them back to the database. If another client +editing the same document saves their changes first, the client gets an edit +conflict error on save. To resolve the update conflict, the latest document +version can be opened, the edits reapplied and the update tried again. + +Document updates (add, edit, delete) are all or nothing, either succeeding +entirely or failing completely. The database never contains partially saved +or edited documents. + +.. _RESTful: http://en.wikipedia.org/wiki/REST + + +ACID Properties +=============== + +The CouchDB file layout and commitment system features all `Atomic Consistent +Isolated Durable` (`ACID`_) properties. On-disk, CouchDB never overwrites +committed data or associated structures, ensuring the database file is always +in a consistent state. This is a "crash-only" design where the CouchDB +server does not go through a shut down process, it's simply terminated. + +Document updates (add, edit, delete) are serialized, except for binary blobs +which are written concurrently. Database readers are never locked out and +never have to wait on writers or other readers. Any number of clients can be +reading documents without being locked out or interrupted by concurrent +updates, even on the same document. CouchDB read operations use a +`Multi-Version Concurrency Control` (`MVCC`_) model where each client sees a +consistent snapshot of the database from the beginning to the end of the read +operation. + +Documents are indexed in `B-trees`_ by their name (DocID) and a Sequence ID. +Each update to a database instance generates a new sequential number. +Sequence IDs are used later for incrementally finding changes in a database. +TheseBb-tree indexes are updated simultaneously when documents are saved or +deleted. The index updates always occur at the end of the file (append-only +updates). + +Documents have the advantage of data being already conveniently packaged for +storage rather than split out across numerous tables and rows in most +databases systems. When documents are committed to disk, the document fields +and metadata are packed into buffers, sequentially one document after another +(helpful later for efficient building of views). + +When CouchDB documents are updated, all data and associated indexes are +flushed to disk and the transactional commit always leaves the database +in a completely consistent state. Commits occur in two steps: + +#. All document data and associated index updates are synchronously flushed + to disk. + +#. The updated database header is written in two consecutive, identical chunks + to make up the first 4k of the file, and then synchronously flushed to disk. + +In the event of an OS crash or power failure during step 1, +the partially flushed updates are simply forgotten on restart. If such a +crash happens during step 2 (committing the header), a surviving copy of the +previous identical headers will remain, ensuring coherency of all previously +committed data. Excepting the header area, consistency checks or fix-ups +after a crash or a power failure are never necessary. + +.. _ACID: http://en.wikipedia.org/wiki/ACID +.. _MVCC: http://en.wikipedia.org/wiki/Multiversion_concurrency_control +.. _B-trees: http://en.wikipedia.org/wiki/B-tree + + +Compaction +========== + +Wasted space is recovered by occasional compaction. On schedule, or when the +database file exceeds a certain amount of wasted space, the compaction process +clones all the active data to a new file and then discards the old file. +The database remains completely online the entire time and all updates and +reads are allowed to complete successfully. The old file is deleted only when +all the data has been copied and all users transitioned to the new file. + + +Views +===== + +ACID properties only deal with storage and updates, we also need the ability +to show our data in interesting and useful ways. Unlike SQL databases where +data must be carefully decomposed into tables, data in CouchDB is stored in +semi-structured documents. CouchDB documents are flexible and each has its +own implicit structure, which alleviates the most difficult problems and +pitfalls of bi-directionally replicating table schemas and their contained data. + +But beyond acting as a fancy file server, a simple document model for data +storage and sharing is too simple to build real applications on -- it simply +doesn't do enough of the things we want and expect. We want to slice and dice +and see our data in many different ways. What is needed is a way to filter, +organize and report on data that hasn't been decomposed into tables. + +.. seealso:: + + :ref:`views` + + +View Model +---------- + +To address this problem of adding structure back to unstructured and +semi-structured data, CouchDB integrates a view model. Views are the method +of aggregating and reporting on the documents in a database, and are built +on-demand to aggregate, join and report on database documents. Views are built +dynamically and don’t affect the underlying document, you can have as many +different view representations of the same data as you like. + +View definitions are strictly virtual and only display the documents from the +current database instance, making them separate from the data they display +and compatible with replication. CouchDB views are defined inside special +**design documents** and can replicate across database instances like +regular documents, so that not only data replicates in CouchDB, +but entire application designs replicate too. + + +Javascript View Functions +------------------------- + +Views are defined using Javascript functions acting as the map part in a +`map-reduce system`_. A :ref:`view function <viewfun>` takes a CouchDB document +as an argument and then does whatever computation it needs to do to determine +the data that is to be made available through the view, if any. +It can add multiple rows to the view based on a single document, +or it can add no rows at all. + +.. _map-reduce system: http://en.wikipedia.org/wiki/MapReduce + +.. seealso:: + + :ref:`viewfun` + + +View Indexes +------------ + +Views are a dynamic representation of the actual document contents of a +database, and CouchDB makes it easy to create useful views of data. +But generating a view of a database with hundreds of thousands or millions of +documents is time and resource consuming, it's not something the system +should do from scratch each time. + +To keep view querying fast, the view engine maintains indexes of its views, +and incrementally updates them to reflect changes in the database. +CouchDB’s core design is largely optimized around the need for efficient, +incremental creation of views and their indexes. + +Views and their functions are defined inside special "design" documents, +and a design document may contain any number of uniquely named view functions. +When a user opens a view and its index is automatically updated, all the views +in the same design document are indexed as a single group. + +The view builder uses the database sequence ID to determine if the view group +is fully up-to-date with the database. If not, the view engine examines the +all database documents (in packed sequential order) changed since the last +refresh. Documents are read in the order they occur in the disk file, +reducing the frequency and cost of disk head seeks. + +The views can be read and queried simultaneously while also being refreshed. +If a client is slowly streaming out the contents of a large view, +the same view can be concurrently opened and refreshed for another client +without blocking the first client. This is true for any number of +simultaneous client readers, who can read and query the view while the index +is concurrently being refreshed for other clients without causing problems +for the readers. + +As documents are examined, their previous row values are removed from the +view indexes, if they exist. If the document is selected by a view function, +the function results are inserted into the view as a new row. + +When view index changes are written to disk, the updates are always appended +at the end of the file, serving to both reduce disk head seek times during +disk commits and to ensure crashes and power failures can not cause +corruption of indexes. If a crash occurs while updating a view index, +the incomplete index updates are simply lost and rebuilt incrementally from +its previously committed state. + + +Security and Validation +======================= + +To protect who can read and update documents, CouchDB has a simple reader +access and update validation model that can be extended to implement custom +security models. + +.. seealso:: + + :ref:`api/db/security` + + +Administrator Access +-------------------- + +CouchDB database instances have administrator accounts. Administrator +accounts can create other administrator accounts and update design documents. +Design documents are special documents containing view definitions and other +special formulas, as well as regular fields and blobs. + + +Update Validation +----------------- + +As documents written to disk, they can be validated dynamically by javascript +functions for both security and data validation. When the document passes +all the formula validation criteria, the update is allowed to continue. +If the validation fails, the update is aborted and the user client gets an +error response. + +Both the user's credentials and the updated document are given as inputs to +the validation formula, and can be used to implement custom security models +by validating a user's permissions to update a document. + +A basic "author only" update document model is trivial to implement, +where document updates are validated to check if the user is listed in an +"author" field in the existing document. More dynamic models are also possible, +like checking a separate user account profile for permission settings. + +The update validations are enforced for both live usage and replicated +updates, ensuring security and data validation in a shared, distributed system. + +.. seealso:: + + :ref:`vdufun` + + +Distributed Updates and Replication +=================================== + +CouchDB is a peer-based distributed database system, it allows for users and +servers to access and update the same shared data while disconnected and then +bi-directionally replicate those changes later. + +The CouchDB document storage, view and security models are designed to work +together to make true bi-directional replication efficient and reliable. +Both documents and designs can replicate, allowing full database applications +(including application design, logic and data) to be replicated to laptops +for offline use, or replicated to servers in remote offices where slow or +unreliable connections make sharing data difficult. + +The replication process is incremental. At the database level, +replication only examines documents updated since the last replication. +Then for each updated document, only fields and blobs that have changed are +replicated across the network. If replication fails at any step, due to network +problems or crash for example, the next replication restarts at the same +document where it left off. + +Partial replicas can be created and maintained. Replication can be filtered +by a javascript function, so that only particular documents or those meeting +specific criteria are replicated. This can allow users to take subsets of a +large shared database application offline for their own use, while maintaining +normal interaction with the application and that subset of data. + + +Conflicts +--------- + +Conflict detection and management are key issues for any distributed edit +system. The CouchDB storage system treats edit conflicts as a common state, +not an exceptional one. The conflict handling model is simple and +"non-destructive" while preserving single document semantics and allowing for +decentralized conflict resolution. + +CouchDB allows for any number of conflicting documents to exist +simultaneously in the database, with each database instance deterministically +deciding which document is the "winner" and which are conflicts. Only the +winning document can appear in views, while "losing" conflicts are still +accessible and remain in the database until deleted or purged during +database compaction. Because conflict documents are still regular documents, +they replicate just like regular documents and are subject to the same +security and validation rules. + +When distributed edit conflicts occur, every database replica sees the same +winning revision and each has the opportunity to resolve the conflict. +Resolving conflicts can be done manually or, depending on the nature of the +data and the conflict, by automated agents. The system makes decentralized +conflict resolution possible while maintaining single document database +semantics. + +Conflict management continues to work even if multiple disconnected users or +agents attempt to resolve the same conflicts. If resolved conflicts result in +more conflicts, the system accommodates them in the same manner, determining +the same winner on each machine and maintaining single document semantics. + +.. seealso:: + + :ref:`replication/conflicts` + + +Applications +------------ + +Using just the basic replication model, many traditionally single server +database applications can be made distributed with almost no extra work. +CouchDB replication is designed to be immediately useful for basic database +applications, while also being extendable for more elaborate and full-featured +uses. + +With very little database work, it is possible to build a distributed +document management application with granular security and full revision +histories. Updates to documents can be implemented to exploit incremental +field and blob replication, where replicated updates are nearly as efficient +and incremental as the actual edit differences ("diffs"). + +The CouchDB replication model can be modified for other distributed update +models. If the storage engine is enhanced to allow multi-document update +transactions, it is possible to perform Subversion-like "all or nothing" +atomic commits when replicating with an upstream server, such that any single +document conflict or validation failure will cause the entire update to fail. +Like Subversion, conflicts would be resolved by doing a "pull" replication to +force the conflicts locally, then merging and re-replicating to the upstream +server. + + +Implementation +============== + +CouchDB is built on the `Erlang OTP platform`_, a functional, +concurrent programming language and development platform. Erlang was +developed for real-time telecom applications with an extreme emphasis on +reliability and availability. + +Both in syntax and semantics, Erlang is very different from conventional +programming languages like C or Java. Erlang uses lightweight "processes" and +message passing for concurrency, it has no shared state threading and all +data is immutable. The robust, concurrent nature of Erlang is ideal for a +database server. + +CouchDB is designed for lock-free concurrency, in the conceptual model and +the actual Erlang implementation. Reducing bottlenecks and avoiding locks +keeps the entire system working predictably under heavy loads. CouchDB can +accommodate many clients replicating changes, opening and updating documents, +and querying views whose indexes are simultaneously being refreshed for +other clients, without needing locks. + +For higher availability and more concurrent users, CouchDB is designed for +"shared nothing" clustering. In a "shared nothing" cluster, each machine +is independent and replicates data with its cluster mates, allowing individual +server failures with zero downtime. And because consistency scans +and fix-ups aren’t needed on restart, +if the entire cluster fails -- due to a power outage in a datacenter, +for example -- the entire CouchDB distributed system becomes immediately +available after a restart. + +CouchDB is built from the start with a consistent vision of a distributed +document database system. Unlike cumbersome attempts to bolt distributed +features on top of the same legacy models and databases, +it is the result of careful ground-up design, engineering and integration. +The document, view, security and replication models, the special purpose query +language, the efficient and robust disk layout and the concurrent and reliable +nature of the Erlang platform are all carefully integrated for a reliable +and efficient system. + +.. _Erlang OTP platform: http://www.erlang.org/ diff --git a/share/doc/src/intro/tour.rst b/share/doc/src/intro/tour.rst new file mode 100644 index 000000000..af91f5e4c --- /dev/null +++ b/share/doc/src/intro/tour.rst @@ -0,0 +1,534 @@ +.. Licensed under the Apache License, Version 2.0 (the "License"); you may not +.. use this file except in compliance with the License. You may obtain a copy of +.. the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +.. License for the specific language governing permissions and limitations under +.. the License. + + +.. _intro/tour: + +=============== +Getting Started +=============== + +In this chapter, we'll take a quick tour of CouchDB's features, +familiarizing ourselves with Futon, the built-in administration interface. +We'll create our first document and experiment with CouchDB views. + + +All Systems Are Go! +=================== + +We'll have a very quick look at CouchDB's bare-bones Application Programming +Interface (API) by using the command-line utility curl. Please note that this +is not the only way of talking to CouchDB. We will show you plenty more +throughout the rest of the book. What's interesting about curl is that it +gives you control over raw HTTP requests, and you can see exactly what is +going on "underneath the hood" of your database. + +Make sure CouchDB is still running, and then do:: + + curl http://127.0.0.1:5984/ + +This issues a GET request to your newly installed CouchDB instance. + +The reply should look something like: + +.. code-block:: javascript + + { + "couchdb": "Welcome", + "uuid": "85fb71bf700c17267fef77535820e371", + "version": "1.4.0", + "vendor": { + "version": "1.4.0", + "name": "The Apache Software Foundation" + } + } + +Not all that spectacular. CouchDB is saying "hello" with the running version +number. + +Next, we can get a list of databases:: + + curl -X GET http://127.0.0.1:5984/_all_dbs + +All we added to the previous request is the _all_dbs string. + +The response should look like:: + + ["_replicator","_users"] + +Oh, that's right, we didn't create any databases yet! All we see is an empty +list. + +.. note:: + + The curl command issues GET requests by default. You can issue POST requests + using ``curl -X POST``. To make it easy to work with our terminal history, + we usually use the ``-X`` option even when issuing GET requests. + If we want to send a POST next time, all we have to change is the method. + + HTTP does a bit more under the hood than you can see in the examples here. + If you're interested in every last detail that goes over the wire, + pass in the ``-v`` option (e.g., ``curl -vX GET``), which will show you + the server curl tries to connect to, the request headers it sends, + and response headers it receives back. Great for debugging! + +Let's create a database:: + + curl -X PUT http://127.0.0.1:5984/baseball + +CouchDB will reply with:: + + {"ok":true} + +Retrieving the list of databases again shows some useful results this time:: + + curl -X GET http://127.0.0.1:5984/_all_dbs + +:: + + ["baseball"] + +.. note:: + + We should mention JavaScript Object Notation (JSON) here, + the data format CouchDB speaks. JSON is a lightweight data interchange format + based on JavaScript syntax. Because JSON is natively compatible with + JavaScript, your web browser is an ideal client for CouchDB. + + Brackets (``[]``) represent ordered lists, and curly braces (``{}``) represent + key/value dictionaries. Keys must be strings, delimited by quotes (``"``), + and values can be strings, numbers, booleans, lists, + or key/value dictionaries. For a more detailed description of JSON, + see Appendix E, JSON Primer. + +Let's create another database:: + + curl -X PUT http://127.0.0.1:5984/baseball + +CouchDB will reply with:: + + {"error":"file_exists","reason":"The database could not be created, + the file already exists."} + +We already have a database with that name, so CouchDB will respond with an +error. Let's try again with a different database name:: + + curl -X PUT http://127.0.0.1:5984/plankton + +CouchDB will reply with:: + + {"ok":true} + +Retrieving the list of databases yet again shows some useful results:: + + curl -X GET http://127.0.0.1:5984/_all_dbs + +CouchDB will respond with:: + + ["baseball", "plankton"] + +To round things off, let's delete the second database:: + + curl -X DELETE http://127.0.0.1:5984/plankton + +CouchDB will reply with:: + + {"ok":true} + +The list of databases is now the same as it was before:: + + curl -X GET http://127.0.0.1:5984/_all_dbs + +CouchDB will respond with:: + + ["baseball"] + +For brevity, we'll skip working with documents, as the next section covers a +different and potentially easier way of working with CouchDB that should +provide experience with this. As we work through the example, +keep in mind that "under the hood" everything is being done by the +application exactly as you have been doing here manually. +Everything is done using GET, PUT, POST, and DELETE with a URI. + + +Welcome to Futon +================ + +After having seen CouchDB's raw API, let's get our feet wet by playing with +Futon, the built-in administration interface. Futon provides full access to +all of CouchDB's features and makes it easy to work with some of the more +complex ideas involved. With Futon we can create and destroy databases; view +and edit documents; compose and run MapReduce views; and trigger replication +between databases. + +To load Futon in your browser, visit:: + + http://127.0.0.1:5984/_utils/ + +If you're running version 0.9 or later, you should see something similar to +:ref:`intro/tour-01`. In later chapters, we'll focus on using CouchDB from +server-side languages such as Ruby and Python. As such, this chapter is a great +opportunity to showcase an example of natively serving up a dynamic web +application using nothing more than CouchDB's integrated web server, something +you may wish to do with your own applications. + +The first thing we should do with a fresh installation of CouchDB is run the +test suite to verify that everything is working properly. This assures us +that any problems we may run into aren't due to bothersome issues with our +setup. By the same token, failures in the Futon test suite are a red flag, +telling us to double-check our installation before attempting to use a +potentially broken database server, saving us the confusion when nothing +seems to be working quite like we expect! + + +.. _intro/tour-01: + +.. figure:: ../../images/intro-tour-01.png + :align: center + :alt: The Futon welcome screen + + Figure 1. The Futon welcome screen + + +Some common network configurations cause the replication test to fail when +accessed via the localhost address. You can fix this by accessing CouchDB via +127.0.0.1, e.g. http://127.0.0.1:5984/_utils/. + +Navigate to the test suite by clicking "Test Suite" on the Futon sidebar, +then click "run all" at the top to kick things off. :ref:`intro/tour-02` +shows the Futon test suite running some tests. + + +.. _intro/tour-02: + +.. figure:: ../../images/intro-tour-02.png + :align: center + :alt: The Futon test suite running some tests + + Figure 2. The Futon test suite running some tests + + +Because the test suite is run from the browser, not only does it test that +CouchDB is functioning properly, it also verifies that your browser's +connection to the database is properly configured, which can be very handy +for diagnosing misbehaving proxies or other HTTP middleware. + +If the test suite has an inordinate number of failures, +you'll need to see the troubleshooting section in Appendix D, +Installing from Source for the next steps to fix your installation. + +Now that the test suite is finished, you've verified that your CouchDB +installation is successful and you're ready to see what else Futon has to offer. + + +Your First Database and Document +================================ + +Creating a database in Futon is simple. From the overview page, +click "Create Database." When asked for a name, enter hello-world and click +the Create button. + +After your database has been created, Futon will display a list of all its +documents. This list will start out empty (:ref:`intro/tour-03`), so let's +create our first document. Click the "New Document" link and then the Create +button in the pop up. Make sure to leave the document ID blank, +and CouchDB will generate a UUID for you. + +For demoing purposes, having CouchDB assign a UUID is fine. When you write +your first programs, we recommend assigning your own UUIDs. If your rely on +the server to generate the UUID and you end up making two POST requests +because the first POST request bombed out, you might generate two docs and +never find out about the first one because only the second one will be +reported back. Generating your own UUIDs makes sure that you'll never end up +with duplicate documents. + +Futon will display the newly created document, with its _id and _rev as the +only fields. To create a new field, click the "Add Field" button. We'll call +the new field hello. Click the green check icon (or hit the Enter key) to +finalize creating the hello field. Double-click the hello field's value +(default null) to edit it. + +You can experiment with other JSON values; e.g., ``[1, 2, "c"]`` or +``{"foo": "bar"}``. Once you've entered your values into the document, +make a note of its ``_rev`` attribute and click "Save Document." The result +should look like :ref:`intro/tour-04` document in Futon". + + +.. _intro/tour-03: + +.. figure:: ../../images/intro-tour-03.png + :align: center + :alt: An empty database in Futon + + Figure 3. An empty database in Futon + + +.. _intro/tour-04: + +.. figure:: ../../images/intro-tour-04.png + :align: center + :alt: A "hello world" document in Futon + + Figure 4. A "hello world" document in Futon + + +You'll notice that the document's _rev has changed. We'll go into more detail +about this in later chapters, but for now, the important thing to note is +that _rev acts like a safety feature when saving a document. As long as you +and CouchDB agree on the most recent _rev of a document, you can successfully +save your changes. + +Futon also provides a way to display the underlying JSON data, +which can be more compact and easier to read, depending on what sort of data +you are dealing with. To see the JSON version of our "hello world" document, +click the Source tab. The result should look like :ref:`intro/tour-05`. + + +.. _intro/tour-05: + +.. figure:: ../../images/intro-tour-05.png + :align: center + :alt: The JSON source of a "hello world" document in Futon + + Figure 5. The JSON source of a "hello world" document in Futon + + +Running a Query Using MapReduce +=============================== + +Traditional relational databases allow you to run any queries you like as +long as your data is structured correctly. In contrast, +CouchDB uses predefined map and reduce functions in a style known as +MapReduce. These functions provide great flexibility because they can adapt +to variations in document structure, and indexes for each document can be +computed independently and in parallel. The combination of a map and a reduce +function is called a view in CouchDB terminology. + +For experienced relational database programmers, MapReduce can take some +getting used to. Rather than declaring which rows from which tables to +include in a result set and depending on the database to determine the most +efficient way to run the query, reduce queries are based on simple range +requests against the indexes generated by your map functions. + +Map functions are called once with each document as the argument. +The function can choose to skip the document altogether or emit one or more +view rows as key/value pairs. Map functions may not depend on any information +outside of the document. This independence is what allows CouchDB views to be +generated incrementally and in parallel. + +CouchDB views are stored as rows that are kept sorted by key. This makes +retrieving data from a range of keys efficient even when there are thousands +or millions of rows. When writing CouchDB map functions, +your primary goal is to build an index that stores related data under nearby +keys. + +Before we can run an example MapReduce view, we'll need some data to run it +on. We'll create documents carrying the price of various supermarket items as +found at different shops. Let's create documents for apples, oranges, +and bananas. (Allow CouchDB to generate the _id and _rev fields.) Use Futon +to create documents that have a final JSON structure that looks like this: + +.. code-block:: javascript + + { + "_id": "00a271787f89c0ef2e10e88a0c0001f4", + "_rev": "1-2628a75ac8c3abfffc8f6e30c9949fd6", + "item": "apple", + "prices": { + "Fresh Mart": 1.59, + "Price Max": 5.99, + "Apples Express": 0.79 + } + } + +This document should look like :ref:`intro/tour-06` when entered into Futon. + + +.. _intro/tour-06: + +.. figure:: ../../images/intro-tour-06.png + :align: center + :alt: An example document with apple prices in Futon + + Figure 6. An example document with apple prices in Futon + + +OK, now that that's done, let's create the document for oranges: + +.. code-block:: javascript + + { + "_id": "00a271787f89c0ef2e10e88a0c0003f0", + "_rev": "1-e9680c5d9a688b4ff8dd68549e8e072c", + "item": "orange", + "prices": { + "Fresh Mart": 1.99, + "Price Max": 3.19, + "Citrus Circus": 1.09 + } + } + +And finally, the document for bananas: + +.. code-block:: javascript + + { + "_id": "00a271787f89c0ef2e10e88a0c00048b", + "_rev": "1-60e25d93dc12884676d037400a6fa189", + "item": "banana", + "prices": { + "Fresh Mart": 1.99, + "Price Max": 0.79, + "Banana Montana": 4.22 + } + } + +Imagine we're catering a big luncheon, but the client is very price-sensitive. +To find the lowest prices, we're going to create our first view, +which shows each fruit sorted by price. Click "hello-world" to return to the +hello-world overview, and then from the "select view" menu choose "Temporary +view…" to create a new view. + +Edit the map function, on the left, so that it looks like the following: + +.. code-block:: javascript + + function(doc) { + var shop, price, value; + if (doc.item && doc.prices) { + for (shop in doc.prices) { + price = doc.prices[shop]; + value = [doc.item, shop]; + emit(price, value); + } + } + } + +This is a JavaScript function that CouchDB runs for each of our documents as +it computes the view. We'll leave the reduce function blank for the time being. + +Click "Run" and you should see result rows like in :ref:`intro/tour-08`, +with the various items sorted by price. This map function could be even more +useful if it grouped the items by type so that all the prices for bananas were +next to each other in the result set. CouchDB's key sorting system allows any +valid JSON object as a key. In this case, we'll emit an array of [item, price] +so that CouchDB groups by item type and price. + + +.. _intro/tour-08: + +.. figure:: ../../images/intro-tour-08.png + :align: center + :alt: The results of running a view in Futon + + Figure 8. The results of running a view in Futon + + +Let's modify the view function so that it looks like this: + +.. code-block:: javascript + + function(doc) { + var shop, price, key; + if (doc.item && doc.prices) { + for (shop in doc.prices) { + price = doc.prices[shop]; + key = [doc.item, price]; + emit(key, shop); + } + } + } + +Here, we first check that the document has the fields we want to use. CouchDB +recovers gracefully from a few isolated map function failures, +but when a map function fails regularly (due to a missing required field or +other JavaScript exception), CouchDB shuts off its indexing to prevent any +further resource usage. For this reason, it's important to check for the +existence of any fields before you use them. In this case, +our map function will skip the first "hello world" document we created +without emitting any rows or encountering any errors. The result of this +query should look like :ref:`intro/tour-09`. + + +.. _intro/tour-09: + +.. figure:: ../../images/intro-tour-09.png + :align: center + :alt: The results of running a view after grouping by item type and price + + Figure 9. The results of running a view after grouping by item type and price + + +Once we know we've got a document with an item type and some prices, +we iterate over the item's prices and emit key/values pairs. The key is an +array of the item and the price, and forms the basis for CouchDB's sorted +index. In this case, the value is the name of the shop where the item can be +found for the listed price. + +View rows are sorted by their keys -- in this example, first by item, +then by price. This method of complex sorting is at the heart of creating +useful indexes with CouchDB. + +MapReduce can be challenging, especially if you've spent years working with +relational databases. The important things to keep in mind are that map +functions give you an opportunity to sort your data using any key you choose, +and that CouchDB's design is focused on providing fast, +efficient access to data within a range of keys. + + +Triggering Replication +====================== + +Futon can trigger replication between two local databases, +between a local and remote database, or even between two remote databases. +We'll show you how to replicate data from one local database to another, +which is a simple way of making backups of your databases as we're working +through the examples. + +First we'll need to create an empty database to be the target of replication. +Return to the overview and create a database called hello-replication. +Now click "Replicator" in the sidebar and choose hello-world as the source +and hello-replication as the target. Click "Replicate" to replicate your +database. The result should look something like :ref:`intro/tour-10`. + + +.. _intro/tour-10: + +.. figure:: ../../images/intro-tour-10.png + :align: center + :alt: Running database replication in Futon + + Figure 10. Running database replication in Futon + + +.. note:: + + For larger databases, replication can take much longer. It is important to + leave the browser window open while replication is taking place. + As an alternative, you can trigger replication via curl or some other HTTP + client that can handle long-running connections. If your client closes the + connection before replication finishes, you'll have to retrigger it. + Luckily, CouchDB's replication can take over from where it left off + instead of starting from scratch. + + +Wrapping Up +=========== + +Now that you've seen most of Futon's features, you'll be prepared to dive in +and inspect your data as we build our example application in the next few +chapters. Futon's pure JavaScript approach to managing CouchDB shows how it's +possible to build a fully featured web application using only CouchDB's HTTP +API and integrated web server. + +But before we get there, we'll have another look at CouchDB's HTTP API -- now +with a magnifying glass. Let's curl up on the couch and relax. diff --git a/share/doc/src/intro/why.rst b/share/doc/src/intro/why.rst new file mode 100644 index 000000000..1b902d8d8 --- /dev/null +++ b/share/doc/src/intro/why.rst @@ -0,0 +1,315 @@ +.. Licensed under the Apache License, Version 2.0 (the "License"); you may not +.. use this file except in compliance with the License. You may obtain a copy of +.. the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +.. License for the specific language governing permissions and limitations under +.. the License. + + +.. _intro/why: + +============ +Why CouchDB? +============ + +Apache CouchDB is one of a new breed of database management systems. +This chapter explains why there's a need for new systems as well as the +motivations behind building CouchDB. + +As CouchDB developers, we're naturally very excited to be using CouchDB. +In this chapter we'll share with you the reasons for our enthusiasm. +We'll show you how CouchDB's schema-free document model is a better fit +for common applications, how the built-in query engine is a powerful way +to use and process your data, and how CouchDB's design lends itself +to modularization and scalability. + + +Relax +===== + +If there's one word to describe CouchDB, it is *relax*. It is in the title of +this book, it is the byline to CouchDB's official logo, +and when you start CouchDB, you see:: + + Apache CouchDB has started. Time to relax. + +Why is relaxation important? Developer productivity roughly doubled in the +last five years. The chief reason for the boost is more powerful tools that +are easier to use. Take Ruby on Rails as an example. It is an infinitely +complex framework, but it's easy to get started with. Rails is a success +story because of the core design focus on ease of use. This is one reason why +CouchDB is relaxing: learning CouchDB and understanding its core concepts +should feel natural to most everybody who has been doing any work on the Web. +And it is still pretty easy to explain to non-technical people. + +Getting out of the way when creative people try to build specialized +solutions is in itself a core feature and one thing that CouchDB aims to get +right. We found existing tools too cumbersome to work with during development +or in production, and decided to focus on making CouchDB easy, even a pleasure, +to use. + +Another area of relaxation for CouchDB users is the production setting. +If you have a live running application, CouchDB again goes out of its way +to avoid troubling you. Its internal architecture is fault-tolerant, +and failures occur in a controlled environment and are dealt with gracefully. +Single problems do not cascade through an entire server system but stay +isolated in single requests. + +CouchDB's core concepts are simple (yet powerful) and well understood. +Operations teams (if you have a team; otherwise, that's you) do not have to +fear random behavior and untraceable errors. If anything should go wrong, +you can easily find out what the problem is, but these situations are rare. + +CouchDB is also designed to handle varying traffic gracefully. For instance, +if a website is experiencing a sudden spike in traffic, CouchDB will generally +absorb a lot of concurrent requests without falling over. It may take a little +more time for each request, but they all get answered. When the spike is over, +CouchDB will work with regular speed again. + +The third area of relaxation is growing and shrinking the underlying hardware +of your application. This is commonly referred to as scaling. CouchDB enforces +a set of limits on the programmer. On first look, CouchDB might seem +inflexible, but some features are left out by design for the simple reason +that if CouchDB supported them, it would allow a programmer to create +applications that couldn't deal with scaling up or down. + +.. note:: + CouchDB doesn't let you do things that would get you in trouble later on. + This sometimes means you'll have to unlearn best practices you might have + picked up in your current or past work. + + +A Different Way to Model Your Data +================================== + +We believe that CouchDB will drastically change the way you build +document-based applications. CouchDB combines an intuitive document storage +model with a powerful query engine in a way that's so simple you'll probably +be tempted to ask, “Why has no one built something like this before?” + + Django may be built for the Web, but CouchDB is built of the Web. I've + never seen software that so completely embraces the philosophies behind + HTTP. CouchDB makes Django look old-school in the same way that Django + makes ASP look outdated. + + -- Jacob Kaplan-Moss, Django developer + +CouchDB's design borrows heavily from web architecture and the concepts of +resources, methods, and representations. It augments this with powerful ways +to query, map, combine, and filter your data. Add fault tolerance, extreme +scalability, and incremental replication, and CouchDB defines a sweet spot +for document databases. + + +A Better Fit for Common Applications +==================================== + +We write software to improve our lives and the lives of others. Usually this +involves taking some mundane information such as contacts, invoices, +or receipts and manipulating it using a computer application. CouchDB is a +great fit for common applications like this because it embraces the natural +idea of evolving, self-contained documents as the very core of its data model. + + +Self-Contained Data +------------------- + +An invoice contains all the pertinent information about a single transaction +the seller, the buyer, the date, and a list of the items or services sold. +As shown in :ref:`intro/why-01`, there's no abstract reference on this +piece of paper that points to some other piece of paper with the seller's +name and address. Accountants appreciate the simplicity of having everything +in one place. And given the choice, programmers appreciate that, too. + + +.. _intro/why-01: + +.. figure:: ../../images/intro-why-01.png + :align: center + :alt: Self-contained documents + + Figure 1. Self-contained documents + + +Yet using references is exactly how we model our data in a relational +database! Each invoice is stored in a table as a row that refers to other +rows in other tables one row for seller information, one for the buyer, +one row for each item billed, and more rows still to describe the item +details, manufacturer details, and so on and so forth. + +This isn't meant as a detraction of the relational model, which is widely +applicable and extremely useful for a number of reasons. Hopefully, though, it +illustrates the point that sometimes your model may not “fit” your data +in the way it occurs in the real world. + +Let's take a look at the humble contact database to illustrate a different +way of modeling data, one that more closely “fits” its real-world counterpart +-- a pile of business cards. Much like our invoice example, a business card +contains all the important information, right there on the cardstock. +We call this “self-contained” data, and it's an important concept +in understanding document databases like CouchDB. + + +Syntax and Semantics +-------------------- + +Most business cards contain roughly the same information -- someone's identity, +an affiliation, and some contact information. While the exact form of this +information can vary between business cards, the general information being +conveyed remains the same, and we're easily able to recognize it as a +business card. In this sense, we can describe a business card as a *real-world +document*. + +Jan's business card might contain a phone number but no fax number, +whereas J. Chris's business card contains both a phone and a fax number. Jan +does not have to make his lack of a fax machine explicit by writing something +as ridiculous as “Fax: None” on the business card. Instead, simply omitting +a fax number implies that he doesn't have one. + +We can see that real-world documents of the same type, such as business cards, +tend to be very similar in *semantics* -- the sort of information they carry, +but can vary hugely in *syntax*, or how that information is structured. As human +beings, we're naturally comfortable dealing with this kind of variation. + +While a traditional relational database requires you to model your data +*up front*, CouchDB's schema-free design unburdens you with a powerful way to +aggregate your data *after the fact*, just like we do with real-world +documents. We'll look in depth at how to design applications with this +underlying storage paradigm. + + +Building Blocks for Larger Systems +================================== + +CouchDB is a storage system useful on its own. You can build many applications +with the tools CouchDB gives you. But CouchDB is designed with a bigger picture +in mind. Its components can be used as building blocks that solve storage +problems in slightly different ways for larger and more complex systems. + +Whether you need a system that's crazy fast but isn't too concerned with +reliability (think logging), or one that guarantees storage in two or more +physically separated locations for reliability, but you're willing to take a +performance hit, CouchDB lets you build these systems. + +There are a multitude of knobs you could turn to make a system work better in +one area, but you'll affect another area when doing so. One example would be +the CAP theorem discussed in the next chapter. To give you an idea of other +things that affect storage systems, see :ref:`Figure 2 <intro/why-figure-02>` +and :ref:`Figure 3 <intro/why-figure-03>`. + +By reducing latency for a given system (and that is true not only for storage +systems), you affect concurrency and throughput capabilities. + + +.. _intro/why-figure-02: + +.. figure:: ../../images/intro-why-02.png + :align: center + :alt: Throughput, latency, or concurrency + + Figure 2. Throughput, latency, or concurrency + + +.. _intro/why-figure-03: + +.. figure:: ../../images/intro-why-03.png + :align: center + :alt: Scaling: read requests, write requests, or data + + Figure 3. Scaling: read requests, write requests, or data + + +When you want to scale out, there are three distinct issues to deal with: +scaling read requests, write requests, and data. Orthogonal to all three and +to the items shown in :ref:`Figure 2 <intro/why-figure-02>` and :ref:`Figure 3 +<intro/why-figure-03>` are many more attributes like reliability or simplicity. +You can draw many of these graphs that show how different features or attributes +pull into different directions and thus shape the system they describe. + +CouchDB is very flexible and gives you enough building blocks to create a +system shaped to suit your exact problem. That's not saying that CouchDB can +be bent to solve any problem -- CouchDB is no silver bullet -- but in the +area of data storage, it can get you a long way. + + +CouchDB Replication +=================== + +CouchDB replication is one of these building blocks. Its fundamental function +is to synchronize two or more CouchDB databases. This may sound simple, +but the simplicity is key to allowing replication to solve a number of +problems: reliably synchronize databases between multiple machines for +redundant data storage; distribute data to a cluster of CouchDB instances +that share a subset of the total number of requests that hit the cluster +(load balancing); and distribute data between physically distant locations, +such as one office in New York and another in Tokyo. + +CouchDB replication uses the same REST API all clients use. HTTP is +ubiquitous and well understood. Replication works incrementally; that is, +if during replication anything goes wrong, like dropping your network +connection, it will pick up where it left off the next time it runs. It also +only transfers data that is needed to synchronize databases. + +A core assumption CouchDB makes is that things can go wrong, +like network connection troubles, and it is designed for graceful error +recovery instead of assuming all will be well. The replication system's +incremental design shows that best. The ideas behind “things that can go +wrong” are embodied in the `Fallacies of Distributed Computing`_: + +- The network is reliable. +- Latency is zero. +- Bandwidth is infinite. +- The network is secure. +- Topology doesn't change. +- There is one administrator. +- Transport cost is zero. +- The network is homogeneous. + +Existing tools often try to hide the fact that there is a network and that +any or all of the previous conditions don't exist for a particular system. +This usually results in fatal error scenarios when something finally goes +wrong. In contrast, CouchDB doesn't try to hide the network; it just handles +errors gracefully and lets you know when actions on your end are required. + +.. _Fallacies of Distributed Computing: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing + + +Local Data Is King +================== + +CouchDB takes quite a few lessons learned from the Web, +but there is one thing that could be improved about the Web: latency. +Whenever you have to wait for an application to respond or a website to +render, you almost always wait for a network connection that isn't as fast as +you want it at that point. Waiting a few seconds instead of milliseconds +greatly affects user experience and thus user satisfaction. + +What do you do when you are offline? This happens all the time -- your DSL or +cable provider has issues, or your iPhone, G1, or Blackberry has no bars, +and no connectivity means no way to get to your data. + +CouchDB can solve this scenario as well, and this is where scaling is +important again. This time it is scaling down. Imagine CouchDB installed on +phones and other mobile devices that can synchronize data with centrally +hosted CouchDBs when they are on a network. The synchronization is not bound +by user interface constraints like subsecond response times. It is easier to +tune for high bandwidth and higher latency than for low bandwidth and very +low latency. Mobile applications can then use the local CouchDB to fetch +data, and since no remote networking is required for that, +latency is low by default. + +Can you really use CouchDB on a phone? Erlang, CouchDB's implementation +language has been designed to run on embedded devices magnitudes smaller and +less powerful than today's phones. + + +Wrapping Up +=========== + +The next chapter further explores the distributed nature of CouchDB. We +should have given you enough bites to whet your interest. Let's go! |
