Merge pull request #1853 from opscode/mcquin/Issues-1849

Autodetect encrypted data bag items in data_bag_item dsl method.

8 files changed, 240 insertions, 31 deletions

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 87d31cf3ba..40ab884c01 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -102,6 +102,7 @@
 * Fix a bug in reporting not to post negative duration values.
 * Add password setting support for Mac 10.7, 10.8 and 10.9 to the dscl user provider.
 * ChefSpec can find freebsd_package resource correctly when a package resource is declared on Freebsd.
+* Autodetect/decrypt encrypted data bag items with data_bag_item dsl method. (Issue 1837, Issue 1849)
 
 ## Last Release: 11.14.2
 
@@ -232,4 +233,3 @@
 * Added DelayedEvaluator support in LWRP using the `lazy {}` key
 * Fixed a bug where nested resources that inherited from Resource::LWRPBase
   would not share the same actions/default_action as their parent
-
diff --git a/DOC_CHANGES.md b/DOC_CHANGES.md
index 7ce531b3b5..1ef0a7830c 100644
--- a/DOC_CHANGES.md
+++ b/DOC_CHANGES.md
@@ -14,9 +14,9 @@ Description of the required change.
 + *  Exact matches take precedence no matter what, and should never throw exceptions.
 + *  Matching multiple constraints raises a <code>RuntimeError</code>.
 + *  The following constraints are allowed: <code><,<=,>,>=,~></code>.
-+ 
++
 + The following is an example of using the method with constraints:
-+ 
++
 + ```ruby
 + value_for_platform(
 +   "os1" => {
@@ -71,3 +71,8 @@ action :configure_startup - sets the startup type on the resource to the value o
 attribute startup_type - the value as a symbol that the startup type should be set to on the service, valid options :automatic, :manual, :disabled
 
 Note that the service resource will also continue to set the startup type to automatic or disabled, respectively, when the enabled or disabled actions are used.
+
+### Fetch encrypted data bag items with dsl method
+DSL method `data_bag_item` now takes an optional String parameter `secret`, which is used to interact with encrypted data bag items.
+If the data bag item being fetched is encrypted and no `secret` is provided, Chef looks for a secret at `Chef::Config[:encrypted_data_bag_secret]`.
+If `secret` is provided, but the data bag item is not encrypted, then a regular data bag item is returned (no decryption is attempted).
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 9fb4eb1f6d..0e207f315b 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -160,6 +160,11 @@ directory will also be inherited correctly.
 
 Informational messages from knife are now sent to stderr, allowing you to pipe the output of knife to other commands without having to filter these messages out.
 
+## Enhance `data_bag_item` to interact with encrypted data bag items
+
+The `data_bag_item` dsl method can be used to load encrypted data bag items when an additional `secret` String parameter is included.
+If no `secret` is provided but the data bag item is encrypted, `Chef::Config[:encrypted_data_bag_secret]` will be checked.
+
 # Internal API Changes in this Release
 
 These changes do not impact any cookbook code, but may impact tools that
diff --git a/lib/chef/dsl/data_query.rb b/lib/chef/dsl/data_query.rb
index 65e7b185a7..3dafbca6bf 100644
--- a/lib/chef/dsl/data_query.rb
+++ b/lib/chef/dsl/data_query.rb
@@ -53,14 +53,60 @@ class Chef
         raise
       end
 
-      def data_bag_item(bag, item)
+      def data_bag_item(bag, item, secret=nil)
         DataBag.validate_name!(bag.to_s)
         DataBagItem.validate_id!(item)
-        DataBagItem.load(bag, item)
+
+        item = DataBagItem.load(bag, item)
+        if encrypted?(item.raw_data)
+          Log.debug("Data bag item looks encrypted: #{bag.inspect} #{item.inspect}")
+
+          # Try to load the data bag item secret, if secret is not provided.
+          # Chef::EncryptedDataBagItem.load_secret may throw a variety of errors.
+          begin
+            secret ||= EncryptedDataBagItem.load_secret
+            item = EncryptedDataBagItem.new(item.raw_data, secret)
+          rescue Exception
+            Log.error("Failed to load secret for encrypted data bag item: #{bag.inspect} #{item.inspect}")
+            raise
+          end
+        end
+        
+        item
       rescue Exception
         Log.error("Failed to load data bag item: #{bag.inspect} #{item.inspect}")
         raise
       end
+
+      private
+
+      # Tries to autodetect if the item's raw hash appears to be encrypted.
+      def encrypted?(raw_data)
+        data = raw_data.reject { |k, _| k == "id" } # Remove the "id" key.
+        # Assume hashes containing only the "id" key are not encrypted.
+        # Otherwise, remove the keys that don't appear to be encrypted and compare
+        # the result with the hash. If some entry has been removed, then some entry
+        # doesn't appear to be encrypted and we assume the entire hash is not encrypted.
+        data.empty? ? false : data.reject { |_, v| !looks_like_encrypted?(v) } == data
+      end
+
+      # Checks if data looks like it has been encrypted by
+      # Chef::EncryptedDataBagItem::Encryptor::VersionXEncryptor. Returns
+      # true only when there is an exact match between the VersionXEncryptor
+      # keys and the hash's keys.
+      def looks_like_encrypted?(data)
+        return false unless data.is_a?(Hash) && data.has_key?("version")
+        case data["version"]
+        when 1
+          Chef::EncryptedDataBagItem::Encryptor::Version1Encryptor.encryptor_keys.sort == data.keys.sort
+        when 2
+          Chef::EncryptedDataBagItem::Encryptor::Version2Encryptor.encryptor_keys.sort == data.keys.sort
+        when 3
+          Chef::EncryptedDataBagItem::Encryptor::Version3Encryptor.encryptor_keys.sort == data.keys.sort
+        else
+          false # version means something else... assume not encrypted.
+        end
+      end
     end
   end
 end
@@ -68,4 +114,3 @@ end
 # **DEPRECATED**
 # This used to be part of chef/mixin/language. Load the file to activate the deprecation code.
 require 'chef/mixin/language'
-
diff --git a/lib/chef/encrypted_data_bag_item.rb b/lib/chef/encrypted_data_bag_item.rb
index f722b5dc38..120eb2a4ae 100644
--- a/lib/chef/encrypted_data_bag_item.rb
+++ b/lib/chef/encrypted_data_bag_item.rb
@@ -128,7 +128,7 @@ class Chef::EncryptedDataBagItem
   def self.load_secret(path=nil)
     path ||= Chef::Config[:encrypted_data_bag_secret]
     if !path
-      raise ArgumentError, "No secret specified to load_secret and no secret found at #{Chef::Config.platform_specific_path('/etc/chef/encrypted_data_bag_secret')}"
+      raise ArgumentError, "No secret specified and no secret found at #{Chef::Config.platform_specific_path('/etc/chef/encrypted_data_bag_secret')}"
     end
     secret = case path
              when /^\w+:\/\//
diff --git a/lib/chef/encrypted_data_bag_item/encryptor.rb b/lib/chef/encrypted_data_bag_item/encryptor.rb
index 6bf340869a..034413c1bd 100644
--- a/lib/chef/encrypted_data_bag_item/encryptor.rb
+++ b/lib/chef/encrypted_data_bag_item/encryptor.rb
@@ -125,6 +125,10 @@ class Chef::EncryptedDataBagItem
       def serialized_data
         FFI_Yajl::Encoder.encode(:json_wrapper => plaintext_data)
       end
+
+      def self.encryptor_keys
+        %w( encrypted_data iv version cipher )
+      end
     end
 
     class Version2Encryptor < Version1Encryptor
@@ -149,6 +153,10 @@ class Chef::EncryptedDataBagItem
           Base64.encode64(raw_hmac)
         end
       end
+
+      def self.encryptor_keys
+        super + %w( hmac )
+      end
     end
 
     class Version3Encryptor < Version1Encryptor
@@ -207,6 +215,10 @@ class Chef::EncryptedDataBagItem
         end
       end
 
+      def self.encryptor_keys
+        super + %w( auth_tag )
+      end
+
     end
 
   end
diff --git a/spec/unit/dsl/data_query_spec.rb b/spec/unit/dsl/data_query_spec.rb
index e31c0725d6..8a985437b7 100644
--- a/spec/unit/dsl/data_query_spec.rb
+++ b/spec/unit/dsl/data_query_spec.rb
@@ -24,43 +24,185 @@ class DataQueryDSLTester
 end
 
 describe Chef::DSL::DataQuery do
-  before(:each) do
-    @language = DataQueryDSLTester.new
-    @node = Hash.new
-    @language.stub(:node).and_return(@node)
+  let(:node) { Hash.new }
+
+  let(:language) do
+    language = DataQueryDSLTester.new
+    language.stub(:node).and_return(@node)
+    language
   end
 
-  describe "when loading data bags and items" do
+  describe "::data_bag" do
     it "lists the items in a data bag" do
-      Chef::DataBag.should_receive(:load).with("bag_name").and_return("item_1" => "http://url_for/item_1", "item_2" => "http://url_for/item_2")
-      @language.data_bag("bag_name").sort.should == %w[item_1 item_2]
+      allow(Chef::DataBag).to receive(:load)
+      .with("bag_name")
+      .and_return("item_1" => "http://url_for/item_1", "item_2" => "http://url_for/item_2")
+      expect( language.data_bag("bag_name").sort ).to eql %w(item_1 item_2)
     end
+  end
 
-    it "validates the name of the data bag you're trying to load" do
-      lambda {@language.data_bag("!# %^&& ")}.should raise_error(Chef::Exceptions::InvalidDataBagName)
+  shared_examples_for "a data bag item" do
+    it "validates the name of the data bag you're trying to load an item from" do
+      expect{ language.send(method_name, " %%^& ", "item_name") }.to raise_error(Chef::Exceptions::InvalidDataBagName)
     end
 
-    it "fetches a data bag item" do
-      @item = Chef::DataBagItem.new
-      @item.data_bag("bag_name")
-      @item.raw_data = {"id" => "item_name", "FUU" => "FUU"}
-      Chef::DataBagItem.should_receive(:load).with("bag_name", "item_name").and_return(@item)
-      @language.data_bag_item("bag_name", "item_name").should == @item
+    it "validates the id of the data bag item you're trying to load" do
+      expect{ language.send(method_name, "bag_name", " 987 (*&()") }.to raise_error(Chef::Exceptions::InvalidDataBagItemID)
     end
 
-    it "validates the name of the data bag you're trying to load an item from" do
-      lambda {@language.data_bag_item(" %%^& ", "item_name")}.should raise_error(Chef::Exceptions::InvalidDataBagName)
+    it "validates that the id of the data bag item is not nil" do
+      expect{ language.send(method_name, "bag_name", nil) }.to raise_error(Chef::Exceptions::InvalidDataBagItemID)
     end
+  end
 
-    it "validates the id of the data bag item you're trying to load" do
-      lambda {@language.data_bag_item("bag_name", " 987 (*&()")}.should raise_error(Chef::Exceptions::InvalidDataBagItemID)
+  describe "::data_bag_item" do
+    let(:bag_name) { "bag_name" }
+
+    let(:item_name) { "item_name" }
+
+    let(:raw_data) {{
+      "id" => item_name,
+      "greeting" => "hello",
+      "nested" => {
+        "a1" => [1, 2, 3],
+        "a2" => { "b1" => true }
+      }
+    }}
+
+    let(:item) do
+      item = Chef::DataBagItem.new
+      item.data_bag(bag_name)
+      item.raw_data = raw_data
+      item
     end
 
-    it "validates that the id of the data bag item is not nil" do
-      lambda {@language.data_bag_item("bag_name", nil)}.should raise_error(Chef::Exceptions::InvalidDataBagItemID)
+    it "fetches a data bag item" do
+      allow( Chef::DataBagItem ).to receive(:load).with(bag_name, item_name).and_return(item)
+      expect( language.data_bag_item(bag_name, item_name) ).to eql item
     end
 
-  end
+    include_examples "a data bag item" do
+      let(:method_name) { :data_bag_item }
+    end
 
-end
+    context "when the item is encrypted" do
+      let(:default_secret) { "abc123SECRET" }
+
+      let(:encoded_data) { Chef::EncryptedDataBagItem.encrypt_data_bag_item(raw_data, default_secret) }
+
+      let(:item) do
+        item = Chef::DataBagItem.new
+        item.data_bag(bag_name)
+        item.raw_data = encoded_data
+        item
+      end
+
+      before do
+        allow( Chef::DataBagItem ).to receive(:load).with(bag_name, item_name).and_return(item)
+      end
 
+      shared_examples_for "encryption detected" do
+        let(:encoded_data) do
+          Chef::Config[:data_bag_encrypt_version] = version
+          Chef::EncryptedDataBagItem.encrypt_data_bag_item(raw_data, default_secret)
+        end
+
+        before do
+          allow( Chef::EncryptedDataBagItem ).to receive(:load_secret).and_return(default_secret)
+        end
+
+        it "detects encrypted data bag" do
+          expect( encryptor ).to receive(:encryptor_keys).at_least(:once).and_call_original
+          expect( Chef::Log ).to receive(:debug).with(/Data bag item looks encrypted/)
+          language.data_bag_item(bag_name, item_name)
+        end
+      end
+
+      context "when encryption version is 1" do
+        include_examples "encryption detected" do
+          let(:version) { 1 }
+          let(:encryptor) { Chef::EncryptedDataBagItem::Encryptor::Version1Encryptor }
+        end
+      end
+
+      context "when encryption version is 2" do
+        include_examples "encryption detected" do
+          let(:version) { 2 }
+          let(:encryptor) { Chef::EncryptedDataBagItem::Encryptor::Version2Encryptor }
+        end
+      end
+
+      context "when encryption version is 3", :ruby_20_only do
+        include_examples "encryption detected" do
+          let(:version) { 3 }
+          let(:encryptor) { Chef::EncryptedDataBagItem::Encryptor::Version3Encryptor }
+        end
+      end
+
+      shared_examples_for "an encrypted data bag item" do
+        it "returns an encrypted data bag item" do
+          expect( language.data_bag_item(bag_name, item_name, secret) ).to be_a_kind_of(Chef::EncryptedDataBagItem)
+        end
+
+        it "decrypts the contents of the data bag item" do
+          expect( language.data_bag_item(bag_name, item_name, secret).to_hash ).to eql raw_data
+        end
+      end
+
+      context "when a secret is supplied" do
+        include_examples "an encrypted data bag item" do
+          let(:secret) { default_secret }
+        end
+      end
+
+      context "when a secret is not supplied" do
+        before do
+          allow( Chef::Config ).to receive(:[]).and_call_original
+          expect( Chef::Config ).to receive(:[]).with(:encrypted_data_bag_secret).and_return(path)
+          expect( Chef::EncryptedDataBagItem ).to receive(:load_secret).and_call_original
+        end
+
+        context "when a secret is located at Chef::Config[:encrypted_data_bag_secret]" do
+          let(:path) { "/tmp/my_secret" }
+
+          before do
+            expect( File ).to receive(:exist?).with(path).and_return(true)
+            expect( IO ).to receive(:read).with(path).and_return(default_secret)
+          end
+
+          include_examples "an encrypted data bag item" do
+            let(:secret) { nil }
+          end
+        end
+
+        shared_examples_for "no secret file" do
+          it "should fail to load the data bag item" do
+            expect( Chef::Log ).to receive(:error).with(/Failed to load secret for encrypted data bag item/)
+            expect( Chef::Log ).to receive(:error).with(/Failed to load data bag item/)
+            expect{ language.data_bag_item(bag_name, item_name) }.to raise_error(error_type, error_message)
+          end
+        end
+
+        context "when Chef::Config[:encrypted_data_bag_secret] is not configured" do
+          include_examples "no secret file" do
+            let(:path) { nil }
+            let(:error_type) { ArgumentError }
+            let(:error_message) { /No secret specified and no secret found/ }
+          end
+        end
+
+        context "when Chef::Config[:encrypted_data_bag_secret] does not exist" do
+          include_examples "no secret file" do
+            before do
+              expect( File ).to receive(:exist?).with(path).and_return(false)
+            end
+
+            let(:path) { "/tmp/my_secret" }
+            let(:error_type) { Errno::ENOENT }
+            let(:error_message) { /file not found/ }
+          end
+        end
+      end
+    end
+  end
+end
diff --git a/spec/unit/encrypted_data_bag_item_spec.rb b/spec/unit/encrypted_data_bag_item_spec.rb
index 24ceb452ef..5ee245b9bc 100644
--- a/spec/unit/encrypted_data_bag_item_spec.rb
+++ b/spec/unit/encrypted_data_bag_item_spec.rb
@@ -451,7 +451,7 @@ describe Chef::EncryptedDataBagItem do
       end
 
       it "load_secret(nil) emits a reasonable error message" do
-        lambda { Chef::EncryptedDataBagItem.load_secret(nil) }.should raise_error(ArgumentError, "No secret specified to load_secret and no secret found at #{Chef::Config.platform_specific_path('/etc/chef/encrypted_data_bag_secret')}")
+        lambda { Chef::EncryptedDataBagItem.load_secret(nil) }.should raise_error(ArgumentError, /No secret specified and no secret found at #{Chef::Config[:encrypted_data_bag_secret]}/)
       end
     end