[WIP] server-to-server cookbook transferssd/cheffs_server_to_server

This code allows a user to use ChefFS to transfer a cookbook from one
Chef server to another.  Files are streamed to a tempfile in transit,
but a full cookbook repo on disk is not required.

The upload_checksum functions is basically a copy-paste from
cookbook_uploader.rb

I would love to get some feedback on how we could improve this.
Some items that feel gross to me:

1) It would be nice to remove some of the duplication around
upload_checksum.  Would that be appropriate to pull out into a module
that could be used in both?  Or perhaps a model object for individual
cookbook files with a send_to(url) method?

2) Switching on the class of other rather than using some sort of
inheretence.

3) Using a temporary file rather than just streaming the downloading
directly into a streaming upload.

2 files changed, 39 insertions, 4 deletions

diff --git a/lib/chef/chef_fs/file_system/cookbooks_dir.rb b/lib/chef/chef_fs/file_system/cookbooks_dir.rb
index a58bfdd1f2..fc44259410 100644
--- a/lib/chef/chef_fs/file_system/cookbooks_dir.rb
+++ b/lib/chef/chef_fs/file_system/cookbooks_dir.rb
@@ -16,6 +16,7 @@
 # limitations under the License.
 #
 
+require 'chef/chef_fs/parallelizer'
 require 'chef/chef_fs/file_system/rest_list_dir'
 require 'chef/chef_fs/file_system/cookbook_dir'
 require 'chef/chef_fs/file_system/operation_failed_error'
@@ -71,7 +72,12 @@ class Chef
         end
 
         def upload_cookbook_from(other, options = {})
-          Chef::Config[:versioned_cookbooks] ? upload_versioned_cookbook(other, options) : upload_unversioned_cookbook(other, options)
+          case other
+          when Chef::ChefFS::FileSystem::CookbookDir
+            streaming_upload_from(other, options)
+          when Chef::ChefFS::FileSystem::ChefRepositoryFileSystemCookbookDir
+            Chef::Config[:versioned_cookbooks] ? upload_versioned_cookbook(other, options) : upload_unversioned_cookbook(other, options)
+          end
         rescue Timeout::Error => e
           raise Chef::ChefFS::FileSystem::OperationFailedError.new(:write, self, e), "Timeout writing: #{e}"
         rescue Net::HTTPServerException => e
@@ -85,6 +91,36 @@ class Chef
           raise Chef::ChefFS::FileSystem::CookbookFrozenError.new(:write, self, e), "Cookbook #{other.name} is frozen"
         end
 
+        def streaming_upload_from(other, options = {})
+          new_sandbox = root.chef_rest.post("sandboxes", {:checksums => other.chef_object.checksums})
+          Chef::ChefFS::Parallelizer.parallelize(new_sandbox['checksums']) do |checksum, info|
+            if info['needs_upload']
+              url = other.chef_object.url_for_checksum(checksum)
+              tmpfile = other.root.chef_rest.get_rest(url, true)
+              upload_checksum(checksum, info['url'], tmpfile)
+              ::File.unlink(tmpfile)
+            end
+          end.to_a
+          root.chef_rest.put_rest(new_sandbox['uri'], {:is_completed => true})
+          root.chef_rest.put_rest(other.chef_object.save_url, other.chef_object)
+        end
+
+        def upload_checksum(checksum, url, file_path)
+          checksum64 = Base64.encode64([checksum].pack("H*")).strip
+          timestamp = Time.now.utc.iso8601
+          file_contents = File.open(file_path) {|f| f.read}
+          headers = { 'content-type' => 'application/x-binary', 'content-md5' => checksum64, "accept" => 'application/json' }
+          if root.chef_rest.signing_key
+            sign_obj = Mixlib::Authentication::SignedHeaderAuth.signing_object(:http_method => :put,
+                                                                               :path        => URI.parse(url).path,
+                                                                               :body        => file_contents,
+                                                                               :timestamp   => timestamp,
+                                                                               :user_id     => root.chef_rest.client_name)
+            headers.merge!(sign_obj.sign(OpenSSL::PKey::RSA.new(root.chef_rest.signing_key)))
+          end
+          Chef::HTTP::Simple.new(url, :headers=>headers).put(url, file_contents)
+        end
+
         # Knife currently does not understand versioned cookbooks
         # Cookbook Version uploader also requires a lot of refactoring
         # to make this work. So instead, we make a temporary cookbook
diff --git a/lib/chef/cookbook_version.rb b/lib/chef/cookbook_version.rb
index 1f0311e33e..2727a51fe5 100644
--- a/lib/chef/cookbook_version.rb
+++ b/lib/chef/cookbook_version.rb
@@ -208,11 +208,10 @@ class Chef
     end
 
     def url_for_checksum(checksum)
-      Chef::CookbookVersion::COOKBOOK_SEGMENTS.each do |segment|
+      COOKBOOK_SEGMENTS.each do |segment|
         f = manifest[segment].find {|c| c["checksum"] == checksum }
-        break if f
+        return f["url"] if f
       end
-      f["url"]
     end
 
     def recipe_filenames=(*filenames)