diff options
author | Claire McQuin <mcquin@users.noreply.github.com> | 2014-09-05 10:06:29 -0700 |
---|---|---|
committer | Claire McQuin <mcquin@users.noreply.github.com> | 2014-09-05 10:06:29 -0700 |
commit | 27b05b2396459f3d84f0ebd924f8adc44a906cd1 (patch) | |
tree | 9a9211ceb41318d9b0bda64c9f7d7dd0b1c13087 | |
parent | 7e93d3f41d516ec9d2f1f84d33efa1ba1a9b6164 (diff) | |
parent | 300cbe8a816d2591e5b14bd290dd81cacfe850d6 (diff) | |
download | chef-27b05b2396459f3d84f0ebd924f8adc44a906cd1.tar.gz |
Merge pull request #1555 from opscode/shain/search_filter
Result filtering on search (also known as Partial Search)
-rw-r--r-- | CHANGELOG.md | 1 | ||||
-rw-r--r-- | DOC_CHANGES.md | 56 | ||||
-rw-r--r-- | RELEASE_NOTES.md | 6 | ||||
-rw-r--r-- | lib/chef/exceptions.rb | 2 | ||||
-rw-r--r-- | lib/chef/knife/search.rb | 68 | ||||
-rw-r--r-- | lib/chef/search/query.rb | 136 | ||||
-rw-r--r-- | spec/unit/search/query_spec.rb | 236 |
7 files changed, 439 insertions, 66 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index c160fae749..471a00500f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -117,6 +117,7 @@ * Deprecate --distro / --template_file options in favor of --boostrap-template * Add `:node_ssl_verify_mode` & `:node_verify_api_cert` options to bootstrap to be able to configure these settings on the bootstrapped node. +* Add partial_search dsl method to Chef::Search::Query, add result filtering to search. ## Last Release: 11.14.2 diff --git a/DOC_CHANGES.md b/DOC_CHANGES.md index c08ab0097a..8b70f40c5a 100644 --- a/DOC_CHANGES.md +++ b/DOC_CHANGES.md @@ -89,3 +89,59 @@ Note that the service resource will also continue to set the startup type to aut DSL method `data_bag_item` now takes an optional String parameter `secret`, which is used to interact with encrypted data bag items. If the data bag item being fetched is encrypted and no `secret` is provided, Chef looks for a secret at `Chef::Config[:encrypted_data_bag_secret]`. If `secret` is provided, but the data bag item is not encrypted, then a regular data bag item is returned (no decryption is attempted). + +### Enhanced search functionality: result filtering +#### Use in recipes +`Chef::Search::Query#search` can take an optional `:filter_result` argument which returns search data in the form of the Hash specified. Suppose your data looks like +```json +{"languages": { + "c": { + "gcc": { + "version": "4.6.3", + "description": "gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) " + } + }, + "ruby": { + "platform": "x86_64-linux", + "version": "1.9.3", + "release_date": "2013-11-22" + }, + "perl": { + "version": "5.14.2", + "archname": "x86_64-linux-gnu-thread-multi" + }, + "python": { + "version": "2.7.3", + "builddate": "Feb 27 2014, 19:58:35" + } +}} +``` +for a node running Ubuntu named `node01`, and you want to get back only information on which versions of c and ruby you have. In a recipe you would write +```ruby +search(:node, "platform:ubuntu", :filter_result => {"c_version" => ["languages", "c", "gcc", "version"], + "ruby_version" => ["languages", "ruby", "version"]}) +``` +and receive +```ruby +[ + {"url" => "https://api.opscode.com/organization/YOUR_ORG/nodes/node01", + "data" => {"c_version" => "4.6.3", "ruby_version" => "1.9.3"}, + # snip other Ubuntu nodes +] +``` +If instead you wanted all the languages data (remember, `"languages"` is only one tiny piece of information the Chef Server stores about your node), you would have `:filter_result => {"languages" => ["laguages"]}` in your search query. + +For backwards compatibility, a `partial_search` method has been added to `Chef::Search::Query` which can be used in the same way as the `partial_search` method from the [partial_search cookbook](https://supermarket.getchef.com/cookbooks/partial_search). Note that this method has been deprecated and will be removed in future versions of Chef. + +#### Use in knife +Search results can likewise be filtered by adding the `--filter-result` (or `-f`) option. Considering the node data above, you can use `knife search` with filtering to extract the c and ruby versions on your Ubuntu platforms: +```bash +$ knife search node "platform:ubuntu" --filter-result "c_version:languages.c.gcc.version, ruby_version:languages.ruby.version" +1 items found + +: + c_version: 4.6.3 + ruby_version: 1.9.3 + +$ +``` diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 30c5d0893e..f5a74d6a43 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -237,3 +237,9 @@ work properly if the remote server implemented only the Chef 10 API. ## CookbookSiteStreamingUploader now uses ssl_verify_mode config option The CookbookSiteStreamingUploader now obeys the setting of ssl_verify_mode in the client config. Was previously ignoring the config setting and always set to VERIFY_NONE. + +## Result filtering on `search` API. +`search` can take an optional `:filter_result`, which returns search data in the form specified +by the given Hash. This works analogously to the partial_search method from the [partial_search cookbook](https://supermarket.getchef.com/cookbooks/partial_search), +with `:filter_result` replacing `:keys`. You can also filter `knife search` results by supplying the `--filter-result` +or `-f` option and a comma-separated string representation of the filter hash. diff --git a/lib/chef/exceptions.rb b/lib/chef/exceptions.rb index f6db5dbe56..23e223f204 100644 --- a/lib/chef/exceptions.rb +++ b/lib/chef/exceptions.rb @@ -347,5 +347,7 @@ class Chef class EncodeError < RuntimeError; end class ParseError < RuntimeError; end end + + class InvalidSearchQuery < ArgumentError; end end end diff --git a/lib/chef/knife/search.rb b/lib/chef/knife/search.rb index bc020c0445..34d12168b6 100644 --- a/lib/chef/knife/search.rb +++ b/lib/chef/knife/search.rb @@ -71,6 +71,11 @@ class Chef :long => "--query QUERY", :description => "The search query; useful to protect queries starting with -" + option :filter_result, + :short => "-f FILTER", + :long => "--filter-result FILTER", + :description => "Only bring back specific attributes of the matching objects; for example: \"ServerName=name, Kernel=kernel.version\"" + def run read_cli_args fuzzify_query @@ -79,7 +84,6 @@ class Chef ui.use_presenter Knife::Core::NodePresenter end - q = Chef::Search::Query.new escaped_query = URI.escape(@query, Regexp.new("[^#{URI::PATTERN::UNRESERVED}]")) @@ -87,14 +91,26 @@ class Chef result_items = [] result_count = 0 - rows = config[:rows] - start = config[:start] + search_args = Hash.new + search_args[:sort] = config[:sort] + search_args[:start] = config[:start] + search_args[:rows] = config[:rows] + if config[:filter_result] + search_args[:filter_result] = create_result_filter(config[:filter_result]) + elsif (not ui.config[:attribute].nil?) && (not ui.config[:attribute].empty?) + search_args[:filter_result] = create_result_filter_from_attributes(ui.config[:attribute]) + end + begin - q.search(@type, escaped_query, config[:sort], start, rows) do |item| - formatted_item = format_for_display(item) - # if formatted_item.respond_to?(:has_key?) && !formatted_item.has_key?('id') - # formatted_item['id'] = item.has_key?('id') ? item['id'] : item.name - # end + q.search(@type, escaped_query, search_args) do |item| + formatted_item = Hash.new + if item.is_a?(Hash) + # doing a little magic here to set the correct name + formatted_item[item["data"]["__display_name"]] = item["data"] + formatted_item[item["data"]["__display_name"]].delete("__display_name") + else + formatted_item = format_for_display(item) + end result_items << formatted_item result_count += 1 end @@ -149,10 +165,38 @@ class Chef end end + # This method turns a set of key value pairs in a string into the appropriate data structure that the + # chef-server search api is expecting. + # expected input is in the form of: + # -f "return_var1=path.to.attribute, return_var2=shorter.path" + # + # a more concrete example might be: + # -f "env=chef_environment, ruby_platform=languages.ruby.platform" + # + # The end result is a hash where the key is a symbol in the hash (the return variable) + # and the path is an array with the path elements as strings (in order) + # See lib/chef/search/query.rb for more examples of this. + def create_result_filter(filter_string) + final_filter = Hash.new + filter_string.gsub!(" ", "") + filters = filter_string.split(",") + filters.each do |f| + return_id, attr_path = f.split("=") + final_filter[return_id.to_sym] = attr_path.split(".") + end + return final_filter + end + + def create_result_filter_from_attributes(filter_array) + final_filter = Hash.new + filter_array.each do |f| + final_filter[f] = f.split(".") + end + # adding magic filter so we can actually pull the name as before + final_filter["__display_name"] = [ "name" ] + return final_filter + end + end end end - - - - diff --git a/lib/chef/search/query.rb b/lib/chef/search/query.rb index 4869ec1484..cc43efe1b1 100644 --- a/lib/chef/search/query.rb +++ b/lib/chef/search/query.rb @@ -23,6 +23,7 @@ require 'chef/node' require 'chef/role' require 'chef/data_bag' require 'chef/data_bag_item' +require 'chef/exceptions' class Chef class Search @@ -34,17 +35,112 @@ class Chef @rest = Chef::REST.new(url ||Chef::Config[:chef_server_url]) end - # Search Solr for objects of a given type, for a given query. If you give - # it a block, it will handle the paging for you dynamically. - def search(type, query="*:*", sort='X_CHEF_id_CHEF_X asc', start=0, rows=1000, &block) - raise ArgumentError, "Type must be a string or a symbol!" unless (type.kind_of?(String) || type.kind_of?(Symbol)) - response = @rest.get_rest("search/#{type}?q=#{escape(query)}&sort=#{escape(sort)}&start=#{escape(start)}&rows=#{escape(rows)}") - if block - response["rows"].each { |o| block.call(o) unless o.nil?} + # This search is only kept for backwards compatibility, since the results of the + # new filtered search method will be in a slightly different format + def partial_search(type, query='*:*', *args, &block) + Chef::Log.warn("DEPRECATED: The 'partial_search' api is deprecated, please use the search api with 'filter_result'") + # accept both types of args + if args.length == 1 && args[0].is_a?(Hash) + args_hash = args[0].dup + # partial_search implemented in the partial search cookbook uses the + # arg hash :keys instead of :filter_result to filter returned data + args_hash[:filter_result] = args_hash[:keys] + else + args_hash = {} + args_hash[:sort] = args[0] if args.length >= 1 + args_hash[:start] = args[1] if args.length >= 2 + args_hash[:rows] = args[2] if args.length >= 3 + end + + unless block.nil? + raw_results = search(type,query,args_hash) + else + raw_results = search(type,query,args_hash,&block) + end + results = Array.new + raw_results[0].each do |r| + results << r["data"] + end + return results + end + + # + # New search input, designed to be backwards compatible with the old method signature + # 'type' and 'query' are the same as before, args now will accept either a Hash of + # search arguments with symbols as the keys (ie :sort, :start, :rows) and a :filter_result + # option. + # + # :filter_result should be in the format of another Hash with the structure of: + # { + # :returned_name1 => ["path", "to", "variable"], + # :returned_name2 => ["shorter", "path"] + # } + # a real world example might be something like: + # { + # :ip_address => ["ipaddress"], + # :ruby_version => ["languages", "ruby", "version"] + # } + # this will bring back 2 variables 'ip_address' and 'ruby_version' with whatever value was found + # an example of the returned json may be: + # {"ip_address":"127.0.0.1", "ruby_version": "1.9.3"} + # + def search(type, query='*:*', *args, &block) + validate_type(type) + validate_args(args) + + scrubbed_args = Hash.new + + # argify everything + if args[0].kind_of?(Hash) + scrubbed_args = args[0] + else + # This api will be deprecated in a future release + scrubbed_args = { :sort => args[0], :start => args[1], :rows => args[2] } + end + + # set defaults, if they haven't been set yet. + scrubbed_args[:sort] ||= 'X_CHEF_id_CHEF_X asc' + scrubbed_args[:start] ||= 0 + scrubbed_args[:rows] ||= 1000 + + do_search(type, query, scrubbed_args, &block) + end + + def list_indexes + @rest.get_rest("search") + end + + private + def validate_type(t) + unless t.kind_of?(String) || t.kind_of?(Symbol) + msg = "Invalid search object type #{t.inspect} (#{t.class}), must be a String or Symbol." + + "Useage: search(:node, QUERY, [OPTIONAL_ARGS])" + + " `knife search environment QUERY (options)`" + raise Chef::Exceptions::InvalidSearchQuery, msg + end + end + + def validate_args(a) + max_args = 3 + raise Chef::Exceptions::InvalidSearchQuery, "Too many arguments! (#{a.size} for <= #{max_args})" if a.size > max_args + end + + def escape(s) + s && URI.escape(s.to_s) + end + + # new search api that allows for a cleaner implementation of things like return filters + # (formerly known as 'partial search'). + # Also args should never be nil, but that is required for Ruby 1.8 compatibility + def do_search(type, query="*:*", args=nil, &block) + query_string = create_query_string(type, query, args) + response = call_rest_service(query_string, args) + unless block.nil? + response["rows"].each { |rowset| block.call(rowset) unless rowset.nil?} unless (response["start"] + response["rows"].length) >= response["total"] - nstart = response["start"] + rows - search(type, query, sort, nstart, rows, &block) + args[:start] = response["start"] + args[:rows] + do_search(type, query, args, &block) end true else @@ -52,14 +148,26 @@ class Chef end end - def list_indexes - @rest.get_rest("search") + # create the full rest url string + def create_query_string(type, query, args) + # create some default variables just so we don't break backwards compatibility + sort = args[:sort] + start = args[:start] + rows = args[:rows] + + return "search/#{type}?q=#{escape(query)}&sort=#{escape(sort)}&start=#{escape(start)}&rows=#{escape(rows)}" end - private - def escape(s) - s && URI.escape(s.to_s) + def call_rest_service(query_string, args) + if args.key?(:filter_result) + response = @rest.post_rest(query_string, args[:filter_result]) + response_rows = response['rows'].map { |row| row['data'] } + else + response = @rest.get_rest(query_string) + response_rows = response['rows'] end + return response + end end end end diff --git a/spec/unit/search/query_spec.rb b/spec/unit/search/query_spec.rb index 7463e3bb3c..c7388a6234 100644 --- a/spec/unit/search/query_spec.rb +++ b/spec/unit/search/query_spec.rb @@ -20,80 +20,236 @@ require 'spec_helper' require 'chef/search/query' describe Chef::Search::Query do - before(:each) do - @rest = double("Chef::REST") - Chef::REST.stub(:new).and_return(@rest) - @query = Chef::Search::Query.new - end + let(:rest) { double("Chef::REST") } + let(:query) { Chef::Search::Query.new } - describe "search" do - before(:each) do - @response = { + shared_context "filtered search" do + let(:query_string) { "search/node?q=platform:rhel&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000" } + let(:server_url) { "https://api.opscode.com/organizations/opscode/nodes" } + let(:args) { { filter_key => filter_hash } } + let(:filter_hash) { + { + 'env' => [ 'chef_environment' ], + 'ruby_plat' => [ 'languages', 'ruby', 'platform' ] + } + } + let(:response) { + { "rows" => [ - { "id" => "for you" }, - { "id" => "hip hop" }, - { "id" => "thought was down by law for you" }, - { "id" => "kept it hard core for you" }, + { "url" => "#{server_url}/my-name-is-node", + "data" => { + "env" => "elysium", + "ruby_plat" => "nudibranch" + } + }, + { "url" => "#{server_url}/my-name-is-jonas", + "data" => { + "env" => "hades", + "ruby_plat" => "i386-mingw32" + } + }, + { "url" => "#{server_url}/my-name-is-flipper", + "data" => { + "env" => "elysium", + "ruby_plat" => "centos" + } + }, + { "url" => "#{server_url}/my-name-is-butters", + "data" => { + "env" => "moon", + "ruby_plat" => "solaris2", + } + } ], "start" => 0, "total" => 4 } - @rest.stub(:get_rest).and_return(@response) - end + } + end + + before(:each) do + Chef::REST.stub(:new).and_return(rest) + rest.stub(:get_rest).and_return(response) + end + + describe "search" do + let(:response) { { + "rows" => [ + { "name" => "my-name-is-node", + "chef_environment" => "elysium", + "platform" => "rhel", + "automatic" => { + "languages" => { + "ruby" => { + "platform" => "nudibranch", + "version" => "1.9.3", + "target" => "ming-the-merciless" + } + } + } + }, + { "name" => "my-name-is-jonas", + "chef_environment" => "hades", + "platform" => "rhel", + "automatic" => { + "languages" => { + "ruby" => { + "platform" => "i386-mingw32", + "version" => "1.9.3", + "target" => "bilbo" + } + } + } + }, + { "name" => "my-name-is-flipper", + "chef_environment" => "elysium", + "platform" => "rhel", + "automatic" => { + "languages" => { + "ruby" => { + "platform" => "centos", + "version" => "2.0.0", + "target" => "uno" + } + } + } + }, + { "name" => "my-name-is-butters", + "chef_environment" => "moon", + "platform" => "rhel", + "automatic" => { + "languages" => { + "ruby" => { + "platform" => "solaris2", + "version" => "2.1.2", + "target" => "random" + } + } + } + }, + ], + "start" => 0, + "total" => 4 + } } it "should accept a type as the first argument" do - lambda { @query.search("foo") }.should_not raise_error - lambda { @query.search(:foo) }.should_not raise_error - lambda { @query.search(Hash.new) }.should raise_error(ArgumentError) + lambda { query.search("node") }.should_not raise_error + lambda { query.search(:node) }.should_not raise_error + lambda { query.search(Hash.new) }.should raise_error(Chef::Exceptions::InvalidSearchQuery, /(Hash)/) end it "should query for every object of a type by default" do - @rest.should_receive(:get_rest).with("search/foo?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000").and_return(@response) - @query = Chef::Search::Query.new - @query.search(:foo) + rest.should_receive(:get_rest).with("search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000").and_return(response) + query.search(:node) end it "should allow a custom query" do - @rest.should_receive(:get_rest).with("search/foo?q=gorilla:dundee&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000").and_return(@response) - @query = Chef::Search::Query.new - @query.search(:foo, "gorilla:dundee") + rest.should_receive(:get_rest).with("search/node?q=platform:rhel&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000").and_return(response) + query.search(:node, "platform:rhel") end it "should let you set a sort order" do - @rest.should_receive(:get_rest).with("search/foo?q=gorilla:dundee&sort=id%20desc&start=0&rows=1000").and_return(@response) - @query = Chef::Search::Query.new - @query.search(:foo, "gorilla:dundee", "id desc") + rest.should_receive(:get_rest).with("search/node?q=platform:rhel&sort=id%20desc&start=0&rows=1000").and_return(response) + query.search(:node, "platform:rhel", "id desc") end it "should let you set a starting object" do - @rest.should_receive(:get_rest).with("search/foo?q=gorilla:dundee&sort=id%20desc&start=2&rows=1000").and_return(@response) - @query = Chef::Search::Query.new - @query.search(:foo, "gorilla:dundee", "id desc", 2) + rest.should_receive(:get_rest).with("search/node?q=platform:rhel&sort=id%20desc&start=2&rows=1000").and_return(response) + query.search(:node, "platform:rhel", "id desc", 2) end it "should let you set how many rows to return" do - @rest.should_receive(:get_rest).with("search/foo?q=gorilla:dundee&sort=id%20desc&start=2&rows=40").and_return(@response) - @query = Chef::Search::Query.new - @query.search(:foo, "gorilla:dundee", "id desc", 2, 40) + rest.should_receive(:get_rest).with("search/node?q=platform:rhel&sort=id%20desc&start=2&rows=40").and_return(response) + query.search(:node, "platform:rhel", "id desc", 2, 40) + end + + it "should throw an exception if you pass to many options" do + lambda { query.search(:node, "platform:rhel", "id desc", 2, 40, "wrong") } + .should raise_error(Chef::Exceptions::InvalidSearchQuery, "Too many arguments! (4 for <= 3)") end it "should return the raw rows, start, and total if no block is passed" do - rows, start, total = @query.search(:foo) - rows.should equal(@response["rows"]) - start.should equal(@response["start"]) - total.should equal(@response["total"]) + rows, start, total = query.search(:node) + rows.should equal(response["rows"]) + start.should equal(response["start"]) + total.should equal(response["total"]) end it "should call a block for each object in the response" do @call_me = double("blocky") - @response["rows"].each { |r| @call_me.should_receive(:do).with(r) } - @query.search(:foo) { |r| @call_me.do(r) } + response["rows"].each { |r| @call_me.should_receive(:do).with(r) } + query.search(:node) { |r| @call_me.do(r) } end it "should page through the responses" do @call_me = double("blocky") - @response["rows"].each { |r| @call_me.should_receive(:do).with(r) } - @query.search(:foo, "*:*", nil, 0, 1) { |r| @call_me.do(r) } + response["rows"].each { |r| @call_me.should_receive(:do).with(r) } + query.search(:node, "*:*", nil, 0, 1) { |r| @call_me.do(r) } + end + + context "when :filter_result is provided as a result" do + include_context "filtered search" do + let(:filter_key) { :filter_result } + + before(:each) do + rest.should_receive(:post_rest).with(query_string, args[filter_key]).and_return(response) + end + + it "should return start" do + start = query.search(:node, "platform:rhel", args)[1] + start.should == response['start'] + end + + it "should return total" do + total = query.search(:node, "platform:rhel", args)[2] + total.should == response['total'] + end + + it "should return rows with the filter applied" do + results = query.search(:node, "platform:rhel", args)[0] + + results.each_with_index do |result, idx| + expected = response["rows"][idx] + + result.should have_key("url") + result["url"].should == expected["url"] + + result.should have_key("data") + filter_hash.keys.each do |filter_key| + result["data"].should have_key(filter_key) + result["data"][filter_key].should == expected["data"][filter_key] + end + end + end + + end + end + end + + describe "#partial_search" do + include_context "filtered search" do + let(:filter_key) { :keys } + + it "should emit a deprecation warning" do + # partial_search calls search, so we'll stub search to return empty + query.stub(:search).and_return( [ [], 0, 0 ] ) + Chef::Log.should_receive(:warn).with("DEPRECATED: The 'partial_search' api is deprecated, please use the search api with 'filter_result'") + query.partial_search(:node, "platform:rhel", args) + end + + it "should return an array of filtered hashes" do + rest.should_receive(:post_rest).with(query_string, args[filter_key]).and_return(response) + results = query.partial_search(:node, "platform:rhel", args) + + results.each_with_index do |result, idx| + expected = response["rows"][idx] + + filter_hash.keys.each do |filter_key| + result.should have_key(filter_key) + result[filter_key].should == expected["data"][filter_key] + end + end + end end end end |