Update rest tests 7.x (#1265)

Updates rest tests to use artifacts - Backports #1262

* [CI] Updates run-elasticsearch script
* [API] Test Runner: Run tests from downloaded artifacts
* [API] Test Runner: Exit and show instruction if tests files not present
* [DOCS] Updates README files in elasticsearch and elasticsearch-api
* [CI] Test Runner: Moves download artifacts task to elasticsearch
* [CI] Test Runner: Run x-pack tests from artifacts
* [CI] Test Runner: Add download artifacts task when running tests in Jenkins
* [CI] Test Runner - Instantiate admin_client early to reuse in subsequent tasks
* [CI] Test Runner: Refactor
* [DOCS] Update docs

Author: picandocodigo

.ci/run-elasticsearch.sh

@@ -7,14 +7,16 @@
 # Export the TEST_SUITE variable, eg. 'free' or 'platinum' defaults to 'free'.
 # Export the NUMBER_OF_NODES variable to start more than 1 node
 
-# Version 1.2.0
+# Version 1.3.0
 # - Initial version of the run-elasticsearch.sh script
 # - Deleting the volume should not dependent on the container still running
 # - Fixed `ES_JAVA_OPTS` config
 # - Moved to STACK_VERSION and TEST_VERSION
 # - Refactored into functions and imports
 # - Support NUMBER_OF_NODES
 # - Added 5 retries on docker pull for fixing transient network errors
+# - Added flags to make local CCR configurations work
+# - Added action.destructive_requires_name=false as the default will be true in v8
 
 script_path=$(dirname $(realpath -s $0))
 source $script_path/functions/imports.sh
@@ -37,6 +39,7 @@ environment=($(cat <<-END
   --env node.attr.testattr=test
   --env path.repo=/tmp
   --env repositories.url.allowed_urls=http://snapshot.test*
+  --env action.destructive_requires_name=false
 END
 ))
 if [[ "$TEST_SUITE" == "platinum" ]]; then
@@ -50,6 +53,7 @@ if [[ "$TEST_SUITE" == "platinum" ]]; then
     --env xpack.security.http.ssl.certificate=certs/testnode.crt
     --env xpack.security.http.ssl.certificate_authorities=certs/ca.crt
     --env xpack.security.transport.ssl.enabled=true
+    --env xpack.security.transport.ssl.verification_mode=certificate
     --env xpack.security.transport.ssl.key=certs/testnode.key
     --env xpack.security.transport.ssl.certificate=certs/testnode.crt
     --env xpack.security.transport.ssl.certificate_authorities=certs/ca.crt
@@ -109,7 +113,7 @@ END
   docker run \
     --name "$node_name" \
     --network "$network_name" \
-    --env "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
+    --env "ES_JAVA_OPTS=-Xms1g -Xmx1g -da:org.elasticsearch.xpack.ccr.index.engine.FollowingEngineAssertions" \
     "${environment[@]}" \
     "${volumes[@]}" \
     --publish "$http_port":9200 \
@@ -127,5 +131,4 @@ END
   if wait_for_container "$es_node_name" "$network_name"; then
     echo -e "\033[32;1mSUCCESS:\033[0m Running on: $node_url\033[0m"
   fi
-
 done

.ci/run-repository.sh

@@ -49,7 +49,7 @@ if [[ $TEST_SUITE != "platinum" ]]; then
            --name elasticsearch-ruby \
            --rm \
            elastic/elasticsearch-ruby \
-           bundle exec rake test:rest_api
+           bundle exec rake test:download_artifacts test:rest_api
 else
     docker run \
            --network="${network_name}" \
@@ -62,5 +62,5 @@ else
            --name elasticsearch-ruby \
            --rm \
            elastic/elasticsearch-ruby \
-           bundle exec rake test:security
+           bundle exec rake test:download_artifacts test:security
 fi

README.md

@@ -99,37 +99,8 @@ This will clone the Elasticsearch repository into the project, and run `bundle i
 
 To run tests, you need to start a testing cluster on port 9250, or provide a different one in the `TEST_CLUSTER_PORT` environment variable.
 
-There's a Rake task to start the testing cluster. By default this is going to try and use the `elasticsearch` command on your system:
+There's a Rake task to start a testing cluster in a Docker container:
 
-```
-rake test:cluster:start
-```
-
-You can also configure where the elasticsearch startup script is found with the `TEST_CLUSTER_COMMAND` environment variable, e.g.:
-```
-TEST_CLUSTER_COMMAND=~/elasticsearch/bin/elasticsearch rake test:cluster:stop
-```
-
-You also can configure the port, number of nodes, and other settings with environment variables:
-
-```
-TEST_CLUSTER_COMMAND=./tmp/builds/elasticsearch-7.10.0-SNAPSHOT/bin/elasticsearch \
-TEST_CLUSTER_PORT=9250 \
-TEST_CLUSTER_NODES=2 \
-TEST_CLUSTER_NAME=my_cluster \
-ES_JAVA_OPTS='-Xms500m -Xmx500m' \
-TEST_CLUSTER_TIMEOUT=120 \
-rake test:cluster:start
-```
-
-You can stop the cluster with a rake task, passing in the `TEST_CLUSTER_COMMAND` variable:
-
-```
-TEST_CLUSTER_COMMAND=./tmp/builds/elasticsearch-7.10.0-SNAPSHOT/bin/elasticsearch \
-rake test:cluster:stop
-```
-
-There's also a rake task for starting up Elasticsearch in a Docker container:
 `rake docker:start[version]` - E.g.: `rake docker:start[7.x-SNAPSHOT]`. To start the container with X-Pack, pass it in as a parameter: `rake docker:start[7.x-SNAPSHOT,xpack]`.
 
 To run tests against unreleased Elasticsearch versions, you can use the `rake elasticsearch:build` Rake task to build Elasticsearch from the cloned source (use `rake elasticsearch:update` to update the repository):

Rakefile

@@ -15,6 +15,38 @@
 # specific language governing permissions and limitations
 # under the License.
 
+# Admin client is used by tests and other rake tasks to communicate with a running cluster.
+def admin_client
+  $admin_client ||= begin
+                      transport_options = {}
+                      test_suite = ENV['TEST_SUITE'].freeze
+
+                      if (hosts = ENV['TEST_ES_SERVER'] || ENV['ELASTICSEARCH_HOSTS'])
+                        split_hosts = hosts.split(',').map do |host|
+                          /(http\:\/\/)?\S+/.match(host)
+                        end
+                        uri = URI.parse(split_hosts.first[0])
+                      end
+
+                      if test_suite == 'platinum'
+                        transport_options.merge!(
+                          ssl: {
+                            verify: false,
+                            ca_path: CERT_DIR
+                          }
+                        )
+
+                        password = ENV['ELASTIC_PASSWORD']
+                        user     = ENV['ELASTIC_USER'] || 'elastic'
+                        url      = "https://#{user}:#{password}@#{uri.host}:#{uri.port}"
+                      else
+                        url = "http://#{uri&.host || 'localhost'}:#{uri&.port || 9200}"
+                      end
+                      puts "Elasticsearch Client url: #{url}"
+                      Elasticsearch::Client.new(host: url, transport_options: transport_options)
+                    end
+end
+
 import 'rake_tasks/elasticsearch_tasks.rake'
 import 'rake_tasks/test_tasks.rake'
 import 'rake_tasks/docker_tasks.rake'
@@ -44,37 +76,6 @@ CERT_DIR = ENV['CERT_DIR'] || '.ci/certs'
 # Import build task after setting constants:
 import 'rake_tasks/unified_release_tasks.rake'
 
-def admin_client
-  $admin_client ||= begin
-    transport_options = {}
-    test_suite = ENV['TEST_SUITE'].freeze
-
-    if hosts = ENV['TEST_ES_SERVER'] || ENV['ELASTICSEARCH_HOSTS']
-      split_hosts = hosts.split(',').map do |host|
-        /(http\:\/\/)?\S+/.match(host)
-      end
-      uri = URI.parse(split_hosts.first[0])
-    end
-
-    if test_suite == 'platinum'
-      transport_options.merge!(
-        ssl: {
-          verify: false,
-          ca_path: CERT_DIR
-        }
-      )
-
-      password = ENV['ELASTIC_PASSWORD']
-      user     = ENV['ELASTIC_USER'] || 'elastic'
-      url      = "https://#{user}:#{password}@#{uri.host}:#{uri.port}"
-    else
-      url = "http://#{uri&.host || 'localhost'}:#{uri&.port || 9200}"
-    end
-    puts "Elasticsearch Client url: #{url}"
-    Elasticsearch::Client.new(host: url, transport_options: transport_options)
-  end
-end
-
 # TODO: Figure out "bundle exec or not"
 # subprojects.each { |project| $LOAD_PATH.unshift CURRENT_PATH.join(project, "lib").to_s }
 

elasticsearch-api/README.md

@@ -13,8 +13,7 @@ It does not provide an Elasticsearch client; see the
 
 The library is compatible with Ruby 1.9 and higher.
 
-It is compatible with Elasticsearch's API versions from 0.90 till current,
-just use a release matching major version of Elasticsearch.
+It is compatible with Elasticsearch's API versions from 0.90 till current, just use a release matching major version of Elasticsearch.
 
 | Ruby          |   | Elasticsearch |
 |:-------------:|:-:| :-----------: |
@@ -190,18 +189,18 @@ Elasticsearch::API.serializer.dump({foo: 'bar'})
 
 ## Development
 
-To work on the code, clone and bootstrap the main repository first --
-please see instructions in the main [README](../README.md#development).
+To work on the code, clone and bootstrap the main repository first -- please see instructions in the main [README](../README.md#development).
 
-To run tests, launch a testing cluster -- again, see instructions
-in the main [README](../README.md#development) -- and use the Rake tasks:
+To run tests, launch a testing cluster -- again, see instructions in the main [README](../README.md#development) -- and use the Rake tasks:
 
 ```
 time rake test:unit
 time rake test:integration
 ```
 
-You can read more about how we test the API using the Elasticsearch spec in [the test runner README](https://github.com/elastic/elasticsearch-ruby/tree/master/api-spec-testing#rest-api-yaml-test-runner).
+We run the test suite for Elasticsearch's Rest API tests. You can read more about this in [the test runner README](https://github.com/elastic/elasticsearch-ruby/tree/master/api-spec-testing#rest-api-yaml-test-runner).
+
+The `rest_api` needs the test files from Elasticsearch. You can run the rake task to download the test artifacts in the root folder of the project. This task needs a running cluster to determine which version and build hash of Elasticsearch to use and test against. `TEST_ES_SERVER=http://localhost:9200 rake test:download_artifacts`. This will download the necessary files used for the integration tests to `./tmp`.
 
 ## License
 

elasticsearch-api/Rakefile

@@ -16,14 +16,7 @@
 # under the License.
 
 require 'bundler/gem_tasks'
-
-def __current__
-  Pathname( File.expand_path('..', __FILE__) )
-end
-
-def git_specs(command, options={})
-  sh "git --git-dir=#{__current__.join('../tmp/elasticsearch/.git')} --work-tree=#{__current__.join('../tmp/elasticsearch')} #{command}", options
-end
+require 'json'
 
 task(:default) { system 'rake --tasks' }
 task test: 'test:unit'
@@ -46,73 +39,28 @@ namespace :test do
     t.exclude_pattern = 'spec/**{,/*/**}/rest_api_yaml_spec.rb'
   end
 
-  # Rest API Spec tests - rake test:rest_api
+  desc "Run Rest API Spec tests"
   RSpec::Core::RakeTask.new(:rest_api) do |t|
-    t.pattern = 'spec/**{,/*/**}/rest_api_yaml_spec.rb'
-  end
-
-  desc 'Update the repository with YAML tests'
-  task :update do
-    git_specs 'fetch origin', verbose: true
-  end
-
-  desc "Run integration tests"
-  task integration: :update do
     require 'elasticsearch'
-
-    branches = `git --git-dir=#{__current__.join('../tmp/elasticsearch/.git')} --work-tree=#{__current__.join('../tmp/elasticsearch')} branch --no-color`
-
-    current_branch = branches.
-      split("\n").
-      select { |b| b =~ /^\*/ }.
-      reject { |b| b =~ /no branch|detached/ }.
-      map    { |b| b.gsub(/^\*\s*/, '') }.
-      first
-
-    unless current_branch
-      STDERR.puts "[!] Unable to determine current branch, defaulting to 'master'"
-      current_branch = 'master'
-    end
-
     # Check if a test cluster is running
     begin
       url = ENV['TEST_CLUSTER_URL'] || ENV['TEST_ES_SERVER']
-      url = "http://localhost:#{ENV['TEST_CLUSTER_PORT'] || 9250}" unless url
+      url = "http://localhost:#{ENV['TEST_CLUSTER_PORT'] || 9200}" unless url
       client = Elasticsearch::Client.new :url => url
       es_version_info = client.info['version']
+      version_number = es_version_info['number']
       build_hash      = es_version_info['build_hash']
-      cluster_running = true
     rescue Faraday::ConnectionFailed
       STDERR.puts "[!] Test cluster not running?"
-      cluster_running = false
+      exit 1
     end
 
-    checkout_specs_version = ENV['TEST_NO_CHECKOUT'].nil? ? true : false
-    checkout_build_hash    = ENV['TEST_BUILD_REF'] || build_hash
-    ENV['TEST_BUILD_REF']  = checkout_build_hash
+    t.pattern = 'spec/**{,/*/**}/rest_api_yaml_spec.rb'
+  end
 
-    begin
-      unless checkout_specs_version
-        STDERR.puts '-'*80, "YAML tests: Not switching, TEST_NO_CHECKOUT=y", '-'*80
-      end
-
-      if checkout_specs_version && !checkout_build_hash
-        STDERR.puts "[!] Cannot determine checkout build hash -- server not running or TEST_BUILD_REF not specified"
-        exit(1)
-      end
-
-      if checkout_specs_version && checkout_build_hash
-        # Checkout the commit corresponding to the running server build, or passed TEST_BUILD_REF
-        name = ENV['CI'] ? checkout_build_hash : "[\e[1m#{checkout_build_hash}\e[0m]"
-        STDERR.puts '-'*80, "YAML tests: Switching to #{name} from #{current_branch}", '-'*80
-        git_specs "checkout #{checkout_build_hash} --force --quiet"
-      end
-
-      Rake::Task['test:rest_api'].invoke
-
-    ensure
-      git_specs "checkout #{current_branch} --force --quiet" if checkout_specs_version && current_branch
-    end
+  desc "Run integration tests"
+  task :integration do
+    Rake::Task['test:rest_api'].invoke
   end
 
   desc 'Run unit and integration tests'
@@ -121,6 +69,42 @@ namespace :test do
     Rake::Task['test:integration'].invoke
   end
 
+  def refresh_artifacts(build_hash, version_number)
+    unless build_hash
+      STDERR.puts "[!] Cannot determine checkout build hash -- server not running"
+      exit(1)
+    end
+
+    puts 'Downloading artifacts file...'
+    filename = 'tmp/artifacts.json'
+    `curl -s https://artifacts-api.elastic.co/v1/versions/#{version_number} -o #{filename}`
+
+    unless File.exists?("./#{filename}")
+      STDERR.puts '[!] Couldn\'t download artifacts file'
+      exit 1
+    end
+
+    artifacts = JSON.parse(File.read('./tmp/artifacts.json'))
+
+    build_hash_artifact = artifacts['version']['builds'].select do |a|
+      a.dig('projects', 'elasticsearch', 'commit_hash') == build_hash
+    end.first
+    # Dig into the elasticsearch packages, search for the rest-resources-zip package and catch the URL:
+    zip_url = build_hash_artifact.dig('projects', 'elasticsearch', 'packages').select { |k,v| k =~ /rest-resources-zip/ }.map { | _, v| v['url'] }.first
+
+    filename = zip_url.split('/').last
+    puts 'Downloading zip file:'
+    `curl -s #{zip_url} -o tmp/#{filename}`
+
+    unless File.exists?("./tmp/#{filename}")
+      STDERR.puts '[!] Couldn\'t download artifact'
+      exit 1
+    end
+
+    puts "Unzipping file #{filename}"
+    `unzip -o tmp/#{filename} -d tmp/`
+  end
+
   namespace :cluster do
     desc "Start Elasticsearch nodes for tests"
     task :start do

elasticsearch-api/spec/elasticsearch/api/rest_api_yaml_spec.rb

@@ -6,6 +6,13 @@
 require 'rest_yaml_tests_helper'
 
 describe 'Rest API YAML tests' do
+  if REST_API_YAML_FILES.empty?
+    logger = Logger.new($stdout)
+    logger.error 'No test files found!'
+    logger.info 'Use rake rake test:download_artifacts in the root directory of the project to download the test artifacts.'
+    exit 1
+  end
+
   # Traverse YAML files and create TestFile object:
   REST_API_YAML_FILES.each do |file|
     begin