Backports for stable 2.0.7 (#9648)

* parity-version: bump stable to 2.0.7

* HF in POA Sokol (2018-09-19) (#9607)

poanetwork/poa-chain-spec#86

* fix failing node-table tests on mac os, closes #9632 (#9633)

* fix(light_fetch): avoid race with BlockNumber::Latest (#9665)

* CI: Remove unnecessary pipes (#9681)

* ci: reduce gitlab pipelines significantly

* ci: build pipeline for PR

* ci: remove dead weight

* ci: remove github release script

* ci: remove forever broken aura tests

* ci: add random stuff to the end of the pipes

* ci: add wind and mac to the end of the pipe

* ci: remove snap artifacts

* ci: (re)move dockerfiles

* ci: clarify job names

* ci: add cargo audit job

* ci: make audit script executable

* ci: ignore snap and docker files for rust check

* ci: simplify audit script

* ci: rename misc to optional

* ci: add publish script to releaseable branches

* ci: more verbose cp command for windows build

* ci: fix weird binary checksum logic in push script

* ci: fix regex in push script for windows

* ci: simplify gitlab caching

* docs: align README with ci changes

* ci: specify default cargo target dir

* ci: print verbose environment

* ci: proper naming of scripts

* ci: restore docker files

* ci: use docker hub file

* ci: use cargo home instead of cargo target dir

* ci: touch random rust file to trigger real builds

* ci: set cargo target dir for audit script

* ci: remove temp file

* ci: don't export the cargo target dir in the audit script

* ci: fix windows unbound variable

* docs: fix gitlab badge path

* rename deprecated gitlab ci variables

https://docs.gitlab.com/ee/ci/variables/#9-0-renaming

* ci: fix git compare for nightly builds

* test: skip c++ example for all platforms but linux

* ci: add random rust file to trigger tests

* ci: remove random rust file

* disable cpp lib test for mac, win and beta (#9686)

* cleanup ci merge

* parity: bump clib

* ci: fix tests

* ci: disable c++ example

* Docker: run as parity user (#9689)

* CI: Skip docs job for nightly (#9693)

* ci: force-tag wiki changes

* ci: force-tag wiki changes

* ci: skip docs job for master and nightly

* ci: revert docs job checking for nightly tag

* ci: exclude docs job from nightly builds in gitlab script

* fix (light/provider) : Make `read_only executions` read-only (#9591)

* `ExecutionsRequest` from light-clients as read-only

This changes so all `ExecutionRequests` from light-clients are executed
as read-only which the `virtual``flag == true ensures.

This boost up the current transaction to always succeed

Note, this only affects `eth_estimateGas` and `eth_call` AFAIK.

* grumbles(revert renaming) : TransactionProof

* grumbles(trace) : remove incorrect trace

* grumbles(state/prove_tx) : explicit `virt`

Remove the boolean flag to determine that a `state::prove_transaction`
whether it should be executed in a virtual context or not.

Because of that also rename the function to
`state::prove_transction_virtual` to make more clear

* ethcore: fix detection of major import (#9552)

* sync: set state to idle after sync is completed

* sync: refactor sync reset

* parity: revert clib bump and fix tests

* Fix path to parity.h (#9274)

* Fix path to parity.h

* Fix other paths as well

* ethcore-io retries failed work steal (#9651)

* ethcore-io uses newer version of crossbeam && retries failed work steal

* ethcore-io non-mio service uses newer crossbeam

Author: 5chdn

.gitlab-ci.yml

@@ -1,243 +1,180 @@
 stages:
   - test
-  - push-release
   - build
-  - docs
+  - publish
+  - optional
+
+image:                             parity/rust:gitlab-ci
+
 variables:
-  RUST_BACKTRACE: "1"
-  RUSTFLAGS: ""
-  CARGOFLAGS: ""
-  CI_SERVER_NAME: "GitLab CI"
+  CI_SERVER_NAME:                  "GitLab CI"
+  CARGO_HOME:                      "${CI_PROJECT_DIR}/.cargo"
+  BUILD_TARGET:                    ubuntu
+  BUILD_ARCH:                      amd64
+  CARGO_TARGET:                    x86_64-unknown-linux-gnu
+
 cache:
-  key: "$CI_BUILD_STAGE-$CI_BUILD_REF_NAME"
+  key:                             "${CI_JOB_NAME}"
   paths:
-    - target/
-  untracked: true
-linux-amd64:
-  stage: build
-  image: parity/rust:gitlab-ci
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
-  script:
-    - rustup default stable
-    # ARGUMENTS: 1. BUILD_PLATFORM (target for binaries) 2. PLATFORM (target for cargo) 3. ARC (architecture) 4. & 5. CC & CXX flags 6. binary identifier
-    - scripts/gitlab-build.sh x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu amd64 gcc g++ linux
-  tags:
-    - rust-stable
-  artifacts:
-    paths:
-    - parity.zip
-    name: "stable-x86_64-unknown-linux-gnu_parity"
-linux-i686:
-  stage: build
-  image: parity/rust-i686:gitlab-ci
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
-  script:
-    - scripts/gitlab-build.sh i686-unknown-linux-gnu i686-unknown-linux-gnu i386 gcc g++ linux
-  tags:
-    - rust-i686
-  artifacts:
-    paths:
-      - parity.zip
-    name: "i686-unknown-linux-gnu"
-  allow_failure: true
-linux-armv7:
-  stage: build
-  image: parity/rust-armv7:gitlab-ci
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
-  script:
-    - scripts/gitlab-build.sh armv7-unknown-linux-gnueabihf armv7-unknown-linux-gnueabihf armhf arm-linux-gnueabihf-gcc arm-linux-gnueabihf-g++ linux
-  tags:
-    - rust-arm
-  artifacts:
-    paths:
-    - parity.zip
-    name: "armv7_unknown_linux_gnueabihf_parity"
-  allow_failure: true
-linux-armhf:
-  stage: build
-  image: parity/rust-arm:gitlab-ci
-  only:
-    - beta
-    - tags
+    - ./target
+    - ./.cargo
+
+.releaseable_branches:             # list of git refs for building GitLab artifacts (think "pre-release binaries")
+  only:                            &releaseable_branches
     - stable
-    - triggers
-  script:
-    - scripts/gitlab-build.sh arm-unknown-linux-gnueabihf arm-unknown-linux-gnueabihf armhf arm-linux-gnueabihf-gcc arm-linux-gnueabihf-g++ linux
-  tags:
-    - rust-arm
-  artifacts:
-    paths:
-    - parity.zip
-    name: "arm-unknown-linux-gnueabihf_parity"
-  allow_failure: true
-linux-aarch64:
-  stage: build
-  image: parity/rust-arm64:gitlab-ci
-  only:
     - beta
     - tags
-    - stable
-    - triggers
-  script:
-    - scripts/gitlab-build.sh aarch64-unknown-linux-gnu aarch64-unknown-linux-gnu arm64 aarch64-linux-gnu-gcc aarch64-linux-gnu-g++ linux
-  tags:
-    - rust-arm
+
+.collect_artifacts:                &collect_artifacts
   artifacts:
+    name:                          "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
+    when:                          on_success
+    expire_in:                     1 mos
     paths:
-    - parity.zip
-    name: "aarch64-unknown-linux-gnu_parity"
-linux-snap:
-  stage: build
-  image: parity/snapcraft:gitlab-ci
-  only:
-    - stable
-    - beta
-    - tags
-    - triggers
-  script:
-    - scripts/gitlab-build.sh x86_64-unknown-snap-gnu x86_64-unknown-linux-gnu amd64 gcc g++ snap
+    - artifacts/
+
+.determine_version:                &determine_version
+  - VERSION="$(sed -r -n '1,/^version/s/^version = "([^"]+)".*$/\1/p' Cargo.toml)"
+  - DATE_STR="$(date +%Y%m%d)"
+  - ID_SHORT="$(echo ${CI_COMMIT_SHA} | cut -c 1-7)"
+  - test "${CI_COMMIT_REF_NAME}" = "nightly" && VERSION="${VERSION}-${ID_SHORT}-${DATE_STR}"
+  - export VERSION
+  - echo "Version = ${VERSION}"
+
+test-linux:
+  stage:                           test
+  variables:
+    RUN_TESTS:                     cargo
+  script:
+    - scripts/gitlab/test-all.sh stable
   tags:
     - rust-stable
-  artifacts:
-    paths:
-    - parity.zip
-    name: "stable-x86_64-unknown-snap-gnu_parity"
-darwin:
-  stage: build
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
+
+build-linux:
+  stage:                           build
+  only:                            *releaseable_branches
+  variables:
+    CARGO_TARGET:                  x86_64-unknown-linux-gnu
   script:
-    - scripts/gitlab-build.sh x86_64-apple-darwin x86_64-apple-darwin macos gcc g++ macos
+    - scripts/gitlab/build-unix.sh
+  <<:                              *collect_artifacts
   tags:
-    - osx
-  artifacts:
-    paths:
-    - parity.zip
-    name: "x86_64-apple-darwin_parity"
-windows:
-  cache:
-    key: "%CI_BUILD_STAGE%-%CI_BUILD_REF_NAME%"
-    untracked: true
-  stage: build
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
-  script:
-    - sh scripts/gitlab-build.sh x86_64-pc-windows-msvc x86_64-pc-windows-msvc amd64 "" "" windows
+    - rust-stable
+
+build-darwin:
+  stage:                           build
+  only:                            *releaseable_branches
+  variables:
+    CARGO_TARGET:                  x86_64-apple-darwin
+    CC:                            gcc
+    CXX:                           g++
+  script:
+    - scripts/gitlab/build-unix.sh
+  tags:
+    - rust-osx
+  <<:                              *collect_artifacts
+
+build-windows:
+  stage:                           build
+  only:                            *releaseable_branches
+  variables:
+    CARGO_TARGET:                  x86_64-pc-windows-msvc
+  script:
+    - sh scripts/gitlab/build-windows.sh
   tags:
    - rust-windows
-  artifacts:
-    paths:
-    - parity.zip
-    name: "x86_64-pc-windows-msvc_parity"
-android-armv7:
-  stage: build
-  image: parity/parity-android:latest
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
-  script:
-    - cargo build --target=armv7-linux-androideabi
+  <<:                              *collect_artifacts
+
+publish-docker:
+  stage:                           publish
+  only:                            *releaseable_branches
+  cache: {}
+  dependencies:
+    - build-linux
   tags:
-    - rust-arm
-  allow_failure: true
-  artifacts:
-    paths:
-    - parity.zip
-    name: "armv7-linux-androideabi_parity"
-docker-build:
-  stage: build
-  only:
-    - tags
-    - master
-    - beta
-    - stable
-    - triggers
-  before_script:
-    - docker info
+    - shell
   script:
-    - if [ "$CI_BUILD_REF_NAME" == "master" ]; then DOCKER_TAG="latest"; else DOCKER_TAG=$CI_BUILD_REF_NAME; fi
-    - echo "Tag:" $DOCKER_TAG
-    - docker login -u $Docker_Hub_User_Parity -p $Docker_Hub_Pass_Parity
-    - scripts/docker-build.sh $DOCKER_TAG
-    - docker logout
-  tags:
-    - docker
-test-coverage:
-  stage: test
+    - scripts/gitlab/publish-docker.sh parity
+
+publish-awss3:
+  stage:                           publish
+  only:                            *releaseable_branches
+  cache: {}
+  dependencies:
+    - build-linux
+    - build-darwin
+    - build-windows
+  before_script:                   *determine_version
+  script:
+    - scripts/gitlab/publish-awss3.sh
+  tags:
+    - shell
+
+docs-jsonrpc:
+  stage:                            optional
   only:
-    - master
+    - tags
+  except:
+    - nightly
+  cache: {}
   script:
-    - scripts/gitlab-test.sh test-coverage
+    - scripts/gitlab/docs-jsonrpc.sh
   tags:
-    - kcov
-  allow_failure: true
-test-rust-stable:
-  stage: test
-  image: parity/rust:gitlab-ci
+    - shell
+
+cargo-audit:
+  stage:                           optional
   script:
-    - scripts/gitlab-test.sh stable
+    - scripts/gitlab/cargo-audit.sh
   tags:
     - rust-stable
-test-rust-beta:
-  stage: test
-  only:
-    - triggers
-    - master
-  image: parity/rust:gitlab-ci
+
+test-android:
+  stage:                           optional
+  image:                           parity/rust-android:gitlab-ci
+  variables:
+    CARGO_TARGET:                  armv7-linux-androideabi
   script:
-    - scripts/gitlab-test.sh beta
+    - scripts/gitlab/test-all.sh stable
   tags:
-    - rust-beta
-  allow_failure: true
-test-rust-nightly:
-  stage: test
-  only:
-    - triggers
-    - master
-  image: parity/rust:gitlab-ci
-  script:
-    - scripts/gitlab-test.sh nightly
+    - rust-arm
+
+test-darwin:
+  stage:                           optional
+  variables:
+    CARGO_TARGET:                  x86_64-apple-darwin
+    CC:                            gcc
+    CXX:                           g++
+    RUN_TESTS:                     cargo
+  script:
+    - scripts/gitlab/test-all.sh stable
+  tags:
+    - rust-osx
+
+test-windows:
+  stage:                           optional
+  variables:
+    CARGO_TARGET:                  x86_64-pc-windows-msvc
+    RUN_TESTS:                     cargo
+  script:
+    - sh scripts/gitlab/test-all.sh stable
   tags:
-    - rust
-    - rust-nightly
-  allow_failure: true
-json-rpc-docs:
-  stage: docs
-  only:
-    - tags
-  image: parity/rust:gitlab-ci
+   - rust-windows
+
+test-beta:
+  stage:                           optional
+  variables:
+    RUN_TESTS:                     cargo
   script:
-    - scripts/gitlab-rpc-docs.sh
+    - scripts/gitlab/test-all.sh beta
   tags:
-    - docs
-  cache: {}
-push-release:
-  stage: push-release
-  only:
-    - tags
-    - triggers
-  image: parity/rust:gitlab-ci
+    - rust-beta
+
+test-nightly:
+  stage:                           optional
+  variables:
+    RUN_TESTS:                     all
   script:
-    - scripts/gitlab-push-release.sh
+    - scripts/gitlab/test-all.sh nightly
   tags:
-    - curl
+    - rust-nightly