Merge branch 'develop' into master

Author: calum-chamberlain

.github/workflows/runtest.yml

@@ -158,6 +158,73 @@ jobs:
           yml: ./codecov.yml
           fail_ci_if_error: true
 
+  test-code-fmf:
+    runs-on: "ubuntu-latest"
+    strategy:
+      matrix:
+        python-version: [3.8]
+      fail-fast: false
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Cache conda
+        uses: actions/cache@v2
+        env:
+          # Increase this value to reset cache if needed by env file has not changed
+          CACHE_NUMBER: 0
+        with:
+          path: ~/conda_pkgs_dir
+          key:
+            ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
+            hashFiles('.github/test_conda_env.yml') }}
+
+      - name: Setup conda
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          miniconda-version: 'latest'
+          python-version: ${{ matrix.python-version }}
+          activate-environment: eqcorrscan-test
+          environment-file: .github/test_conda_env.yml
+          condarc-file: .github/test_condarc.yml
+          use-only-tar-bz2: true  # Must be set for caching to work properly
+
+      - name: install fmf
+        shell: bash -l {0}
+        run: |
+          cd ..
+          git clone https://github.com/beridel/fast_matched_filter.git
+          cd fast_matched_filter
+          pip install -e .
+          cd ../EQcorrscan
+
+      - name: install eqcorrscan
+        shell: bash -l {0}
+        run: |
+          pip install -e .
+
+      - name: print package info
+        shell: bash -l {0}
+        run: |
+          conda info -a
+          conda list
+
+      - name: run correlation test
+        shell: bash -l {0}
+        run: |
+          export OMP_NUM_THREADS=2
+          py.test eqcorrscan/tests/correlate_test.py --cov-report=xml
+
+      - name: upload coverage
+        uses: codecov/codecov-action@v1
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          file: ./coverage.xml
+          flags: unittests
+          name: codecov-umbrella
+          yml: ./codecov.yml
+          fail_ci_if_error: true
+
 # This is a very useful step for debugging, it allows you to ssh into the CI
 # machine (https://github.com/marketplace/actions/debugging-with-tmate).
 # Make sure to open the log before the job starts else you cant see the tmate

CHANGES.md

@@ -1,7 +1,26 @@
 ## Current
-* utils.mag_calc:
- - relative_magnitude: implemented full magnitude bias-correction for CC and SNR
- - relative_amplitude: returns dicts for SNR measurements
+* core.match_filter.tribe
+ - Detect now allows passing of pre-processed data
+* utils.correlate
+ - Fast Matched Filter now supported natively for version >= 1.4.0
+ - Only full correlation stacks are returned now (e.g. where fewer than than
+   the full number of channels are in the stack at the end of the stack, zeros
+   are returned).
+* utils.mag_calc.relative_magnitude
+ - fixed bug where S-picks / traces were used for relative-magnitude calculation
+   against user's choice.
+ - implemented full magnitude bias-correction for CC and SNR
+* utils.mag_calc.relative_amplitude:
+ - returns dicts for SNR measurements
+* utils.catalog_to_dd.write_correlations
+ - Fixed bug on execution of parallel execution.
+ - Added parallel-options for catalog-dt measurements and for stream-preparation
+   before cross correlation-dt measurements.
+ - Default parallelization of dt-computation is now across events (loads CPUs
+   more efficiently), and there is a new option ``max_trace_workers` to use
+   the old parallelization strategy across traces.
+ - Now includes `all_horiz`-option that will correlate all matching horizontal
+   channels no matter to which of these the S-pick is linking.
 
 ## 0.4.3
 * core.match_filter

eqcorrscan/core/match_filter/tribe.py

@@ -428,7 +428,7 @@ def detect(self, stream, threshold, threshold_type, trig_int, plot=False,
                xcorr_func=None, concurrency=None, cores=None,
                ignore_length=False, ignore_bad_data=False, group_size=None,
                overlap="calculate", full_peaks=False, save_progress=False,
-               process_cores=None, **kwargs):
+               process_cores=None, pre_processed=False, **kwargs):
         """
         Detect using a Tribe of templates within a continuous stream.
 
@@ -505,6 +505,10 @@ def detect(self, stream, threshold, threshold_type, trig_int, plot=False,
         :param process_cores:
             Number of processes to use for pre-processing (if different to
             `cores`).
+        :type pre_processed: bool
+        :param pre_processed:
+            Whether the stream has been pre-processed or not to match the
+            templates.
 
         :return:
             :class:`eqcorrscan.core.match_filter.Party` of Families of
@@ -581,12 +585,16 @@ def detect(self, stream, threshold, threshold_type, trig_int, plot=False,
         """
         party = Party()
         template_groups = group_templates(self.templates)
+        if len(template_groups) > 1 and pre_processed:
+            raise NotImplementedError(
+                "Inconsistent template processing and pre-processed data - "
+                "something is wrong!")
         # now we can compute the detections for each group
         for group in template_groups:
             group_party = _group_detect(
                 templates=group, stream=stream.copy(), threshold=threshold,
                 threshold_type=threshold_type, trig_int=trig_int,
-                plot=plot, group_size=group_size, pre_processed=False,
+                plot=plot, group_size=group_size, pre_processed=pre_processed,
                 daylong=daylong, parallel_process=parallel_process,
                 xcorr_func=xcorr_func, concurrency=concurrency, cores=cores,
                 ignore_length=ignore_length, overlap=overlap, plotdir=plotdir,

eqcorrscan/doc/submodules/utils.correlate.fast_matched_filter.rst

@@ -58,7 +58,7 @@ Selecting a correlation function
 
 EQcorrscan strives to use sensible default algorithms for calculating
 correlation values, however, you may want to change how correlations are
-caclulated to be more advantageous to your specific needs.
+calculated to be more advantageous to your specific needs.
 
 
 There are currently 3 different correlations functions currently included in EQcorrscan:
@@ -71,6 +71,9 @@ There are currently 3 different correlations functions currently included in EQc
 
 Number 3 is the default.
 
+A further time-domain correlation backend is available, see note below on using
+Fast Matched Filter within EQcorrscan.
+
 Setting FFT length
 ~~~~~~~~~~~~~~~~~~
 For version 0.4.0 onwards, the "fftw" backend allows the user to pass an `fft_len`
@@ -92,11 +95,8 @@ Using Fast Matched Filter within EQcorrscan
 |FMF| provides fast time-domain correlations for both CPU and GPU
 architectures.  For massively multi-threaded environment this runs faster than
 the frequency-domain routines native to EQcorrscan (when more than 40 CPU cores,
-or an NVIDIA GPU card is available) with less memory consumption. Documentation
-on how to call Fast Matched Filter from within EQcorrscan is provided here: 
-fast_matched_filter_
-
-.. _fast_matched_filter: utils.correlate.fast_matched_filter.html
+or an NVIDIA GPU card is available) with less memory consumption. |FMF| is now
+supported natively, and can be used as a backend by setting `xcorr_func="fmf"`.
 
 .. |FMF| raw:: html