Misc build tweaks.

Author: metasim

.travis.yml

@@ -40,6 +40,8 @@ log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
 log4j.logger.org.locationtech.rasterframes=WARN
 log4j.logger.org.locationtech.rasterframes.ref=WARN
 log4j.logger.org.apache.parquet.hadoop.ParquetRecordReader=OFF
+log4j.logger.geotrellis.spark=INFO
+log4j.logger.geotrellis.raster.gdal=ERROR
 
 # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
 log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL

core/src/it/resources/log4j.properties

@@ -23,14 +23,15 @@
   - Revisit use of `Tile` equality since [it's more strict](https://github.com/locationtech/geotrellis/pull/2991)
   - Update `reference.conf` to use `geotrellis.raster.gdal` namespace.
   - Replace all uses of `TileDimensions` with `geotrellis.raster.Dimensions[Int]`.
+* Upgraded to `gdal-warp-bindings` 1.0.0.  
+* Upgraded to Spark 2.4.5
 * Formally abandoned support for Python 2. Python 2 is dead. Long live Python 2.
 * Introduction of type hints in Python API. 
 * Add functions for changing cell values based on either conditions or to achieve a distribution of values. ([#449](https://github.com/locationtech/rasterframes/pull/449))
     * Add `rf_local_min`, `rf_local_max`, and `rf_local_clip` functions.
     * Add cell value scaling functions `rf_rescale` and `rf_standardize`.
     * Add `rf_where` function, similar in spirit to numpy's `where`, or a cell-wise version of Spark SQL's `when` and `otherwise`.
 * Add `rf_sqrt` function to compute cell-wise square root.
-* Upgraded to Spark 2.4.5
 
 ## 0.8.x
 

docs/src/main/paradox/release-notes.md

@@ -20,6 +20,25 @@
 
 from pweave import PwebPandocFormatter
 
+# Setuptools/easy_install doesn't properly set the execute bit on the Spark scripts,
+# So this preemptively attempts to do it.
+def _chmodit():
+    try:
+        from importlib.util import find_spec
+        import os
+        module_home = find_spec("pyspark").origin
+        print(module_home)
+        bin_dir = os.path.join(os.path.dirname(module_home), 'bin')
+        for filename in os.listdir(bin_dir):
+            try:
+                os.chmod(os.path.join(bin_dir, filename), mode=0o555, follow_symlinks=True)
+            except OSError:
+                pass
+    except ImportError:
+        pass
+
+_chmodit()
+
 
 class PegdownMarkdownFormatter(PwebPandocFormatter):
     def __init__(self, *args, **kwargs):

pyrasterframes/src/main/python/docs/__init__.py

@@ -68,6 +68,12 @@ def find_pyrasterframes_assembly() -> Union[bytes, str]:
     return jarpath[0]
 
 
+def quiet_logs(sc):
+    logger = sc._jvm.org.apache.log4j
+    logger.LogManager.getLogger("geotrellis.raster.gdal").setLevel(logger.Level.ERROR)
+    logger.LogManager.getLogger("akka").setLevel(logger.Level.ERROR)
+
+
 def create_rf_spark_session(master="local[*]", **kwargs: str) -> SparkSession:
     """ Create a SparkSession with pyrasterframes enabled and configured. """
     jar_path = find_pyrasterframes_assembly()
@@ -86,6 +92,8 @@ def create_rf_spark_session(master="local[*]", **kwargs: str) -> SparkSession:
              .config(conf=conf)  # user can override the defaults
              .getOrCreate())
 
+    quiet_logs(spark)
+
     try:
         spark.withRasterFrames()
         return spark