Merge pull request #19 from pndaproject/RELEASE-0.3.0

Release 0.3.0

Author: klyr

CHANGELOG.md

@@ -1,49 +1,57 @@
-# Change Log
-All notable changes to this project will be documented in this file.
-
-## [0.2.1] 2016-12-12
-### Changed
-- Externalized build logic from Jenkins to shell script so it can be reused
-- Refactored the information returned by the Application Detail API to include the YARN application state and also to return information for jobs that have ended. Made the implementation more performant by using the YARN Resource Manager REST API instead of the CLI.
-
-## [0.2.0] 2016-10-21
-### Added
-- PNDA-2233 Jupyter notebook plugin added to deployment manager
-
-## [0.1.1] 2016-09-13
-### Changes
-- Improvements to documentation
-- Enhanced CI support
-
-## [0.1.0] 2016-07-01
-### First version
-
-## [Pre-release]
-
-### Added
-
-- Add hue endpoint to environment endpoints API
-- Application names checked to only contain alphanumeric characters (a-z A-Z 0-9 - and _) because they are used directly in file paths.
-- Added ability to discover HDFS namedservices
-- Added information field to status reports
-- Using an external pacakge repository API instead of internal swift integration
-- Application detail API (GET /applications/<application>/detail) now returns YARN IDs assigned to the running tasks for that application.
-- Oozie error messages are reported when querying for status of an application creation call.
-- Packages are validated on deployment and the error messages reported when querying for status of a package deployment call.
-- Added support for opentsdb.json descriptor for creating metrics when deploying applications.
-- Callback events are sent to the console data logger.
-- Application detail API now completed to return Yarn IDs for any Yarn applications associated with a PNDA application.
-
-### Fixed 
-
-- Return IP address for webhdfs/HTTPFS endpoint instead of hostname
-- Timeout calls to package repository at 120 seconds.
-- Deploying a package that does not exist in the package repository now results in a useful error message being returned to the caller.
-- Fixed defect preventing '-' being used in application names.
-- Fix Zookeeper quorum bug issue
-- Improve package validation to catch packages without 3 point version numbers and where the folder inside the tar does not match the package name.
-- Add list of zookeeper nodes to quorum
-- Remove port=8020 for named service
-- Oozie creator plugin sets 'oozie.wf.application.path' and 'oozie.coord.application.path' to point at the folder not the xml files.
-- Removed some stdout printouts
-- Fixed bug preventing recency parameter being used on the repository/packages API.
+# Change Log
+All notable changes to this project will be documented in this file.
+
+## [0.3.0] 2017-01-20
+### Fixed
+- PNDA-2498: Application package data is now stored in HDFS with a reference to the path only held in the HBase record. This prevents HBase being overloaded with large packages (100MB+).
+
+### Changed
+- PNDA-2485: Pinned all python libraries to strict version numbers
+- PNDA-2499: Return all exceptions to API caller
+
+## [0.2.1] 2016-12-12
+### Changed
+- Externalized build logic from Jenkins to shell script so it can be reused
+- Refactored the information returned by the Application Detail API to include the YARN application state and also to return information for jobs that have ended. Made the implementation more performant by using the YARN Resource Manager REST API instead of the CLI.
+
+## [0.2.0] 2016-10-21
+### Added
+- PNDA-2233 Jupyter notebook plugin added to deployment manager
+
+## [0.1.1] 2016-09-13
+### Changed
+- Improvements to documentation
+- Enhanced CI support
+
+## [0.1.0] 2016-07-01
+### First version
+
+## [Pre-release]
+
+### Added
+
+- Add hue endpoint to environment endpoints API
+- Application names checked to only contain alphanumeric characters (a-z A-Z 0-9 - and _) because they are used directly in file paths.
+- Added ability to discover HDFS namedservices
+- Added information field to status reports
+- Using an external pacakge repository API instead of internal swift integration
+- Application detail API (GET /applications/<application>/detail) now returns YARN IDs assigned to the running tasks for that application.
+- Oozie error messages are reported when querying for status of an application creation call.
+- Packages are validated on deployment and the error messages reported when querying for status of a package deployment call.
+- Added support for opentsdb.json descriptor for creating metrics when deploying applications.
+- Callback events are sent to the console data logger.
+- Application detail API now completed to return Yarn IDs for any Yarn applications associated with a PNDA application.
+
+### Fixed
+
+- Return IP address for webhdfs/HTTPFS endpoint instead of hostname
+- Timeout calls to package repository at 120 seconds.
+- Deploying a package that does not exist in the package repository now results in a useful error message being returned to the caller.
+- Fixed defect preventing '-' being used in application names.
+- Fix Zookeeper quorum bug issue
+- Improve package validation to catch packages without 3 point version numbers and where the folder inside the tar does not match the package name.
+- Add list of zookeeper nodes to quorum
+- Remove port=8020 for named service
+- Oozie creator plugin sets 'oozie.wf.application.path' and 'oozie.coord.application.path' to point at the folder not the xml files.
+- Removed some stdout printouts
+- Fixed bug preventing recency parameter being used on the repository/packages API.

api/src/main/resources/app.py

@@ -83,7 +83,11 @@ def finish():
                 self.finish(ex.msg)
             else:
                 self.set_status(500)
-                self.finish()
+                if "information" in str(ex):
+                    msg = str(ex)
+                else:
+                    msg = {"status": "UNKNOWN", "information": str(ex)}
+                self.finish(msg)
 
         IOLoop.instance().add_callback(callback=finish)
 
@@ -297,12 +301,16 @@ def main():
 
     deployer_utils.fill_hadoop_env(config['environment'])
 
-    package_repository = PackageRepoRestClient(config['config']["package_repository"])
+    package_repository = PackageRepoRestClient(config['config']["package_repository"], config['config']['stage_root'])
     dm = deployment_manager.DeploymentManager(package_repository,
                                               package_registrar.HbasePackageRegistrar(
-                                                  config['environment']['hbase_rest_server']),
+                                                  config['environment']['hbase_thrift_server'],
+                                                  config['environment']['webhdfs_host'],
+                                                  'hdfs',
+                                                  config['environment']['webhdfs_port'],
+                                                  config['config']['stage_root']),
                                               application_registrar.HbaseApplicationRegistrar(
-                                                  config['environment']['hbase_rest_server']),
+                                                  config['environment']['hbase_thrift_server']),
                                               config['environment'],
                                               config['config'])
 

api/src/main/resources/application_creator.py

@@ -22,7 +22,6 @@
 
 import tarfile
 import os
-import io
 import json
 import re
 
@@ -46,14 +45,14 @@ def __init__(self, config, environment, service):
                                  environment['webhdfs_port'],
                                  'hdfs')
 
-    def create_application(self, package_data, package_metadata, application_name, property_overrides):
+    def create_application(self, package_data_path, package_metadata, application_name, property_overrides):
 
         logging.debug("create_application: %s", application_name)
 
         if not re.match('^[a-zA-Z0-9_-]+$', application_name):
             raise FailedCreation('Application name %s may only contain a-z A-Z 0-9 - _' % application_name)
 
-        stage_path = self._stage_package(package_data)
+        stage_path = self._stage_package(package_data_path)
 
         # create each class of components in the package, aggregating any
         # component specific return data for destruction
@@ -164,16 +163,14 @@ def _load_creator(self, component_type):
 
         return creator
 
-    def _stage_package(self, package_data):
+    def _stage_package(self, package_data_path):
 
         logging.debug("_stage_package")
 
         if not os.path.isdir(self._config['stage_root']):
             os.mkdir(self._config['stage_root'])
 
-        file_like_object = io.BytesIO(package_data)
-        tar = tarfile.open(fileobj=file_like_object)
+        tar = tarfile.open(package_data_path)
         stage_path = "%s/%s" % (self._config['stage_root'], uuid.uuid4())
         tar.extractall(path=stage_path)
-        file_like_object.close()
         return stage_path

api/src/main/resources/application_registrar.py

@@ -23,7 +23,7 @@
 import logging
 import json
 import happybase
-from happybase.hbase.ttypes import AlreadyExists
+from Hbase_thrift import AlreadyExists
 
 from lifecycle_states import ApplicationState
 

api/src/main/resources/deployer_utils.py

@@ -97,16 +97,14 @@ def fill_hadoop_env(env):
                     env['yarn_resource_manager_mr_port%s' % rm_instance] = '8032'
                 if role.type == "NODEMANAGER":
                     if 'yarn_node_managers' in env:
-                        env['yarn_node_managers'] = '%s,%s' % (
-                            env['yarn_node_managers'], api.get_host(role.hostRef.hostId).hostname)
+                        env['yarn_node_managers'] = '%s,%s' % (env['yarn_node_managers'], api.get_host(role.hostRef.hostId).hostname)
                     else:
                         env['yarn_node_managers'] = '%s' % api.get_host(
                             role.hostRef.hostId).hostname
         elif service.type == "MAPREDUCE":
             for role in service.get_all_roles():
                 if role.type == "JOBTRACKER":
-                    env['job_tracker'] = '%s:8021' % api.get_host(
-                        role.hostRef.hostId).hostname
+                    env['job_tracker'] = '%s:8021' % api.get_host(role.hostRef.hostId).hostname
                     break
         elif service.type == "ZOOKEEPER":
             for role in service.get_all_roles():
@@ -119,42 +117,37 @@ def fill_hadoop_env(env):
         elif service.type == "HBASE":
             for role in service.get_all_roles():
                 if role.type == "HBASERESTSERVER":
-                    env['hbase_rest_server'] = '%s' % api.get_host(
-                        role.hostRef.hostId).hostname
+                    env['hbase_rest_server'] = '%s' % api.get_host(role.hostRef.hostId).hostname
                     env['hbase_rest_port'] = '20550'
-                    break
+                elif role.type == "HBASETHRIFTSERVER":
+                    env['hbase_thrift_server'] = '%s' % api.get_host(role.hostRef.hostId).hostname
         elif service.type == "OOZIE":
             for role in service.get_all_roles():
                 if role.type == "OOZIE_SERVER":
-                    env['oozie_uri'] = 'http://%s:11000/oozie' % api.get_host(
-                        role.hostRef.hostId).hostname
+                    env['oozie_uri'] = 'http://%s:11000/oozie' % api.get_host(role.hostRef.hostId).hostname
                     break
         elif service.type == "HIVE":
             for role in service.get_all_roles():
                 if role.type == "HIVESERVER2":
-                    env['hive_server'] = '%s' % api.get_host(
-                        role.hostRef.hostId).hostname
+                    env['hive_server'] = '%s' % api.get_host(role.hostRef.hostId).hostname
                     env['hive_port'] = '10000'
                     break
         elif service.type == "IMPALA":
             for role in service.get_all_roles():
                 if role.type == "IMPALAD":
-                    env['impala_host'] = '%s' % api.get_host(
-                        role.hostRef.hostId).hostname
+                    env['impala_host'] = '%s' % api.get_host(role.hostRef.hostId).hostname
                     env['impala_port'] = '21050'
                     break
         elif service.type == "KUDU":
             for role in service.get_all_roles():
                 if role.type == "KUDU_MASTER":
-                    env['kudu_host'] = '%s' % api.get_host(
-                        role.hostRef.hostId).hostname
+                    env['kudu_host'] = '%s' % api.get_host(role.hostRef.hostId).hostname
                     env['kudu_port'] = '7051'
                     break
         elif service.type == "HUE":
             for role in service.get_all_roles():
                 if role.type == "HUE_SERVER":
-                    env['hue_host'] = '%s' % api.get_host(
-                        role.hostRef.hostId).hostname
+                    env['hue_host'] = '%s' % api.get_host(role.hostRef.hostId).hostname
                     env['hue_port'] = '8888'
                     break
 
@@ -240,6 +233,25 @@ def create_file(self, data, remote_file_path):
             sio,
             overwrite=True)
 
+    def append_file(self, data, remote_file_path):
+
+        logging.debug('append to: %s', remote_file_path)
+
+        self._hdfs.append_file(canonicalize(remote_file_path), data)
+
+
+    def stream_file_to_disk(self, remote_file_path, local_file_path):
+        chunk_size = 10*1024*1024
+        offset = 0
+        with open(local_file_path, 'wb') as dest_file:
+            data = self._hdfs.read_file(canonicalize(remote_file_path), offset=offset, length=chunk_size)
+            while True:
+                dest_file.write(data)
+                if len(data) < chunk_size:
+                    break
+                offset += chunk_size
+                data = self._hdfs.read_file(canonicalize(remote_file_path), offset=offset, length=chunk_size)
+
     def read_file(self, remote_file_path):
 
         data = self._hdfs.read_file(canonicalize(remote_file_path))