Add startup script

Add precompilation step to docker and setup script
Added Readme content
Refactored functionality into modules

Author: jacobmsft

Dockerfile

@@ -29,7 +29,18 @@ RUN apt-get update && \
         rsync \
     	file \
     	gettext && \
-        apt-get clean
+        apt-get clean && \
+        ln -s /usr/bin/python3.8 /usr/bin/python && \
+        ln -s /usr/bin/pip3 /usr/bin/pip 
+
+# Install .NET Core for tools/builds
+RUN cd /tmp && \
+    wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb && \
+    dpkg -i packages-microsoft-prod.deb && \
+    add-apt-repository universe && \
+    apt-get update && \
+    rm packages-microsoft-prod.deb
+RUN apt-get install -y dotnet-sdk-3.1
 
 # Clone our setup and run scripts
 #RUN git clone https://github.com/microsoft/codeql-container /usr/local/startup_scripts
@@ -44,14 +55,13 @@ ENV CODEQL_HOME /usr/local/codeql-home
 # record the latest version of the codeql-cli
 RUN python3 /usr/local/startup_scripts/get-latest-codeql-version.py > /tmp/codeql_version
 RUN mkdir -p ${CODEQL_HOME} \
-${CODEQL_HOME}/codeql-cli \
 ${CODEQL_HOME}/codeql-repo \
 ${CODEQL_HOME}/codeql-go-repo \
 /opt/codeql
 
 RUN CODEQL_VERSION=$(cat /tmp/codeql_version) && \
     wget -q https://github.com/github/codeql-cli-binaries/releases/download/${CODEQL_VERSION}/codeql-linux64.zip -O /tmp/codeql_linux.zip && \
-    unzip /tmp/codeql_linux.zip -d ${CODEQL_HOME}/codeql-cli && \
+    unzip /tmp/codeql_linux.zip -d ${CODEQL_HOME} && \
     rm /tmp/codeql_linux.zip
 
 # get the latest codeql queries and record the HEAD
@@ -60,9 +70,9 @@ RUN git clone https://github.com/github/codeql ${CODEQL_HOME}/codeql-repo && \
 RUN git clone https://github.com/github/codeql-go ${CODEQL_HOME}/codeql-go-repo && \
     git --git-dir ${CODEQL_HOME}/codeql-go-repo/.git log --pretty=reference -1 > /opt/codeql/codeql-go-repo-last-commit
 
-ENV PATH="${CODEQL_HOME}/codeql-cli/codeql:${PATH}"
+ENV PATH="${CODEQL_HOME}/codeql:${PATH}"
 
 # Pre-compile our queries to save time later
 #RUN codeql query compile --threads=0 ${CODEQL_HOME}/codelq-repo/*/ql/src/codeql-suites/*-.qls
 #RUN codeql query compile --threads=0 ${CODEQL_HOME}/codelq-go-repo/ql/src/codeql-suites/*-.qls
-#ENTRYPOINT ["python3", "/usr/local/startup_scripts/setup.py"]
+ENTRYPOINT ["python3", "/usr/local/startup_scripts/startup.py"]

README.md

@@ -1,7 +1,67 @@
+## CodeQL Container
+
+> **Note:** CodeQL container is currently in **public preview**. Please report any bugs to https://github.com/microsoft/codeql-container/issues.
+> Current version of CodeQL only works for interpreted languages. We will add compiled languages support on future versions.
+
+CodeQL Container is a project aimed at making it easier to start using CodeQL (https://github.com/github/codeql). This project
+contains a Docker file which builds a container with the latest version of codeql-cli and codeql queries precompiled. 
+It also contains scripts to keep the toolchain in the container updated. You can use this container to:
+
+* Start using codeql-cli and run queries on your projects without installing it on your local machine.
+* Use is as an environment to develop codeql queries and test them.
+* Test how the queries perform in windows and linux environments.
+
+We shall continue to add more features and would be happy to accept contributions from the community.
+
+### Basic Usage
+
+#### Downloading a pre-built container
+We keep updating the docker image periodically and uploading it to the Microsoft Container Registry at: mcr.microsoft.com/codeql/codeql-container.
+You can run the image by running the command:
+```
+$ docker run --rm mcr.microsoft.com/codeql/codeql-container
+```
+
+If you want to analyze a particular source directory with codeql, run the container as:
+```
+$ docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=<query run...>
+```
+where /dir/to/analyze contains the source files that have to be analyzed, and /dir/for/results is where the result output 
+needs to be stored, and you can specify QL_PACKS environment variable for specific QL packs to be run on the provided code.
+For more information on CodeQL and QL packs, please visit https://www.github.com/github/codeql.
+CODEQL_CLI_ARGS are the arguments that will be directly passed on to the codeql-cli. Some examples of CODEQL_CLI_ARGS are:
+```
+CODEQL_CLI_ARGS = database create /opt/src/source_db
+```
+**Note:** If you map your source volume to some other mountpoint other than /opt/src, you will have to make the corresponding changes
+in the CODEQL_CLI_ARGS.
+
+There are some additional docker environment variables that you can specify to control the execution of the container:
+    * CHECK_LATEST_CODEQL_CLI - If there is a newer version of codeql-cli, download and install it
+    * CHECK_LATEST_QUERIES - if there is are updates to the codeql queries repo, download and use it
+    * PRECOMPILE_QUERIES - If we downloaded new queries, precompile all new query packs (query execution will be faster)
+        WARNING: Precompiling query packs might take a few hours, depending on speed of your machine and the CPU/Memory limits (if any) 
+        you have placed on the container.
+
+Since codeql first creates a database of the code representation, and then analyzes the db for issues, we need a few commands to 
+analyze a source code repo. 
+For example, if you want to analyze a python project source code placed in /dir/to/analyze (or C:\dir\to\analyze for example, in windows), 
+to analyze and get a sarif result file, you will have to run:
+$ docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database create --language=python /opt/src/source_db"
+$ docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database analyze --format=sarifv2 --output=/opt/results/issues.sarif /opt/src/source_db
+
+```
+This command will run all the ql packs related to security and output the results to the results folder.
+
+#### Building the container
+Building the container should be pretty straightforward.
+git clone ...
+cd src
+docker build . -f Dockerfile -t codeql-container
 
 # Contributing
 
-This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+This project welcomes contributions and suggestions. Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
 the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
 

container/get-latest-codeql-version.py

@@ -3,7 +3,6 @@
 
 import os
 import sys
-sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
 
 from libs.github import get_latest_github_repo_version
 

container/libs/codeql.py

@@ -20,7 +20,6 @@ class CodeQL:
     TEMP_DIR ='/tmp'
 
     # Error codes
-    ERROR_EXECUTING_COMMAND = 1
     ERROR_EXECUTING_CODEQL = 2
     ERROR_UNKNOWN_OS = 3
     ERROR_GIT_COMMAND = 4
@@ -65,7 +64,7 @@ def download_and_install_latest_codeql_queries(self):
             exit(ERROR_GIT_COMMAND)
 
     def get_current_local_version(self):
-        ret_string = check_output_wrapper(f'{self.CODEQL_HOME}/codeql-cli/codeql/codeql version', shell=True).decode("utf-8")
+        ret_string = check_output_wrapper(f'{self.CODEQL_HOME}/codeql/codeql version', shell=True).decode("utf-8")
         if ret_string is CalledProcessError:
             logger.error("Could not run codeql command")
             exit(ERROR_EXECUTING_CODEQL)
@@ -84,3 +83,13 @@ def install_codeql_cli(self, download_path):
         codeql_dir = f'{self.CODEQL_HOME}/codeql-cli'
         wipe_and_create_dir(codeql_dir)
         ret1 = check_output_wrapper(f'unzip {download_path} -d {codeql_dir}', shell=True)
+        
+    def precompile_queries(self):
+        execute_codeql_command(f' query compile --search-path {self.CODEQL_HOME} {self.CODEQL_HOME}/codeql-repo/*/ql/src/codeql-suites/*.qls')
+
+    def execute_codeql_command(self, args):
+        ret_string = check_output_wrapper(f'{self.CODEQL_HOME}/codeql/codeql {args}', shell=True).decode("utf-8")
+        if ret_string is CalledProcessError:
+            logger.error("Could not run codeql command")
+            exit(ERROR_EXECUTING_CODEQL)
+        return ret_string

container/libs/utils.py

@@ -1,13 +1,14 @@
-from sys import exit
+import sys
 from shutil import rmtree
 from os import environ, mkdir, name as os_name
 from subprocess import check_output, CalledProcessError
-from logging import getLogger
+from logging import Logger, getLogger, INFO, StreamHandler, Formatter
 
 logger = getLogger('codeql-container')
+ERROR_EXECUTING_COMMAND = 1
 
 # get secrets from the env
-def get_env_variable(self, var_name, optional=False):
+def get_env_variable(var_name, optional=False):
     """
     Retrieve an environment variable. Any failures will cause an exception
     to be thrown.
@@ -37,3 +38,18 @@ def check_output_wrapper(*args, **kwargs):
 def wipe_and_create_dir(dirname):
     rmtree(dirname)
     mkdir(dirname)
+
+def get_logger(log_level=INFO):
+    try:
+        logger = getLogger(sys._getframe(1).f_code.co_name)
+        logger.setLevel(log_level)
+        if not logger.handlers:
+            log_handler = StreamHandler(sys.stdout)
+            log_handler.setLevel(log_level)
+            log_handler.setFormatter(Formatter('[%(asctime)s] %(levelname)s: %(message)s'))
+
+            logger = getLogger(sys._getframe(1).f_code.co_name)
+            logger.addHandler(log_handler)
+        return logger
+    except Exception as ex:
+        print(str(ex))

container/setup.py

@@ -2,45 +2,65 @@
 
 import os
 import sys
-from logging import Logger, getLogger, INFO
+import argparse
 from sys import path as syspath
 from libs.utils import *
 from libs.github import *
 from libs.codeql import *
 
- # get the parent directory of the script, to link libs
-sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
+CODEQL_HOME = get_env_variable('CODEQL_HOME')
 
-CODEQL_HOME = environ['CODEQL_HOME']
+# should we update the local copy of codeql-cli if a new version is available?
+CHECK_LATEST_CODEQL_CLI = get_env_variable('CHECK_LATEST_CODEQL_CLI', True)
 
-logger = getLogger('codeql-container')
+# should we update the local copy of codeql queries if a new version is available?
+CHECK_LATEST_QUERIES = get_env_variable('CHECK_LATEST_QUERIES', True)
+
+# if we are downloading new queries, should we precompile them 
+PRECOMPILE_QUERIES = get_env_variable('PRECOMPILE_QUERIES', True)
+
+
+logger = getLogger('codeql-container-setup')
 logger.setLevel(INFO)
 
+def parse_arguments():
+
+    parser = argparse.ArgumentParser(description='Setup codeql components.')
+    parser.add_argument("-c", "--check-latest-cli", help="check the latest codeql-cli package available and install it", 
+                        default=False, action="store_true")
+    parser.add_argument("-q", "--check-latest-queries", help="check the latest codeql queries available and install it",
+                        default=False, action="store_true")
+    #(makes query execution faster, but building the container build slower).
+    parser.add_argument("-p", "--precompile-latest-queries", help="if new queries were downloaded, precompile it",    
+                        default=False, action="store_true")
+    args = parser.parse_args()
+
+    return args
+
 def setup():
     """
     Download and install the latest codeql cli
     Download and install the latest codeql queries
     """
-
+    args = parse_arguments()
     # check version and download the latest version
-    get_latest_codeql()
-    # install vscode?
-    # clone codeql libs
-    # setup vscode + codeql
-    # wait for user
-
+    get_latest_codeql(args)
 
-def get_latest_codeql():
+def get_latest_codeql(args):
     # what version do we have?
     codeql = CodeQL(CODEQL_HOME)
     current_installed_version = codeql.get_current_local_version()
     logger.info(f'Current codeql version: {current_installed_version}')
     latest_online_version = codeql.get_latest_codeql_github_version()
-    if current_installed_version != latest_online_version.title:
+    if current_installed_version != latest_online_version.title and args.check_latest_cli:
         # we got a newer version online, download and install it
         codeql.download_and_install_latest_codeql(latest_online_version)
     # get the latest queries regardless (TODO: Optimize by storing and checking the last commit hash?)
-    codeql.download_and_install_latest_codeql_queries()
+    if args.check_latest_queries:
+        codeql.download_and_install_latest_codeql_queries()
+    if args.precompile_latest_queries:
+        codeql.precompile_queries()
 
+logger = get_logger()
 setup()
 

container/startup.py

@@ -1,18 +1,55 @@
 import os
+import sys
+from time import sleep
+from libs.utils import get_env_variable, check_output_wrapper, get_logger
+from libs.codeql import *
+
+CODEQL_HOME = get_env_variable('CODEQL_HOME')
 
 # should we update the local copy of codeql-cli if a new version is available?
-CHECK_LATEST_CODEQL_CLI = False
+CHECK_LATEST_CODEQL_CLI = get_env_variable('CHECK_LATEST_CODEQL_CLI', True)
 
 # should we update the local copy of codeql queries if a new version is available?
-CHECK_LATEST_QUERIES = False
-
+CHECK_LATEST_QUERIES = get_env_variable('CHECK_LATEST_QUERIES', True)
 
 # if we are downloading new queries, should we precompile them 
 #(makes query execution faster, but building the container build slower).
-PRECOMPILE_QUERIES = False
+PRECOMPILE_QUERIES = get_env_variable('PRECOMPILE_QUERIES', True)
+
+# ql packs, requested to run, if any
+CODEQL_CLI_ARGS = get_env_variable('CODEQL_CLI_ARGS', True)
+
+# should we just exit after execution, or should we wait for user to stop container?
+WAIT_AFTER_EXEC = get_env_variable('WAIT_AFTER_EXEC', True)
 
 def main():
-    # get all the command-line args/envs required
-    # check if the latest codeql cli need to be downloaded
-    # check if the latest codeql queries need to be downloaded
-    # check if we need to precompile the new queries
+    # do the setup, if requested
+    scripts_dir = os.path.dirname(os.path.realpath(__file__)) # get the parent directory of the script
+    setup_script_args = ''
+    if CHECK_LATEST_CODEQL_CLI:
+        setup_script_args += ' --check-latest-cli'
+    if CHECK_LATEST_QUERIES:
+        setup_script_args += ' --check-latest-queries'
+    if PRECOMPILE_QUERIES:
+        setup_script_args += ' --precompile-latest-queries'
+
+    run_result = check_output_wrapper(
+        f"{scripts_dir}/setup.py {setup_script_args}", 
+        shell=True).decode("utf-8")
+
+    # what command did the user ask to run? 
+    if CODEQL_CLI_ARGS == False or CODEQL_CLI_ARGS == None or CODEQL_CLI_ARGS == ' ':
+        # nothing to do
+        logger.info("No argument passed in for codeql-cli, nothing to do. To perform some task, please set the CODEQL_CLI_ARGS environment variable to a valid argument..")
+    else:
+        codeql = CodeQL(CODEQL_HOME)
+        run_result = codeql.execute_codeql_command(CODEQL_CLI_ARGS)
+        print(run_result)
+
+    if WAIT_AFTER_EXEC:
+        logger.info("Wait forever specified, waiting...")
+        while True:
+            sleep(10)
+
+logger = get_logger()
+main()