Merge remote-tracking branch 'origin/bduffany-misc' into development

Conflicts:
	tests/home/test_home.py

Author: freskoulix

README.md

@@ -1,15 +1,130 @@
 # Codebender Selenium Tests
 
-This repo contains Selenium tests for the codebender website.
-The tests are written in Python 3.
+This repo contains Selenium tests for the codebender website.  The tests are
+written in Python 3, and utilize pytest as a testing framework and Selenium
+for browser automation.
 
 ## Running Tests
 
-To run tests locally, you'll need to be running a selenium server. See
-[here](https://selenium-python.readthedocs.org/installation.html#downloading-selenium-server)
-for instructions.
+### Dependencies
 
-Once you've got a Selenium server running, simply run `$ tox` from within the
-repo. If you don't have tox, run `$ sudo pip3 install -r requirements-dev.txt`
-from within the repo to install it.
+To run these tests, you'll need to have Python 3 installed. In addition, it is
+advantageous to have pip, a package manager for Python, in order to install
+dependencies.
+
+Notably, the pip2 (for Python 2) and pip3 (for Python 3) packages both attempt
+to link `/usr/local/bin/pip` to the `pip2` or `pip3` executable, respectively.
+To deal with this, you could explicitly type out `pip3` or `pip2` instead of
+`pip` whenever you use pip via the command line. (It may be best to just remove
+`/usr/local/bin/pip` entirely).
+
+To install `pip` in Ubuntu, run `$ sudo apt-get install python3
+python3-setuptools`, then `$ sudo easy_install3 pip`.
+
+After getting set up with pip and cloning the seleniumTests repo, you should
+make sure to install all the seleniumTests dependencies by `cd`ing to your local
+clone of the repo and running `$ sudo pip3 install -r requirements-dev.txt`.
+
+### Invoking Tests via `tox`
+
+After installing dependencies, you should have the `tox` command available. To
+run all of the tests, you can simply run `$ tox` from within the cloned repo.
+
+You can also run individual tests by providing the appropriate directory or
+filename as an argument, for example: `$ tox tests/sketch`.
+
+Invoking tox will also run `flake8`, which is essentially a lint checker for
+Python. It is best to fix any issues reported by `flake8` before committing
+to the repo. It can be run on its own via the command `$ flake8`.
+
+#### Specifying a URL for Tests
+
+Tests can either be run for the
+[bachelor](https://github.com/codebendercc/bachelor) version of the site,
+running locally, or for the live site. The version of the site that is running
+is inferred from the `--url` parameter. You can run `$ tox --url
+http://localhost` to run the tests for a locally running bachelor site (this is
+the default url), or `$ tox --url http://codebender.cc` to run the tests for the
+live site.
+
+Certain tests are specially written for one site or the other. This is
+implemented with a custom `pytest` marker. Tests that require a certain `--url`
+are decorated with `@pytest.mark.requires_url(<url>)`.
+
+### Changing Test Configuration
+
+Various global configuration parameters are specified in
+`codebender_testing/config.py`.  Such parameters include URLs and site endpoints
+which are subject to change.  This is also where the webdrivers (Firefox and
+Chrome) are specified.
+
+## Compilation Logs
+
+Certain tests exist to iterate through groups of sketches and compile them
+one-by-one.  Since these tests take a long time, they are not run in full by
+default. You can run them by specifying the `--full` option; for example: `$ tox
+tests/cb_compile_tester --full`.
+
+The following test cases are compile tests that generate such logs:
+- `tests/libraries/test_libraries.py::TestLibraryExamples`
+- `tests/compile_tester/test_compile_tester_projects.py::TestCompileTester`
+
+The generated logs are placed in the `logs` directory. They give detailed output
+in JSON format containing the codebender site URL that was used to run the
+tests, along with the URLs of the individual sketches that were compiled, and
+whether they succeeded or failed to compile.
+
+## Framework Overview
+
+The following outlines the structure of the repository as well as important
+framework components.
+
+### Directory Structure
+
+#### `tests/`
+
+The `tests/` directory contains all of the actual unit tests for the codebender
+site. That is, all of the tests discovered by `py.test` should come from this
+directory.
+
+**`tests/conftest.py`** contains the global configuration for pytest,
+including specifying the webdriver fixtures as well as the available command
+line arguments.
+
+#### `codebender_testing/`
+
+This is where all major components of the testing framework live. All of the
+unit tests rely on the files in this directory.
+
+**`codebender_testing/config.py`** specifies global configuration parameters for
+testing (see "Changing Test Configuration" above).
+
+**`codebender_testing/utils.py`** defines codebender-specific utilities used to
+test the site. These mostly consist of abstractions to the Selenium framework.
+The most important class is `SeleniumTestCase`, which all of the unit test cases
+inherit from. This grants them access (via `self`) to a number of methods and
+attributes that are useful for performing codebender-specific actions.
+
+#### `batch/`
+
+The `batch/` directory contains any executable scripts not directly used to
+perform tests. For example, it contains a script `fetch_projects.py` which can
+be used to download all of the public projects of a particular codebender user.
+
+#### `extensions/`
+
+The `extensions/` directory contains the codebender browser extensions to be
+used by the Selenium webdrivers.
+
+#### `test_data/`
+
+The `test_data/` directory contains any data used for testing. For example, it
+contains example projects that we should successfully be able to upload and
+compile.
+
+#### `logs/`
+
+The `logs/` directory contains the results of running certain tests, e.g.
+whether certain sets of sketches have compiled successfully (see "Compilation
+Logs").
 

codebender_testing/config.py

@@ -38,8 +38,8 @@ def _rel_path(*args):
 # Directory in which the local compile tester files are stored.
 COMPILE_TESTER_DIR = os.path.join(TEST_DATA_DIR, 'cb_compile_tester')
 
-# Set up Selenium Webdrivers to be used for selenium tests
 
+# Set up Selenium Webdrivers to be used for selenium tests
 def _get_firefox_profile():
     """Returns the Firefox profile to be used for the FF webdriver.
     Specifically, we're equipping the webdriver with the Codebender
@@ -51,8 +51,15 @@ def _get_firefox_profile():
     )
     return firefox_profile
 
+# Webdrivers to be used for testing. Specifying additional webdrivers here
+# will cause every test to be re-run using that webdriver.
+# These webdrivers are specified as lambdas to allow for "lazy" evaluation.
+# The lambda invocation will return the actual webdriver and open up a
+# browser window, and we don't want a browser window to open whenever this
+# module is imported (hence the need for lazy evaluation).
 WEBDRIVERS = {
-    "firefox": webdriver.Firefox(firefox_profile=_get_firefox_profile())
+    "firefox": lambda: webdriver.Firefox(firefox_profile=_get_firefox_profile()),
+    # "chrome": lambda: webdriver.Chrome()
 }
 
 # Credentials to use when logging into the site via selenium

codebender_testing/utils.py

@@ -4,6 +4,7 @@
 import json
 import os
 import re
+import shutil
 import tempfile
 
 from selenium.common.exceptions import NoSuchElementException
@@ -77,9 +78,8 @@ def temp_copy(fname):
     """
     extension = fname.split('.')[-1]
     with tempfile.NamedTemporaryFile(mode='w+b', suffix='.%s' % extension) as copy:
-        with open(fname, 'r') as original:
-            for line in original:
-                copy.write(line)
+        with open(fname, 'rb') as original:
+            shutil.copyfileobj(original, copy)
         copy.flush()
         yield copy
 
@@ -101,7 +101,7 @@ def start(url=None, webdriver=None):
         """
         if webdriver is None:
             webdriver = WEBDRIVERS.keys()[0]
-        self.driver = WEBDRIVERS[webdriver]
+        self.driver = WEBDRIVERS[webdriver]()
 
         if url is None:
             url = BASE_URL
@@ -203,6 +203,15 @@ def login(self):
             # 'Log In' is not displayed, so we're already logged in.
             pass
 
+    def logout(self):
+        """Logs out of the site."""
+        try:
+            logout_button = self.driver.find_element_by_id("logout")
+            logout_button.send_keys(Keys.ENTER)
+        except NoSuchElementException:
+            # 'Log out' is not displayed, so we're already logged out.
+            pass
+
     def get_element(self, *locator):
         """Waits for an element specified by *locator (a tuple of
         (By.<something>, str)), then returns it if it is found."""
@@ -217,11 +226,11 @@ def get_elements(self, *locator):
             expected_conditions.visibility_of_all_elements_located_by(locator))
         return self.driver.find_elements(*locator)
 
-    def get(self, selector):
+    def find(self, selector):
         """Alias for `self.get_element(By.CSS_SELECTOR, selector)`."""
         return self.get_element(By.CSS_SELECTOR, selector)
 
-    def get_all(self, selector):
+    def find_all(self, selector):
         """Alias for `self.get_elements(By.CSS_SELECTOR, selector)`."""
         return self.get_elements(By.CSS_SELECTOR, selector)
 
@@ -285,9 +294,13 @@ def compile_sketches(self, sketches, iframe=False, logfile=None):
         format string, which will be formatted appropriately.
         `iframe` specifies whether the urls pointed to by `selector` are contained
         within an iframe.
+        If the `--full` argument is provided (and hence
+        `self.run_full_compile_tests` is `True`, we do not log, and limit the
+        number of sketches compiled to 1.
         """
-        if logfile is None:
-            for sketch in sketches:
+        sketch_limit = None if self.run_full_compile_tests else 1
+        if logfile is None or not self.run_full_compile_tests:
+            for sketch in sketches[:sketch_limit]:
                 self.compile_sketch(sketch, iframe=iframe)
         else:
             log_entry = {'url': self.site_url, 'succeeded': [], 'failed': []}
@@ -320,19 +333,24 @@ class SeleniumTestCase(CodebenderSeleniumBot):
 
     @classmethod
     @pytest.fixture(scope="class", autouse=True)
-    def _testcase_attrs(cls, webdriver, testing_url):
+    def _testcase_attrs(cls, webdriver, testing_url, testing_full):
         """Sets up any class attributes to be used by any SeleniumTestCase.
         Here, we just store fixtures as class attributes. This allows us to avoid
         the pytest boilerplate of getting a fixture value, and instead just
         refer to the fixture as `self.<fixture>`.
         """
         cls.driver = webdriver
         cls.site_url = testing_url
+        cls.run_full_compile_tests = testing_full
 
     @pytest.fixture(scope="class")
     def tester_login(self):
         self.login()
 
+    @pytest.fixture(scope="class")
+    def tester_logout(self):
+        self.logout()
+
 
 class VerificationError(Exception):
     """An exception representing a failed verification of a sketch."""

tests/compile_tester/test_compile_tester_projects.py

@@ -28,8 +28,10 @@ def test_compile_local_files(self, tester_login):
         """Tests that we can upload all of cb_compile_tester's projects
         (stored locally in test_data/cb_compile_tester), compile them,
         and finally delete them."""
+        upload_limit = None if self.run_full_compile_tests else 1
+
         test_files = [os.path.join(COMPILE_TESTER_DIR, name)
-            for name in next(os.walk(COMPILE_TESTER_DIR))[2]]
+            for name in next(os.walk(COMPILE_TESTER_DIR))[2]][:upload_limit]
         projects = [self.upload_project(fname) for fname in test_files]
         project_names, project_urls = zip(*projects)
 

tests/conftest.py

@@ -1,3 +1,11 @@
+"""The configuration file for `py.test`.
+
+This file specifies global test fixtures, which include the selenium
+webdrivers.
+
+This is also where command-line arguments are defined.
+"""
+
 import pytest
 
 from codebender_testing.config import BASE_URL
@@ -10,24 +18,34 @@ def webdriver(request):
     and registers a finalizer to close the browser once the session is
     complete. The entire test session is repeated once per driver.
     """
-    driver = WEBDRIVERS[request.param]
+    driver = WEBDRIVERS[request.param]()
     request.addfinalizer(lambda: driver.quit())
     return driver
 
 def pytest_addoption(parser):
     """Adds command line options to py.test."""
     parser.addoption("--url", action="store", default=BASE_URL,
                      help="URL to use for testing, e.g. http://localhost, http://codebender.cc")
+    parser.addoption("--full", action="store_true", default=False,
+                     help="Whether to run the complete set of compile tests.")
 
 @pytest.fixture(scope="class")
 def testing_url(request):
+    """A fixture to get the --url parameter."""
     return request.config.getoption("--url")
 
+@pytest.fixture(scope="class")
+def testing_full(request):
+    """A fixture to get the --full parameter."""
+    return request.config.getoption("--full")
+
 @pytest.fixture(autouse=True)
 def skip_by_site(request, testing_url):
     """Skips tests that require a certain site URL in order to run properly."""
     if request.node.get_marker('requires_url'):
         required_url = request.node.get_marker('requires_url').args[0]
         if required_url.rstrip('/') != testing_url.rstrip('/'):
-            pytest.skip('skipped test that requires --url=%s')
+            pytest.skip('skipped test that requires --url=%s' % required_url)
+
+# TODO: add a --full option to run the complete test suite.