Outdated dependencies make metashare insecure

[zeehio]

From the dependency list found at the installation manual, I compiled a non-exhaustive requirements.txt file, which can be used to track package dependencies. This kind of file is commonly used by pip.

Then, I registered myself at https://requires.io (it's free) and I found out the dependency status for metashare. From this I realized that django-1.3 is insecure as no security patches are provided anymore to django-1.3.

Would it be possible to update metashare dependencies to prevent security issues?

[jsteffen]

The complexity of the META-SHARE data model made it necessary to apply numerous adaptations to the Django source code itself, and I think to some 3rd-party modules, too. So the META-SHARE software is basically hard-wired with the Django version available at that time (1.3). Updating Django and its dependencies to a current version would probably come with a large effort for which there are currently no resources available.

[zeehio]

I was not aware of the Django source code changes and to third party modules.

After checking it out (and just for providing more details) the only change I found from META-SHARE bundled django with respect to django 1.3.5 was a line in django/contrib/auth/models.py where a default parameter value was changed from 50 to 250.

--- /tmp/django/django/contrib/auth/models.py
+++ /tmp/META-SHARE/lib/python2.7/site-packages/django/contrib/auth/models.py
@@ -74,7 +74,8 @@

     Three basic permissions -- add, change and delete -- are automatically created for each Django model.
     """
-    name = models.CharField(_('name'), max_length=50)
+    # cfedermann: extended permission name length to 250 characters.
+    name = models.CharField(_('name'), max_length=250)
     content_type = models.ForeignKey(ContentType)
     codename = models.CharField(_('codename'), max_length=100)
     objects = PermissionManager()

If you don't mind the noise, I will try to check and document in this thread other changes in other modules, just for the sake of having a clear documentation of META-SHARE dependencies.

I completely understand the resource limitations. I do not know if I would have the time/knowledge to try and provide a patch to upgrade 1.3->1.4 by myself, but I would appreciate to know if there are enough resources to review and merge the patch in case I was able to prepare it (using for starters https://docs.djangoproject.com/en/1.5/releases/1.4/ ). I understand that even the review and merge process may need some resources (at least in terms of thorough testing), so feel free to tell me "I don't think so" if you can't review it.

PS: Just in case you are wondering "who is this guy?" I just help maintaining the metashare node at UPC. The node is currently down temporarily to provide a fix to the shellshock vulnerability.

[jsteffen]

Feel free to document any changes in this thread.

Please be aware that my knowledge about problems with updating the code is just "second hand" from old conversations with the core developers that are no longer available. But the consent was that updating would be difficult, also because the META-SHARE code probably relies on Django 1.3 specific properties that no longer hold in 1.4 or later and there are quite a number of backwards incompatible changes, too. So updating Django would definitely require code adaptations in META-SHARE. And btw, is Django 1.4 still supported? The latest version is 1.7 now.

We would of course be happy if you provide a patch to upgrade to 1.4, but apart from missing resources to review and merge the patch, we additionally just plainly lack the detailed expertise with Django/META-SHARE. I'm the last guy at DFKI with "some" knowledge about the system, but I'm no Django expert. If some else in the META-SHARE universe feels he/she is able to do it, please let us know.

[zeehio]

Well, at least it is good to know the status of the project...

Django 1.4 is a Long Term Support release (LTS) and therefore is supported until March 2015. Also, porting from 1.3 to 1.7 on a single step might be much more difficult.

Based on the django mailing list I believe on the possibility of having a another LTS after 1.4, possibly 1.8. If that is the case, all the people sticking to 1.4 will have to move to 1.8 facing the same problems and hopefully providing good and tested answers.

[zeehio]

Five months have passed and I have been making some good progress on this issue.

Things I have learned about META-SHARE

Most of META-SHARE dependencies are bundled in the lib/ directory. Most of those dependencies are outdated (newer version available) and few others are known to be insecure (with unfixed security issues). The documentation does not reflect the modifications and patches to the dependencies.

META-SHARE has a very good testsuite that covers more than 80% of the code. The testsuite is based on python, django and selenium standard tools so it is quite easy to set up. Some tests gave me issues when running them because of old selenium versions, that required old Firefox browser versions. Making changes to META-SHARE is "easy" thanks to the testsuite.

The META-SHARE documentation (available in misc/docs) is a mixture of .doc, .docx and .odt document formats. It is very hard to track changes to those documents with git, as they are binary.

Proposed roadmap

1. Convert META-SHARE documentation to a plain text based format, such as reStructuredText. This is the standard for Python documents and generates web pages like this: http://django.readthedocs.org/en/1.3.X/ . By doing this, future changes to the documentation are easily tracked using git. PDF versions of the manuals can also be generated. You can see the result here.
2. Compare the lib/ directory with the referenced upstream versions of all META-SHARE dependencies and track which dependencies have been patched and how. Add a requirements.txt file that is able to list all the dependencies in their required versions, or the git commit with the patched version. This is the standard for listing python dependencies.
3. Write a script that installs all the dependencies in a python Virtual environment. This is the standard way of creating the "lib/" directory that we currently have. The lib/ and bin/ repository could then be deleted from the repository.
4. Set up Travis Continuous Integration for continuously testing the changes to META-SHARE. Use Sauce Labs for running the web browser based tests and Coveralls to measure the code covered by the testsuite.
5. Update to the latest Django 1.3.x version, porting the required patches. Try to remove some custom patches to the META-SHARE dependencies if there is a workaround or an update available.
6. Update to the latest Django 1.4.x version. Update the other dependencies as high as possible.

If you are willing to accept the patches, I can start submitting a pull request for the documentation changes. If that pull request is merged, I will send a second pull request that deals with the rest of the points in the roadmap, except the Django 1.4 update. This second pull request will be large, as it removes the bundled dependencies. It will also update the documentation to reflect the new (and simple) installation steps.

I already have a working version of META-SHARE with Django 1.4 (supported until October 2015). I want to do further testing and make sure all the code is properly covered. I will try it with the http://metashare.talp.cat node at some point. Once all testing is done I will submit a pull request with the update.

All this depends of course on your availability to merge pull requests.

[jsteffen]

Sounds like a good plan. I guess you know what you're doing ;-)
Thanks a lot for your effort!

[zeehio]

Thanks for the support!
I will submit a pull request as soon as I have everything in order and with proper documentation.

I have already tested the changes I've been making and now metashare.talp.cat is running with django 1.4 without any issues so far!