Import existing surveys

[stimon]

Hi!

Is there an import mechanism for existing surveys/projects?
We need different people to be able to review the recognition and it is not possible for us to launch the desktop GUI in our environment, so I'm thinking sdaps-web would be a nice solution.

Thanks

[benzea]

Hmmm, well, it isn't supported currently.

If you manage to set up the webinterface, then I think you have two choices:

1. Rebuild the exact questionnaire, and hope that it compiles to the exact same document. I doubt that this can work.
2. Just create a random survey and run it through the steps until it is ready. At that point, you have a directory on disk for the survey. Simply replacing that directory should make things works.

[benzea]

Feel free to drop by IRC/matrix and we might be able to help. I can't promise fast responses right now though unfortunately. And, I also don't know much about the current code base of the web module (I have not gotten demo.sdaps.org up and running again, but that was because I got stuck on avoiding docker :-/).

[stimon]

Thank you @benzea
Sorry for the delayed response, I didn't have the chance to check it until now.

Approach 2 seems to trick it into thinking that the survey is set up and ready, but things don't work correctly. I'm not sure how much is because of the workaround or because of actual bugs, I'm not familiar with django. Important to note that I'm trying the docker-compose setup, as I would have to use a container approach in our production server.

After seeing that sdaps-web consider the survey ready, I went on to add scans, but I get an Internal Server Error after hitting start:

web_1       | [25/May/2022 10:11:34] "GET /surveys/BDAEDX/ HTTP/1.1" 200 8360
web_1       | [25/May/2022 10:11:44] "GET /surveys/BDAEDX/add_scans/ HTTP/1.1" 200 9776
web_1       | [25/May/2022 10:11:44] "GET /static/sdaps_ctl/img/loading.gif HTTP/1.1" 404 1699
web_1       | [25/May/2022 10:11:44] "GET /surveys/BDAEDX/upload/ HTTP/1.1" 200 13
web_1       | Internal Server Error: /surveys/BDAEDX/upload/
web_1       | Traceback (most recent call last):
web_1       |   File "/home/app/.local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
web_1       |     response = get_response(request)
web_1       |   File "/home/app/.local/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
web_1       |     response = self.process_exception_by_middleware(e, request)
web_1       |   File "/home/app/.local/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
web_1       |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
web_1       |   File "/home/app/.local/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
web_1       |     return self.dispatch(request, *args, **kwargs)
web_1       |   File "/home/app/.local/lib/python3.8/site-packages/guardian/mixins.py", line 206, in dispatch
web_1       |     return super().dispatch(request, *args, **kwargs)
web_1       |   File "/home/app/.local/lib/python3.8/site-packages/django/views/generic/base.py", line 97, in dispatch
web_1       |     return handler(request, *args, **kwargs)
web_1       |   File "/project/sdaps_ctl/views.py", line 444, in post
web_1       |     match = self.content_range_pattern.match(content_range)
web_1       | AttributeError: 'SurveyUploadScansFiles' object has no attribute 'content_range_pattern'
web_1       | [25/May/2022 10:11:55] "POST /surveys/BDAEDX/upload/ HTTP/1.1" 500 14659
rabbitmq_1  |

Maybe this is another issue, and I'm forcing sdaps-web out of its current dev state.

[benzea]

Hmm, I think that would be fixed by the following patch:

diff --git a/sdaps_ctl/views.py b/sdaps_ctl/views.py
index 3d0107a..289d00b 100644
--- a/sdaps_ctl/views.py
+++ b/sdaps_ctl/views.py
@@ -383,7 +383,6 @@ class SurveyAddScans(PermissionRequiredMixin, generic.edit.UpdateView):
     (SurveyUploadScansFiles) and adding them to the survey for recognition (this View).
     """
     permission_required = 'can_upload_scans'
-    content_range_pattern = re.compile(r'^bytes (?P<start>\d+)-(?P<end>\d+)/(?P<size>\d+)')
 
     def get_object(self):
         return get_object_or_404(models.Survey, slug=self.kwargs['slug'], initialized=True)
@@ -419,6 +418,7 @@ class SurveyUploadScansFiles(PermissionRequiredMixin, generic.edit.UpdateView):
     View for handling single scan files before adding them to survey for recognition.
     """
     permission_required = 'can_upload_scans'
+    content_range_pattern = re.compile(r'^bytes (?P<start>\d+)-(?P<end>\d+)/(?P<size>\d+)')
 
     def get_object(self):
         return get_object_or_404(models.Survey, slug=self.kwargs['slug'], initialized=True)

i.e. move content_range_pattern from SurveyAddScans to SurveyUploadScansFiles where it is actually used in the post function.

[benzea]

Note that this will only happen if the browser is sending a HTTP_CONTENT_RANGE. So, it could well be that simple testing never hit this code path after refactorings happened.

[stimon]

Nice! That worked.

However, I get the message

File already uploaded or in an error state. Please, rename the file and upload again.

The only thing I see is this in the celery logs:

[2022-05-25 13:35:53,317: WARNING/ForkPoolWorker-7] No files to add
[2022-05-25 13:35:53,352: WARNING/ForkPoolWorker-8] 0 sheets
[2022-05-25 13:35:53,432: ERROR/ForkPoolWorker-8] Task sdaps_ctl.tasks.recognize_scan[864204db-356c-48e8-a771-322f4b40b396] raised unexpected: OperationalError('attempt to write a readonly database')
Traceback (most recent call last):
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/tasks.py", line 143, in recognize_scan
    recognize(survey, filter)
  File "/usr/lib/python3/dist-packages/sdaps/recognize/__init__.py", line 33, in recognize
    survey.save()
  File "/usr/lib/python3/dist-packages/sdaps/model/survey.py", line 305, in save
    c.execute('INSERT OR REPLACE INTO surveys (rowid, json) VALUES (?, ?)', (self._survey_rowid, json.dumps(self, default=db.toJson)))
sqlite3.OperationalError: attempt to write a readonly database

But I see the tiff hanging under proj/1/uploads.
If I go to review it is empty, not surprisingly.

PS: The scan should work, I have tested it in CLI before trying sdaps-web out.

[benzea]

Stupid question. But could it be that the web server does not have permissions to write the database file? i.e. the survey.sqlite file?

[stimon]

I also thought of that, but I checked the file owner and permissions, and seems to be the same as in any other project created via web... :(

[benzea]

I also thought of that, but I checked the file owner and permissions, and seems to be the same as in any other project created via web... :(

And the directory is fine too, right? I assume that would be the case as you are able scans into the directory already. Just thinking sqlite needs temporary files for locking and journalling.

[stimon]

Unless it needs write permissions for group I would think it is fine

$ ls -lh proj/
total 0
drwxr-xr-x  40 myuser  admin   1.3K May 25 19:24 2
drwxr-xr-x   5 myuser  admin   160B May 27 10:55 deleted

[benzea]

Really, there must be some sort of access control issue. Either for the survey.sqlite file or the directory.

As what user is the celery stuff running? Does that user really has access? Can you run e.g. sdaps info PROJECT as the same user and it succeeds (currently even that simple command writes the DB, even though it is not needed).

[stimon]

Well, I thought of starting fresh and resetting the whole thing... somehow I broke it :(
Now when I create a survey and get into the edit step, sdaps is not creating the questionnaire.pdf

Attaching to sdaps-web_web_1
web_1       | Watching for file changes with StatReloader
web_1       | [30/May/2022 08:37:57] "GET / HTTP/1.1" 302 0
web_1       | [30/May/2022 08:37:57] "GET /surveys HTTP/1.1" 301 0
web_1       | [30/May/2022 08:37:57] "GET /surveys/ HTTP/1.1" 302 0
web_1       | [30/May/2022 08:37:57] "GET /accounts/login/?next=/surveys/ HTTP/1.1" 200 4166
web_1       | [30/May/2022 08:38:16] "POST /accounts/login/ HTTP/1.1" 302 0
web_1       | [30/May/2022 08:38:16] "GET /surveys/ HTTP/1.1" 200 3171
web_1       | [30/May/2022 08:38:26] "GET /surveys/create/ HTTP/1.1" 200 5622
web_1       | [30/May/2022 08:38:37] "POST /surveys/create/ HTTP/1.1" 302 0
web_1       | [30/May/2022 08:38:37] "GET /surveys HTTP/1.1" 301 0
web_1       | [30/May/2022 08:38:37] "GET /surveys/ HTTP/1.1" 200 3610
web_1       | [30/May/2022 08:38:40] "GET /surveys/MMOZ5Z/ HTTP/1.1" 200 7929
web_1       | [30/May/2022 08:38:42] "GET /surveys/MMOZ5Z/edit/ HTTP/1.1" 200 27022
web_1       | Not Found: /surveys/MMOZ5Z/questionnaire.pdf
web_1       | [30/May/2022 08:38:42] "GET /surveys/MMOZ5Z/questionnaire.pdf HTTP/1.1" 404 1759
web_1       | [30/May/2022 08:38:42] "GET /surveys/MMOZ5Z/edit/questionnaire/ HTTP/1.1" 200 0
web_1       | [30/May/2022 08:38:43] "POST /surveys/MMOZ5Z/edit/questionnaire/ HTTP/1.1" 202 0
web_1       | Not Found: /surveys/MMOZ5Z/questionnaire.pdf
web_1       | [30/May/2022 08:38:43] "GET /surveys/MMOZ5Z/questionnaire.pdf HTTP/1.1" 404 1759
web_1       | Not Found: /surveys/MMOZ5Z/questionnaire.pdf
web_1       | [30/May/2022 08:38:44] "GET /surveys/MMOZ5Z/questionnaire.pdf HTTP/1.1" 404 1759
web_1       | Not Found: /surveys/MMOZ5Z/questionnaire.pdf
web_1       | [30/May/2022 08:38:44] "GET /surveys/MMOZ5Z/questionnaire.pdf HTTP/1.1" 404 1759

I headed to celery, and it looks like it is not receiving the JSON it expects:

$ easy/exec cat /tmp/logs/celery.log
 
 -------------- celery@cc868df1297b v4.2.1 (windowlicker)
---- **** ----- 
--- * ***  * -- Linux-5.10.76-linuxkit-x86_64-with-glibc2.29 2022-05-30 08:37:53
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         sdaps_web:0x7f6a86c03610
- ** ---------- .> transport:   amqp://guest:**@rabbitmq:5672//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: ON
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[2022-05-30 08:37:53,606: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds...

[2022-05-30 08:37:55,613: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq:5672//: [Errno 111] Connection refused.
Trying again in 4.00 seconds...

[2022-05-30 08:38:00,669: WARNING/MainProcess] /home/app/.local/lib/python3.8/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2022-05-30 08:38:37,357: DEBUG/ForkPoolWorker-8] sdaps_ctl.tasks.write_questionnaire[efb6e4aa-7681-4ac7-b339-74b76a0d8ecf]: Starting Task: WRITE_QUESTIONNAIRE
[2022-05-30 08:38:37,361: DEBUG/ForkPoolWorker-1] sdaps_ctl.tasks.render_questionnaire[7c5bd576-15ad-4f64-8502-fe978dde40ed]: Render questionnaire (with slug MMOZ5Z)
[2022-05-30 08:38:37,361: DEBUG/ForkPoolWorker-1] sdaps_ctl.tasks.render_questionnaire[7c5bd576-15ad-4f64-8502-fe978dde40ed]: Rendering path "/tmp/projects/1")
[2022-05-30 08:38:37,363: WARNING/ForkPoolWorker-1] Running pdflatex now three times to generate the questionnaire.
[2022-05-30 08:38:37,363: WARNING/ForkPoolWorker-1] Executing: pdflatex -halt-on-error -output-directory /tmp/sdaps-web-k03o75se -interaction batchmode questionnaire.tex /tmp/projects/1
[2022-05-30 08:38:37,363: DEBUG/ForkPoolWorker-8] sdaps_ctl.tasks.write_questionnaire[efb6e4aa-7681-4ac7-b339-74b76a0d8ecf]: Write questionnaire (with slug MMOZ5Z)
[2022-05-30 08:38:37,372: ERROR/ForkPoolWorker-8] Task sdaps_ctl.tasks.write_questionnaire[efb6e4aa-7681-4ac7-b339-74b76a0d8ecf] raised unexpected: TypeError('the JSON object must be str, bytes or bytearray, not list')
Traceback (most recent call last):
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/tasks.py", line 201, in write_questionnaire
    texwriter(djsurvey)
  File "/project/sdaps_ctl/texwriter.py", line 163, in texwriter
    questionnaire = json.loads(djsurvey.questionnaire)
  File "/usr/lib/python3.8/json/__init__.py", line 341, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
entering extended mode
[2022-05-30 08:38:37,400: WARNING/ForkPoolWorker-1] Executing: pdflatex -halt-on-error -output-directory /tmp/sdaps-web-k03o75se -interaction batchmode questionnaire.tex /tmp/projects/1
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
entering extended mode
[2022-05-30 08:38:37,429: WARNING/ForkPoolWorker-1] Executing: pdflatex -halt-on-error -output-directory /tmp/sdaps-web-k03o75se -interaction batchmode questionnaire.tex /tmp/projects/1
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
entering extended mode
[2022-05-30 08:38:37,488: ERROR/ForkPoolWorker-1] Task sdaps_ctl.tasks.render_questionnaire[7c5bd576-15ad-4f64-8502-fe978dde40ed] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/lib/python3.8/shutil.py", line 791, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/sdaps-web-k03o75se/questionnaire.log' -> '/tmp/projects/1/questionnaire.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/tasks.py", line 227, in render_questionnaire
    if utils.atomic_latex_compile(djsurvey.path, 'questionnaire.tex'):
  File "/home/app/.local/lib/python3.8/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 642, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/task.py", line 375, in __call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/utils.py", line 66, in atomic_latex_compile
    shutil.move(os.path.join(tmpdir, output_log), os.path.join(path, output_log))
  File "/usr/lib/python3.8/shutil.py", line 811, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.8/shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/sdaps-web-k03o75se/questionnaire.log'
[2022-05-30 08:38:43,324: DEBUG/ForkPoolWorker-3] sdaps_ctl.tasks.write_questionnaire[14b3f9dc-c7b2-40c4-9cbd-95b093b1d808]: Starting Task: WRITE_QUESTIONNAIRE
[2022-05-30 08:38:43,326: DEBUG/ForkPoolWorker-4] sdaps_ctl.tasks.render_questionnaire[1a6b7431-5556-452c-a0cf-fbd29dd2b57f]: Render questionnaire (with slug MMOZ5Z)
[2022-05-30 08:38:43,327: DEBUG/ForkPoolWorker-4] sdaps_ctl.tasks.render_questionnaire[1a6b7431-5556-452c-a0cf-fbd29dd2b57f]: Rendering path "/tmp/projects/1")
[2022-05-30 08:38:43,328: WARNING/ForkPoolWorker-4] Running pdflatex now three times to generate the questionnaire.
[2022-05-30 08:38:43,328: WARNING/ForkPoolWorker-4] Executing: pdflatex -halt-on-error -output-directory /tmp/sdaps-web-65t82k5c -interaction batchmode questionnaire.tex /tmp/projects/1
[2022-05-30 08:38:43,328: DEBUG/ForkPoolWorker-3] sdaps_ctl.tasks.write_questionnaire[14b3f9dc-c7b2-40c4-9cbd-95b093b1d808]: Write questionnaire (with slug MMOZ5Z)
[2022-05-30 08:38:43,334: ERROR/ForkPoolWorker-3] Task sdaps_ctl.tasks.write_questionnaire[14b3f9dc-c7b2-40c4-9cbd-95b093b1d808] raised unexpected: JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
Traceback (most recent call last):
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/tasks.py", line 201, in write_questionnaire
    texwriter(djsurvey)
  File "/project/sdaps_ctl/texwriter.py", line 163, in texwriter
    questionnaire = json.loads(djsurvey.questionnaire)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
entering extended mode
[2022-05-30 08:38:43,359: WARNING/ForkPoolWorker-4] Executing: pdflatex -halt-on-error -output-directory /tmp/sdaps-web-65t82k5c -interaction batchmode questionnaire.tex /tmp/projects/1
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
entering extended mode
[2022-05-30 08:38:43,386: WARNING/ForkPoolWorker-4] Executing: pdflatex -halt-on-error -output-directory /tmp/sdaps-web-65t82k5c -interaction batchmode questionnaire.tex /tmp/projects/1
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
entering extended mode
[2022-05-30 08:38:43,421: ERROR/ForkPoolWorker-4] Task sdaps_ctl.tasks.render_questionnaire[1a6b7431-5556-452c-a0cf-fbd29dd2b57f] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/lib/python3.8/shutil.py", line 791, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/sdaps-web-65t82k5c/questionnaire.log' -> '/tmp/projects/1/questionnaire.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/tasks.py", line 227, in render_questionnaire
    if utils.atomic_latex_compile(djsurvey.path, 'questionnaire.tex'):
  File "/home/app/.local/lib/python3.8/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 642, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/task.py", line 375, in __call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/utils.py", line 66, in atomic_latex_compile
    shutil.move(os.path.join(tmpdir, output_log), os.path.join(path, output_log))
  File "/usr/lib/python3.8/shutil.py", line 811, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.8/shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/sdaps-web-65t82k5c/questionnaire.log'

[benzea]

OSError: [Errno 18] Invalid cross-device link: '/tmp/sdaps-web-k03o75se/questionnaire.log' -> '/tmp/projects/1/questionnaire.log'

Wait, that is actually a bug! We can't move a file from /tmp into the project directory, it needs to be copied. I thought that shutil would automatically detect that error and do a copy+unlink instead. Maybe this is only happening because the log has not been generated though. So bug, but not the real issue.

Traceback (most recent call last):
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/app/.local/lib/python3.8/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/project/sdaps_ctl/tasks.py", line 201, in write_questionnaire
    texwriter(djsurvey)
  File "/project/sdaps_ctl/texwriter.py", line 163, in texwriter
    questionnaire = json.loads(djsurvey.questionnaire)
  File "/usr/lib/python3.8/json/__init__.py", line 341, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list

That seems rather odd. I think the storage is messed up completely. i.e. we have a json field, but then just store a string into it which again contains json. sigh

Thinking something along the lines of:

diff --git a/sdaps_ctl/models.py b/sdaps_ctl/models.py
index 6ae2ce5..86f9213 100644
--- a/sdaps_ctl/models.py
+++ b/sdaps_ctl/models.py
@@ -75,7 +75,7 @@ class Survey(models.Model):
     created_at = models.DateTimeField(auto_now_add=True)
     updated_at = models.DateTimeField(auto_now=True)
 
-    questionnaire = JSONField(default = list)
+    questionnaire = JSONField(default = '[]')
 
     owner = models.ManyToManyField(
             User,
diff --git a/sdaps_ctl/texwriter.py b/sdaps_ctl/texwriter.py
index 0a464e0..a1585c3 100644
--- a/sdaps_ctl/texwriter.py
+++ b/sdaps_ctl/texwriter.py
@@ -17,8 +17,6 @@
 
 import os
 
-import json
-
 def get(d, attr, val):
     try:
         return d[attr]
@@ -160,9 +158,7 @@ def texwriter(djsurvey):
 
     content = []
 
-    questionnaire = json.loads(djsurvey.questionnaire)
-
-    for qobject in questionnaire:
+    for qobject in djsurvey.questionnaire:
         content.append(render_qobject(qobject))
 
     data['content'] = '\n'.join(content)
diff --git a/sdaps_ctl/views.py b/sdaps_ctl/views.py
index 3d0107a..454fa28 100644
--- a/sdaps_ctl/views.py
+++ b/sdaps_ctl/views.py
@@ -250,7 +250,7 @@ def questionnaire(request, slug):
         if survey.initialized:
             raise Http404
         else:
-            survey.questionnaire = request.body.decode("utf-8")
+            survey.questionnaire = json.loads(request.body.decode("utf-8"))
             survey.save()
 
             survey_id = str(survey.id)
@@ -259,7 +259,7 @@ def questionnaire(request, slug):
             return HttpResponse(status=202)
 
     elif request.method == 'GET':
-        response = HttpResponse(survey.questionnaire, content_type="application/json")
+        response = HttpResponse(json.dumps(survey.questionnaire), content_type="application/json")
         response['Content-Disposition'] = ('attachment; filename="%s_questionnaire.json"' % slug)
 
         return response
@@ -383,7 +383,6 @@ class SurveyAddScans(PermissionRequiredMixin, generic.edit.UpdateView):
     (SurveyUploadScansFiles) and adding them to the survey for recognition (this View).
     """
     permission_required = 'can_upload_scans'
-    content_range_pattern = re.compile(r'^bytes (?P<start>\d+)-(?P<end>\d+)/(?P<size>\d+)')
 
     def get_object(self):
         return get_object_or_404(models.Survey, slug=self.kwargs['slug'], initialized=True)
@@ -419,6 +418,7 @@ class SurveyUploadScansFiles(PermissionRequiredMixin, generic.edit.UpdateView):
     View for handling single scan files before adding them to survey for recognition.
     """
     permission_required = 'can_upload_scans'
+    content_range_pattern = re.compile(r'^bytes (?P<start>\d+)-(?P<end>\d+)/(?P<size>\d+)')
 
     def get_object(self):
         return get_object_or_404(models.Survey, slug=self.kwargs['slug'], initialized=True)

But, it'll also need database migration and stuff to be committable.

[benzea]

I suspect you can work around the bug simply be editing the questionnaire. i.e. just adding a question before exiting the editor.

[benzea]

Sorry for the late response :-/

Feel free to drop by IRC/matrix, or ping again. I do forget these things.