Ability to pass formatting arguments to JSON backends

[timbernat]

Feature description

It would be nice if one could natively specify how JSONCollection instances format fields to on JSON dump, e.g. assigning indent, ensure_ascii, separators, etc arguments as found in the json.dump signature

Proposed solution

As a loose implementation, one might unpack the **kwargs passed to JSONCollection.init and isolate those which match the call signature of, e.g. json.dumps, which is only called once AFAIS in the JSON backends implementations. This is the most "brutal", naive way to acheive the desired behavior, and more elegant countersolutions are most welcome given developer input about interactions within synced_collections that I'm unaware of.

Additional context

Such control would be useful for example in Signac Projects, where the one-line job statepoint and document JSON files often become unwieldy and poorly human-readable when inspecting individual jobs. For current projects, I've relied on Project-level hooks to set the indentation on each JSON file write, but it seems like there ought to be an easier way to configure formatting which is natively supported by the stdlib json module (no "crazy" custom formatting being requested here).

[cbkerr]

Thanks for the detailed writeup!

I thought adding formatting to the state point written to disk would break the consistency between the state point file and the job id, but it's already not consistent! At first I thought that it must be the same code path for hashing the state point to compute the job id as for writing the state point to disk, but the state point written to disk isn't sorted.

Finding that, I'm not as opposed to adding the additional formatting options as I initially thought. I'm curious how your hook is working out in terms of day to day usage?

To solve the problem of making the state point on disk more readable, does your editor support json formatting when viewing the file? I had the same problem, and I use the emacs json-pretty-print-buffer function.

[timbernat]

break the consistency between the state point file and the job id, but it's already not consistent!

Could you elaborate on what you mean by this? I didn't realize there was a possibility of the formatting of the statepoint changing the hash; I haven't seen it happen when I reformat in practice

To solve the problem of making the state point on disk more readable, does your editor support json formatting when viewing the file?

I use VSCode for most things; quick search came up with this "beautify JSON" extension, which I can confirm gives me nice, readable indentation, albeit with a couple keystrokes. Given I only need this in cases where I'm manually intervening anyway, I think that's a good-enough solution

I'm curious how your hook is working out in terms of day to day usage?

It does what I describe here, re-indenting the JSON after a successful exit of each operation. It behaves the way I would expect and I haven't run into any issues with the job hashes shifting under my feet because of it. To slake your curiosity fully, the hooks I'm using (which also cache runtime of each operation) are defined exactly as such:

from typing import Callable, Optional
import time

import json
from pathlib import Path

from signac.job import Job
from signac.project import Project

# JSON FORMATTING
def reindent_job_json_factory(filename : str) -> Callable[[Job, Optional[int]], None]:
    '''Build a function which accepts a Job and indents the file'''
    if not filename.endswith('.json'): # TODO - add support/handling for Path objects
        raise ValueError('Filename must point to file with .json extension')
    
    def reindent_job_json(job : Job, indent : Optional[int]=4) -> None:
        # prechecks
        if (indent is not None) and indent < 0:
            raise ValueError(f'Must provide positive indent amount (not {indent})')
        
        if not job.isfile(filename):
            raise FileNotFoundError(job.fn(filename)) # TODO - find way to log this failure
            
        # read/re-write
        with open(job.fn(filename), 'r') as file:
            data = json.load(file)

        with open(job.fn(filename), 'w') as file:
            json.dump(data, file, indent=indent)
    
    return reindent_job_json

reindent_json_statepoint : Callable[[Job, Optional[int]], None] = reindent_job_json_factory(Job.FN_STATE_POINT)
reindent_json_document   : Callable[[Job, Optional[int]], None] = reindent_job_json_factory(Job.FN_DOCUMENT)

# JOB HOOKS
class ProjectHooks:
    def __init__(
        self,
        operation_times_attr : str='operation_times_sec',
        indent_amount : Optional[int]=4,
    ) -> None:
        self.operation_times_attr = operation_times_attr
        self.indent_amount = indent_amount
        
    # OPERATION TIMING
    def operation_start_time_hook(self) -> Callable[[str, Job], None]:
        '''Define a hook which records operation start times'''
        def record_operation_start_time(operation_name : str, job : Job) -> None:
            '''Record into the job document when a particular operation began'''
            job.doc.setdefault(self.operation_times_attr, {})
            job.doc[self.operation_times_attr].update({f'{operation_name}_start' : time.time()})
        return record_operation_start_time
        
    def operation_duration_hook(self) -> Callable[[str, Job], None]:
        '''Define a hook which records operation completion times, assuming that a start time has been recorded'''
        def record_operation_duration(operation_name : str, job : Job) -> None:
            '''Record into the job document how long a particular operation took'''
            op_times = job.doc.get(self.operation_times_attr, {})
            start_time = op_times.pop(f'{operation_name}_start') # look up and withdraw start time
            if start_time is None: # NOTE: dicts in Signac documents are NOT pure Python dicts; their "pop()" method returns None by default and never raises KeyError
                raise ValueError(f'No start time recorded for operation "{operation_name}"; cannot calculate operation duration')
            
            job.doc[self.operation_times_attr].update({operation_name : (time.time() - start_time)})
        return record_operation_duration
    
    # JSON FORMATTING
    def operation_indent_statepoint_hook(self)-> Callable[[str, Job], None]:
        def indent_statepoint(operation_name : str, job : Job) -> None:
            '''Apply indentation to Job statepoint output once an operation completes'''
            reindent_json_statepoint(job, indent=self.indent_amount)
        return indent_statepoint

    def operation_indent_document_hook(self)-> Callable[[str, Job], None]:
        def indent_document(operation_name : str, job : Job) -> None:
            '''Apply indentation to Job document output once an operation completes'''
            reindent_json_document(job, indent=self.indent_amount)
        return indent_document
    
    # HOOK INSTALLATION
    def install_hooks(self, project : Project) -> Project:
        '''Inject hooks defined at the project level'''
        project.project_hooks.on_start.extend([
            self.operation_start_time_hook(),
        ])
        project.project_hooks.on_exit.extend([
            self.operation_duration_hook(), # for now, record times for failing and succeeding operations the same
            self.operation_indent_document_hook(),
            self.operation_indent_statepoint_hook(),
        ]) 
        # project.project_hooks.on_success = ...
        # project.project_hooks.on_exception = ... # TODO: add distinct timing mode for cases where operations fail before exit

        return project