diff --git a/README.md b/README.md index e6505aed..78f64809 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ - +example # ISS Data Collection Tool @@ -30,6 +30,7 @@ See [docs/](/docs) for manuals and walkthroughs: - [Goals](/docs/goals.md) - [Statistics](/docs/statistics.md) - [Downloads](/docs/downloads.md) +- [Annotation](/docs/annotation.md) ## ⚙️ Architecture diff --git a/docs/annotation.md b/docs/annotation.md new file mode 100644 index 00000000..d0144059 --- /dev/null +++ b/docs/annotation.md @@ -0,0 +1,63 @@ +# 🧾 Annotation +When exporting your dataset—or just annotations, a structured JSON file is generated. + +It contains metadata for all files matching the current filters and their annotations—without media. + +## 🧬 Example Structure + +```json +[ + { + "id": "6848a6709aeaf5eb62cf01a8", + "related_duplicates": 0, + "attributes": [ + { + "uid": "f35846ca-0c7e-4120-8225-9204b054bb1a", + "attributes": [ + [16, 0, "type_1", ""], + [18, 0, "subtype_2", ""], + [20, 1, "r", ""] + ] + } + ], + "author_name": "Curator", + "validator_name": "Curator", + "file_name": "6788191feac1f1a81ab2fadd.webp", + "file_type": "image", + "status": "a", + "upload_date": "2025-06-10 21:41", + "update_date": "2025-06-10 21:41", + "rebound_project": null, + "rebound": null + } +] +``` + +Where: +- **id** - is generated file id by which the file could be found in the system +- **related_duplicates** - number of duplicates associated to this file +- **attributes** - list of attribute (label) groups with annotated data +- **author_name** - the username of user who uploaded this file +- **validator_name** - the username of user who validated this file +- **file_name** - original file name +- **file_type** - media type (image/video) +- **status** - validation status, `a - accepted`, `d - declined`, `v - waits for validation` +- **upload_date** - file upload date +- **update_date** - file update date +- **rebound_project** - project / bucket where the file is located in case of file transfer +- **rebound** - file project / bucket (in terms of storage system) + +## Accessing a file + +Since the project in **wip** state there are some temprorary solutions. +We have a situation when we want to move some files to other project. +Since proper object movement is not implemented yet we could use a trick to just associate +file project to the new one but since file is stored in original bucket we have to specify it. + +Using orm / db model you might encounter **rebound** field as well. +This one is proper and intentional and used for rebounding file lookup in cases when uploaded file marked as duplicated one. +Since we use best quality for all the duplicates this best one has related_duplicates count and all the duplicats have this rebound field set with id of "original" file. + +So in order to access a file you must follow this: `project_/` object lookup. + +When exporting datasets, rebound is resolved internally. Each exported entry is final, and you can access it directly via its id and extension (from file_name). diff --git a/docs/assets/goal_all.gif b/docs/assets/goal_all.gif new file mode 100644 index 00000000..9fe70630 Binary files /dev/null and b/docs/assets/goal_all.gif differ diff --git a/docs/assets/label_del_copy.gif b/docs/assets/label_del_copy.gif new file mode 100644 index 00000000..1536b618 Binary files /dev/null and b/docs/assets/label_del_copy.gif differ diff --git a/docs/assets/label_delete.gif b/docs/assets/label_delete.gif new file mode 100644 index 00000000..02f8e6fe Binary files /dev/null and b/docs/assets/label_delete.gif differ diff --git a/docs/assets/label_filters.gif b/docs/assets/label_filters.gif new file mode 100644 index 00000000..d1bf5538 Binary files /dev/null and b/docs/assets/label_filters.gif differ diff --git a/docs/assets/label_group.gif b/docs/assets/label_group.gif new file mode 100644 index 00000000..6eb56695 Binary files /dev/null and b/docs/assets/label_group.gif differ diff --git a/docs/assets/label_payload.gif b/docs/assets/label_payload.gif new file mode 100644 index 00000000..29fc661c Binary files /dev/null and b/docs/assets/label_payload.gif differ diff --git a/docs/assets/label_tree.gif b/docs/assets/label_tree.gif new file mode 100644 index 00000000..f4c68b40 Binary files /dev/null and b/docs/assets/label_tree.gif differ diff --git a/docs/assets/preview.gif b/docs/assets/preview.gif new file mode 100644 index 00000000..4082eba8 Binary files /dev/null and b/docs/assets/preview.gif differ diff --git a/docs/assets/project_create.gif b/docs/assets/project_create.gif new file mode 100644 index 00000000..3b6d35b6 Binary files /dev/null and b/docs/assets/project_create.gif differ diff --git a/docs/assets/project_download.gif b/docs/assets/project_download.gif new file mode 100644 index 00000000..6c0e5e8a Binary files /dev/null and b/docs/assets/project_download.gif differ diff --git a/docs/assets/project_goals.gif b/docs/assets/project_goals.gif new file mode 100644 index 00000000..bfd8f73f Binary files /dev/null and b/docs/assets/project_goals.gif differ diff --git a/docs/assets/project_roles.gif b/docs/assets/project_roles.gif new file mode 100644 index 00000000..0bf7f786 Binary files /dev/null and b/docs/assets/project_roles.gif differ diff --git a/docs/assets/project_stats.gif b/docs/assets/project_stats.gif new file mode 100644 index 00000000..7b80b710 Binary files /dev/null and b/docs/assets/project_stats.gif differ diff --git a/docs/assets/project_upload.gif b/docs/assets/project_upload.gif new file mode 100644 index 00000000..6e1c4406 Binary files /dev/null and b/docs/assets/project_upload.gif differ diff --git a/docs/assets/project_validate.gif b/docs/assets/project_validate.gif new file mode 100644 index 00000000..61c1e075 Binary files /dev/null and b/docs/assets/project_validate.gif differ diff --git a/docs/assets/registration.gif b/docs/assets/registration.gif new file mode 100644 index 00000000..ffcfaf74 Binary files /dev/null and b/docs/assets/registration.gif differ diff --git a/docs/assets/stats.gif b/docs/assets/stats.gif new file mode 100644 index 00000000..d4fa7700 Binary files /dev/null and b/docs/assets/stats.gif differ diff --git a/docs/assets/stats_diff.gif b/docs/assets/stats_diff.gif new file mode 100644 index 00000000..383cfdcc Binary files /dev/null and b/docs/assets/stats_diff.gif differ diff --git a/docs/assets/stats_export.gif b/docs/assets/stats_export.gif new file mode 100644 index 00000000..44611c81 Binary files /dev/null and b/docs/assets/stats_export.gif differ diff --git a/docs/assets/upload_label.gif b/docs/assets/upload_label.gif new file mode 100644 index 00000000..bbc9bd61 Binary files /dev/null and b/docs/assets/upload_label.gif differ diff --git a/docs/assets/upload_zoom.gif b/docs/assets/upload_zoom.gif new file mode 100644 index 00000000..2c20526e Binary files /dev/null and b/docs/assets/upload_zoom.gif differ diff --git a/docs/assets/validation_duplicate.gif b/docs/assets/validation_duplicate.gif new file mode 100644 index 00000000..975bd5d0 Binary files /dev/null and b/docs/assets/validation_duplicate.gif differ diff --git a/docs/downloads.md b/docs/downloads.md index 20146588..b9985b69 100644 --- a/docs/downloads.md +++ b/docs/downloads.md @@ -2,14 +2,12 @@ This section lets you download collected data for a project. - +download ## 🎛️ Filter First Use the filter block above to define which files should be included in the export. - - Once filters are set, you have two options: - `get archive`: @@ -22,8 +20,6 @@ Since it may include many files, this can take some time. This gives you only annotation data—no media included. It includes all annotations matching the current filters. - - ## 📋 Archive Table Once requested, all archives for the project are listed in a table below. @@ -32,17 +28,17 @@ Column **requested** shows the exact filters used to generate each archive. Click the download button to get the archive. - - Once the process completes, the row turns green. This means you can get your archive. - - The archive process may fail. In that case, the row turns red and the **result message** column shows what went wrong. - - ## ➕ Only New Files When re-downloading later (e.g., the next day), you might want to include only new files. Enable the `not downloaded before` flag to skip any files already included in past archives (based on the filters). + +--- + +### ✅ Next Step + +[Annotation](/docs/annotation.md) diff --git a/docs/goals.md b/docs/goals.md index d6ebc667..b04f2d97 100644 --- a/docs/goals.md +++ b/docs/goals.md @@ -2,7 +2,7 @@ This tab lets you define and track collection targets for specific labels. - +goals ## ➕ Create a Goal To define a goal @@ -13,8 +13,6 @@ To define a goal - 🖼️ Images (e.g., 1) - 🎞️ Videos (e.g., 2) - - A goal is considered fulfilled when the weighted sum of validated data reaches the target: Example - Goal = 5, image weight = 1, video weight = 2 @@ -33,11 +31,11 @@ Active goals are listed with: The table is sorted by progress (least done on top) for better prioritization. - - Goals are hidden once fulfilled, to keep focus on active targets. Toggle `show all` to view past goals or completed ones. +goals + --- ### ✅ Next Step diff --git a/docs/labels.md b/docs/labels.md index 068acdce..deaece15 100644 --- a/docs/labels.md +++ b/docs/labels.md @@ -10,22 +10,16 @@ Each project uses a label schema to annotate collected data. Labels help ensure When creating a project or later in the Project → Edit tab: - +labels_flags 1. Add Tree-like Levels - - 2. Add Values per level, with optional nesting. Each value is an actual label used during annotation. - - 3. Set flags like: - `required` - `multiple choice` - - 4. Each label value could have a payload. This is a meta information in valid `json/string` format. You can set restricted flag which sets if payload is required Examples: @@ -33,21 +27,17 @@ Examples: - `{"field1": "value1", "type": 1, "list": ["metalist1", "metalist2"]}` - `["meta1", "meta2"]` - +labels_payload 5. You can do a quick renaming with special form. - - 6. You can change alignment of values. - - 7. Deleting may be performed only when no media is assigned to such label or level of labels. When you hit remove (`minus`) button - the popup will tell you if this item cannot be removed. If you really want to remove it even when it's restricted you could change labeling at validation tab to remove association - +labels_delete 8. Grouping Each block represents a separate attribute tree (e.g., color, shape, type). @@ -56,7 +46,7 @@ I.E. each feature has its own tree / block/ To remove a group completely you have to delete all the levels. - +labels_add ## 📤 Applying Schema @@ -66,31 +56,27 @@ When [uploading data](/docs/uploads.md), you assign the schema: These methods can be used simultaneously - +labels_apply The label hierarchy defined in the project could be set several times by clicking `add object` button. This will create a new block with its own hierarchy. This might be useful if your image for example has several annotated objects like cars or animals. - - Each group can be easily deleted or copied. - +labels_apply_del-cp ## 📝 Validation Stage During [validation](/docs/validation.md), labels can be you have same tree you assigned on Upload stage with the same management. - - ## 🔍 Filtering At some pages there is an option to use Schema labels as filters. The usage is almost the same as you apply labels to the media. When you are done with tree click `select` button. - +labels_filter --- diff --git a/docs/projects.md b/docs/projects.md index bc9ee3ba..eed01467 100644 --- a/docs/projects.md +++ b/docs/projects.md @@ -3,8 +3,6 @@ Projects are the core unit of organization in the ISS Data Collection Tool. Each project defines its own label system, goals, and data scope. - - ## 🔹 What is a Project? A project groups: @@ -27,13 +25,11 @@ To create a project: - **Project Description** - **Label Schema** - +project_create Next you will be redirected back to list of projects. Select your newly created one. - - Each project has the following sections: (admin users will see all sections; regular users will only see a subset) @@ -54,7 +50,7 @@ Admins have full access to the whole application and items; Non admin users by default don't have any access to project or its data so you need to manually edit the permissions for it if needed. - +permissions See [Users & Roles](/docs/users.md) for how to manage users and set permissions @@ -64,7 +60,7 @@ After creating you can upload images or videos. After labeling and sending media to the server they will appear in validation section. Note: media processing on the server side may cause a slight delay before visibility. - +upload See [Uploads](/docs/uploads.md) for how to upload media @@ -73,7 +69,7 @@ See [Uploads](/docs/uploads.md) for how to upload media When you are done with uploading you can validate this data whether it's applicable to your project or not. Labels could be corrected there too. - +validate See [Validation](/docs/validation.md) for how to validate/ see uploaded images @@ -86,15 +82,13 @@ Each project might have a label schema, defining what’s being collected. Label - Required - Multiple items - - See [Labels](/docs/labels.md) for how to create and assign them. ## 🎯 Goals Optionally, you can define target counts for labels (e.g. "Need 100 images of `Dog`"). - +goals See [Goals](/docs/goals.md) for how to create and track them. @@ -105,7 +99,7 @@ Each project has a brief dashboard shows the collected count grouped by: - Media type - Validation type - +stats See [Statistics](/docs/statistics.md) @@ -113,7 +107,7 @@ See [Statistics](/docs/statistics.md) You can export annotated datasets as `.zip` archives with optional filters - +download See [Downloads](/docs/downloads.md) for how to request archives @@ -126,8 +120,6 @@ After you are done with editing click `SUBMIT EDIT` User roles could set with dedicated button. - - --- ### ✅ Next Step diff --git a/docs/statistics.md b/docs/statistics.md index 693e4cdd..eacf9c05 100644 --- a/docs/statistics.md +++ b/docs/statistics.md @@ -2,7 +2,7 @@ This tab displays how labels are used and how annotations are validated, broken down by media type. - +stats ## 🔍 Explore by Attribute or User @@ -10,12 +10,8 @@ You can group stats by: - **Attribute** (label and its nested structure) - - - **User** (who uploaded the data) - - Click on a `parent row` to expand child items. ## 📥 Export Options @@ -26,7 +22,7 @@ Export the table in: - `json` - `xlsx` - +stats_export Useful for further processing, reporting, or audits. @@ -37,7 +33,7 @@ The diff view splits data around a reference upload date. For example, setting `Diff from`: *2025-01-01* highlights what’s been uploaded since then—helpful for tracking changes over time. No data is displayed until a reference date is selected. - +stats_diff --- diff --git a/docs/uploads.md b/docs/uploads.md index c0a9e1fa..5920cd80 100644 --- a/docs/uploads.md +++ b/docs/uploads.md @@ -1,6 +1,6 @@ # 📤 Uploads - +upload ## 🔍 Select Files @@ -10,9 +10,6 @@ You can do that by: - Drag and dropping your files - Clicking `Add Media` button at the top left corner - - - ## 🧱 Annotate with Label Trees Each image/video could be annotated using a label schema you've defined earlier. @@ -23,7 +20,7 @@ You can: - `Delete Group`: Removes that label instance. - `Copy Group`: Clones an existing label set within the item. - +uploads_groups ## 🌀 Batch Apply The block on the left is for applying one label tree to all selected media. @@ -36,7 +33,7 @@ Hit `apply to all` → auto-fills that tree across items. Click any image to zoom in. Useful for small objects or cluttered scenes. This helps especially when annotating dense scenes or small objects. - +uploads_zoom ## ⏳ Uploading @@ -44,17 +41,11 @@ Once done labeling: Click `Upload` at the top right corner Files get submitted to the server for processing. - - You will be redirected to dedicated page showing the uploading state - - Once all media marked either green or red according to success status you are free to leave the page. - - Server will: - **Check** for duplicates diff --git a/docs/users.md b/docs/users.md index a38567ea..e82fbc9a 100644 --- a/docs/users.md +++ b/docs/users.md @@ -14,7 +14,7 @@ make init-admin - Admins: created manually or promoted from a common user - Self-registered users: have no access until granted project-specific permissions - +register ## Permissions Permissions are managed per project under: @@ -23,7 +23,7 @@ Project → Edit tab → `USER VISIBILITY` There you’ll see a list of users and a cross-table of permissions. Click `SUBMIT VISIBILITY` to save changes. - +permissions Permission types: - `Can view project` – appears in list and is accessible diff --git a/docs/validation.md b/docs/validation.md index 2b228ffc..9b91c89f 100644 --- a/docs/validation.md +++ b/docs/validation.md @@ -1,9 +1,9 @@ # ✅ Validation - - ## 📂 Browse Uploads +validation + See all uploaded media waiting for review. Left sidebar lists files – click to load one into view. Item card contains: @@ -11,24 +11,19 @@ Item card contains: - **Upload user** - **Upload date** - - - ✅ Green = **approved** - ❌ Red = **rejected** - 🔵 Blue = **untouched** - - +- 🟣 Purple = **duplicated** Use filters at the top to narrow by status, label, date or author. - +validation_filter ## 🎯 Review & Confirm Labels Main canvas shows the media + label overlays. - - Right sidebar holds the full label tree for that file along with file info such as: - **File id** @@ -37,8 +32,6 @@ Right sidebar holds the full label tree for that file along with file info such - **Label tree** - **Manage buttons** - - You can: - **View** the media, move it, zoom @@ -72,13 +65,11 @@ Key Features - **Automatic Detection**: Upon image upload or validation, the system checks whether the image already exists in the current project. Detected duplicates are flagged immediately. - **Visual Indicator**: Duplicates are visually marked with a large `DUPLICATE` tag displayed directly on the image view. - +validation_right - **Best Quality Selection**: When duplicates are found, the system automatically selects the **best quality version** (e.g., higher resolution) as the *primary reference image*. This version will be used upon downloading - **Duplication Browser**: A new **"show duplicates"** button has been added, allowing quick access to all known duplicates of a given image, side-by-side. - - - **User Decisions**: Annotators can manually resolve duplicates. Duplicate checks currently operate **within a single project only**.