fix: [CI/CD] Disk free issues #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

SteveClement wants to merge 62 commits into ail-project:dev from SteveClement:CI/CD

.github/workflows/ail_framework_test.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -17,9 +17,27 @@ jobs: @@
         # The type of runner that the job will run on
         runs-on: ubuntu-latest
-        strategy:
-           matrix:
-             python-version: ['3.7', '3.8', '3.9', '3.10']
+        # TODO: Matrix strategy for Python versions is defined but never used.
+        # Currently all jobs use the same system Python, making this redundant.
+        # Either add 'actions/setup-python' to use matrix.python-version, or remove the matrix.
+        #
+        # To enable multi-version Python testing:
+        #
+        # Step 1: Uncomment the matrix below (defines the Python versions to test):
+        # strategy:
+        #    matrix:
+        #      python-version: ['3.7', '3.8', '3.9', '3.10']
+        #
+        # Step 2: Add this step after checkout (before "Free up disk space"):
+        #      - name: Set up Python ${{ matrix.python-version }}
+        #        uses: actions/setup-python@v4
+        #        with:
+        #          python-version: ${{ matrix.python-version }}
+        #
+        # ORIGINAL (commented out - not used, makes tests 4x slower with no benefit):
+        # strategy:
+        #    matrix:
+        #      python-version: ['3.7', '3.8', '3.9', '3.10']
         # Steps represent a sequence of tasks that will be executed as part of the job
@@ Expand All / @@ -30,6 +48,26 @@ jobs: @@
               submodules: 'recursive'
               fetch-depth: 500
+          # ---------------------------------------------
+          # NEW STEP: clean up disk BEFORE installing deps
+          # ---------------------------------------------
+          - name: Free up disk space
+            run: |
+              echo "Disk usage BEFORE cleanup:"
+              df -h
+              # Safe: Clear APT cache and lists (can be regenerated)
+              sudo apt-get clean
+              sudo rm -rf /var/lib/apt/lists/*
+              # Probably safe: Remove tools AIL doesn't need (check if exist first)
+              [ -d /usr/share/dotnet ] && sudo rm -rf /usr/share/dotnet || true
+              [ -d /opt/ghc ] && sudo rm -rf /opt/ghc || true
+              [ -d /usr/local/lib/android ] && sudo rm -rf /usr/local/lib/android || true
+              # Risky but needed: Remove hosted tool cache (contains Python, Node, etc.)
+              # AIL workflow uses system Python, so this should be safe
+              [ -d /opt/hostedtoolcache ] && sudo rm -rf /opt/hostedtoolcache || true
+              echo "Disk usage AFTER cleanup:"
+              df -h
+          # ---------------------------------------------
           # Runs a single command using the runners shell
           - name: Install AIL
@@ Expand Down @@

.gitignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -18,6 +18,7 @@ PASTES @@
     CRAWLED_SCREENSHOT
     IMAGES
     FAVICONS
+    FILES
     BASE64
     HASHS
     DATA_ARDB
@@ Expand Down @@

bin/LAUNCH.sh

-Original file line number
+Diff line change
@@ Expand Up / @@ -206,6 +206,8 @@ function launching_scripts { @@
         sleep 0.1
         screen -S "Script_AIL" -X screen -t "D4_client" bash -c "cd ${AIL_BIN}/core; ${ENV_PY} ./D4_client.py; read x"
         sleep 0.1
+        screen -S "Script_AIL" -X screen -t "Translation" bash -c "cd ${AIL_BIN}/modules; ${ENV_PY} ./Translation.py; read x"
+        sleep 0.1
         screen -S "Script_AIL" -X screen -t "UpdateBackground" bash -c "cd ${AIL_BIN}; ${ENV_PY} ./update-background.py; read x"
         sleep 0.1
@@ Expand Down Expand Up / @@ -619,7 +621,7 @@ function launch_tests() { @@
       echo -e $GREEN"\t* Flask:   $isflasked"$DEFAULT
       echo -e ""
       echo -e ""
-      python3 -m nose2 --start-dir $tests_dir --coverage $bin_dir --with-coverage test_api test_modules
+      python3 -m nose2 --start-dir $tests_dir --coverage $bin_dir --with-coverage test_api test_modules test_api_crawler
       exit $?
     }
@@ Expand Down @@

bin/crawlers/Crawler.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -281,6 +281,7 @@ def enqueue_capture(self, task_uuid, priority): @@
                                               user_agent=task.get_user_agent(),
                                               proxy=task.get_proxy(),
                                               cookies=task.get_cookies(),
+                                              storage=task.get_local_storage(),
                                               with_favicon=True,
                                               force=force,
                                               general_timeout_in_sec=90)  # TODO increase timeout if onion ????
@@ Expand Down @@

bin/exporter/MailExporter.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -143,7 +143,7 @@ def export(self, tracker, obj, matches=[]): @@
                     body += f'\nMatch {nb}: {match[0]}\nExtract:\n{match[1]}\n\n'
                     nb += 1
-            ail_link = f'AIL url:{obj.get_link()}\n\n'
+            ail_link = f'AIL url: {obj.get_link()}\n\n'
             for mail in tracker.get_mails():
                 if ail_users.exists_user(mail):
                     body = ail_link + body
@@ Expand Down @@

bin/importer/feeders/Default.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -61,6 +61,9 @@ def get_json_meta(self): @@
         def get_meta(self):
             return self.json_data.get('meta')
+        def get_meta_field(self, field, default=None):
+            return self.json_data.get('meta', {}).get(field, default)
         def get_payload(self):
             return self.json_data.get('data')
@@ Expand All / @@ -77,7 +80,7 @@ def get_gzip64_content(self): @@
             return self.json_data.get('data')
         def get_obj_type(self):
-            meta = self.get_json_meta()
+            meta = self.get_meta()
             return meta.get('type', 'item')
         ## OVERWRITE ME ##
@@ Expand Down @@

bin/importer/feeders/abstract_chats_feeder.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -12,6 +12,8 @@ @@
     import sys
     import time
+    import pymupdf4llm
     from abc import ABC
     sys.path.append(os.environ['AIL_BIN'])
@@ Expand All / @@ -20,12 +22,14 @@ @@
     ##################################
     from importer.feeders.Default import DefaultFeeder
     from lib.ail_core import get_chat_instance_name
+    from lib.objects import Authors
     from lib.objects.Chats import Chat
     from lib.objects import ChatSubChannels
     from lib.objects import ChatThreads
     from lib.objects import Images
     from lib.objects import Items
     from lib.objects import Messages
+    from lib.objects import PDFs
     from lib.objects import FilesNames
     # from lib.objects import Files
     from lib.objects import UsersAccount
@@ Expand Down Expand Up / @@ -168,6 +172,8 @@ def get_obj(self): @@
                 instance_name = get_chat_instance_name(self.get_chat_instance_uuid())
                 item_id = f'{instance_name}/{d[0:4]}/{d[4:6]}/{d[6:8]}/{self.json_data["data-sha256"]}.gz'
                 self.obj = Items.Item(item_id)
+            elif obj_type == 'pdf':
+                self.obj = PDFs.PDF(self.json_data['data-sha256'])
             else:
                 obj_id = Messages.create_obj_id(self.get_chat_instance_uuid(), chat_id, message_id, timestamp, thread_id=thread_id)
                 self.obj = Messages.Message(obj_id)
@@ Expand All @@
             if meta_chat.get('icon'):
                 img = Images.create(meta_chat['icon'], b64=True)
-                img.add(date, chat)
-                chat.set_icon(img.get_global_id())
-                if new_objs:
-                    new_objs.add(img)
+                if img:
+                    img.add(date, chat)
+                    chat.set_icon(img.get_global_id())
+                    if new_objs:
+                        new_objs.add(img)
             if meta_chat.get('username'):
                 username = Username(meta_chat['username'], self.get_chat_protocol())
@@ Expand Down Expand Up @@
             if meta.get('icon'):
                 img = Images.create(meta['icon'], b64=True)
-                img.add(date, chat)
-                chat.set_icon(img.get_global_id())
-                new_objs.add(img)
+                if img:
+                    img.add(date, chat)
+                    chat.set_icon(img.get_global_id())
+                    new_objs.add(img)
             if meta.get('username'):
                 username = Username(meta['username'], self.get_chat_protocol())
@@ Expand Down Expand Up @@
             if meta.get('icon'):
                 img = Images.create(meta['icon'], b64=True)
-                img.add(date, user_account)
-                user_account.set_icon(img.get_global_id())
-                new_objs.add(img)
+                if img:
+                    img.add(date, user_account)
+                    user_account.set_icon(img.get_global_id())
+                    new_objs.add(img)
             if meta.get('info'):
                 user_account.set_info(meta['info'])
@@ Expand Down Expand Up / @@ -363,9 +372,10 @@ def process_sender(self, new_objs, obj, date, timestamp): @@
             if meta.get('icon'):
                 img = Images.create(meta['icon'], b64=True)
-                img.add(date, user_account)
-                user_account.set_icon(img.get_global_id())
-                new_objs.add(img)
+                if img:
+                    img.add(date, user_account)
+                    user_account.set_icon(img.get_global_id())
+                    new_objs.add(img)
             if meta.get('info'):
                 user_account.set_info(meta['info'])
@@ Expand Down Expand Up / @@ -405,7 +415,8 @@ def process_meta(self): # TODO CHECK MANDATORY FIELDS @@
                 media_name = self.get_media_name()
                 if media_name:
                     print(media_name)
-                    FilesNames.FilesNames().create(media_name, date, obj)
+                    f = FilesNames.FilesNames().create(media_name, date, obj)
+                    objs.add(f)
                 for reaction in self.get_reactions():
                     obj.add_reaction(reaction['reaction'], int(reaction['count']))
@@ Expand All / @@ -430,13 +441,50 @@ def process_meta(self): # TODO CHECK MANDATORY FIELDS @@
                     if self.obj.type == 'image':
                         obj = Images.create(self.get_message_content())
-                        obj.add(date, message)
-                        obj.set_parent(obj_global_id=message.get_global_id())
+                        if obj:
+                            obj.add(date, message)
+                            obj.set_parent(obj_global_id=message.get_global_id())
+                            # FILENAME
+                            media_name = self.get_media_name()
+                            if media_name:
+                                f = FilesNames.FilesNames().create(media_name, date, message, file_obj=obj)
+                                objs.add(f)
+                    elif self.obj.type == 'pdf':
+                        # content
+                        if not self.obj.exists():
+                            obj = PDFs.create(self.obj.id, self.get_message_content())
+                            if not obj:
+                                raise Exception('PDF not created, Size limit reached')
+                            obj.set_parent(obj_global_id=message.get_global_id())
+                            pdf_meta = self.get_meta_field('file_metadata')
+                            if pdf_meta:
+                                obj.set_file_meta(pdf_meta)
+                                print(pdf_meta)
+                                if 'Author' in pdf_meta:
+                                    print(pdf_meta['Author'])
+                                    author = Authors.create(pdf_meta['Author'], obj)
+                                    author.add(date, obj)
+                            md_content = pymupdf4llm.to_markdown(obj.get_filepath())
+                            item_id = f'pdf/{date[0:4]}/{date[4:6]}/{date[6:8]}/{obj.id}.gz'
+                            item = Items.Item(item_id)
+                            if not item.exists():
+                                item.create(md_content, content_type='str')
+                                objs.add(item)
+                                print(item_id)
+                            obj.add_children('item', '', item_id)
+                            obj.add_correlation('item', '', item_id)
+                        self.obj.add(date, message)
                         # FILENAME
                         media_name = self.get_media_name()
                         if media_name:
-                            FilesNames.FilesNames().create(media_name, date, message, file_obj=obj)
+                            f = FilesNames.FilesNames().create(media_name, date, message, file_obj=self.obj)
+                            objs.add(f)
                     elif self.obj.type == 'item':
                         obj = self.obj
@@ Expand All / @@ -447,8 +495,9 @@ def process_meta(self): # TODO CHECK MANDATORY FIELDS @@
                         # FILENAME
                         media_name = self.get_media_name()
                         if media_name:
-                            file_name = FilesNames.FilesNames().create(media_name, date, message, file_obj=obj)
+                            f = file_name = FilesNames.FilesNames().create(media_name, date, message, file_obj=obj)
                             file_name.add_correlation('item', '', obj.id)
+                            objs.add(f)
             for obj in objs:  # TODO PERF avoid parsing metas multiple times
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: [CI/CD] Disk free issues #297

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

fix: [CI/CD] Disk free issues #297

Are you sure you want to change the base?

Uh oh!

fix: [CI/CD] Disk free issues #297

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!