Filter of MAME BIOSes, Devices and Mechanical machines.

Author: Wintermute0110

changelog.txt

@@ -54,12 +54,6 @@ WIP      [CORE] Filtering of BIOS/Devices not working properly for MAME ROMs.
          Use the offline scraper DB to determine what is a BIOS/Device in MAME.
          See https://forum.kodi.tv/showthread.php?tid=287826&pid=2697000#pid2697000
 
-WIP      [CORE] Unified ROM artwork directory. This will require heavy changes in categories.xml.
-         I still have to think how to properly implement this. This is related to the
-         desing of a prototype Kodi games database.
-         See https://forum.kodi.tv/showthread.php?tid=287826&pid=2865413#pid2865413
-         See https://forum.kodi.tv/showthread.php?tid=287826&pid=2867785#pid2867785
-
 WIP      [CORE] Automatic generation of 3D Boxes using BoxFront and BoxSpine.
          Copy this feature from AML.
 
@@ -74,7 +68,8 @@ WIP      [CORE] Use Libretro_BIOS_cores.json for the Retroarch BIOS audit and no
          database. However, Libretro_BIOS_cores.json does not have the ROM size.
 
 WIP      [FAVOURITES] Change the way artwork is stored in Favourites, Launchers and
-         Categories directories. Avoid using IDs in the artwork name unless absolutely necessary.
+         Categories directories. Avoid using IDs in the artwork name unless absolutely
+         necessary.
 
 WIP      [MULTIDISC] Edit multidisc ROMs. In general, improve the support for
          multidisc ROMs.
@@ -107,8 +102,6 @@ WIP      [SCRAPERS] ScreenScraper must choose the correct artwork depending on t
          AEL must have an official list of regions/languages, extracted from the No-Intro/Redump
          DATs, to translate to ScreenScraper values.
 
-WIP      [SCRAPERS] MAME offline scraper: include NPlayers field. Take it from nplayers.ini
-
 WIP      [SCRAPERS] MAME offline scraper: include ESRB field. Take it from GameInfoDB.
 
 WIP      [SCRAPERS] MAME offline scraper: include plot field. Take it from GameInfoDB.
@@ -119,7 +112,13 @@ WIP      [SCRAPERS] Persistent scraper cache browser. Ability to browse the stor
 
 [B]Advanced Emulator Launcher | version 0.9.10 | xx April 2020[/B]
 
-WIP      [CORE] New fields in database to be prepared for the future.
+WIP      [CORE] Unified ROM artwork directory. This will require heavy changes in categories.xml.
+         I still have to think how to properly implement this. This is related to the
+         desing of a prototype Kodi games database.
+         See https://forum.kodi.tv/showthread.php?tid=287826&pid=2865413#pid2865413
+         See https://forum.kodi.tv/showthread.php?tid=287826&pid=2867785#pid2867785
+
+WIP      [CORE] New fields in database to be prepared for the future. See NOTES.md
 
 WIP      [CORE] Support for Spines artwork. Support for BoxTextures artwork.
          Have a look at LaunchBox for other types of artwork that may be interesting.
@@ -165,6 +164,8 @@ FEATURE  [CORE] Changed the Offline Scraper XML fields. Now the XML files are no
 
 FEATURE  [CORE] Update the platform icons. Github issue #112.
 
+FEATURE  [SCRAPERS] MAME offline scraper: include NPlayers field. Take it from nplayers.ini
+
 FEATURE  [SCRAPERS] Store when there is a bad JSON from the scrapers for further
          analysis. Working for ScreenScraper scraper.
 
@@ -173,6 +174,9 @@ FEATURE  [SCRAPERS] Update MAME Offline Scraper database to version latest versi
 FEATURE  [SCRAPERS] Include isBIOS, isDevice, isMechanical in AOL MAME database. This is
          required to filter BIOSes and Devices in MAME launchers.
 
+WIP      [SCRAPERS] Filter MAME BIOSes, Devices and Mechanical machines with the new
+         Offline Scraper JSON databases.
+
 FEATURE  [SCRAPERS] Add a delay of 1200 ms between downloads to ScreenScraper scraper 
          to avoid scraper overloading. This is configurable in the scraper object.
          See https://forum.kodi.tv/showthread.php?tid=287826&pid=2923181#pid2923181

resources/main.py

@@ -9464,16 +9464,16 @@ def _roms_import_roms(self, launcherID):
 
         # --- Open ROM scanner report file ---
         launcher_report_FN = g_PATHS.REPORTS_DIR.pjoin(launcher['roms_base_noext'] + '_report.txt')
-        log_info('Report file OP "{0}"'.format(launcher_report_FN.getOriginalPath()))
-        log_info('Report file  P "{0}"'.format(launcher_report_FN.getPath()))
+        log_info('Report file OP "{}"'.format(launcher_report_FN.getOriginalPath()))
+        log_info('Report file  P "{}"'.format(launcher_report_FN.getPath()))
         report_fobj = open(launcher_report_FN.getPath(), "w")
         report_fobj.write('*** Starting ROM scanner ... ***\n'.format())
-        report_fobj.write('Launcher name  "{0}"\n'.format(launcher['m_name']))
-        report_fobj.write('Launcher ID    "{0}"\n'.format(launcher['id']))
-        report_fobj.write('ROM path       "{0}"\n'.format(rom_path.getPath()))
-        report_fobj.write('ROM ext        "{0}"\n'.format(launcher_exts))
-        report_fobj.write('ROM extra path "{0}"\n'.format(rom_extra_path.getPath()))
-        report_fobj.write('Platform       "{0}"\n'.format(launcher['platform']))
+        report_fobj.write('Launcher name  "{}"\n'.format(launcher['m_name']))
+        report_fobj.write('Launcher ID    "{}"\n'.format(launcher['id']))
+        report_fobj.write('ROM path       "{}"\n'.format(rom_path.getPath()))
+        report_fobj.write('ROM ext        "{}"\n'.format(launcher_exts))
+        report_fobj.write('ROM extra path "{}"\n'.format(rom_extra_path.getPath()))
+        report_fobj.write('Platform       "{}"\n'.format(launcher['platform']))
 
         # Check if there is an XML for this launcher. If so, load it.
         # If file does not exist or is empty then return an empty dictionary.
@@ -9494,14 +9494,17 @@ def _roms_import_roms(self, launcherID):
         # Check if scraper is ready for operation. Otherwise disable it internally.
         scraper_strategy.scanner_check_before_scraping()
 
+        # Create ROMFilter object. Loads filter databases for MAME.
+        romfilter = FilterROM(g_PATHS, self.settings, launcher['platform'])
+
         # --- Assets/artwork stuff ----------------------------------------------------------------
         # Ensure there is no duplicate asset dirs. Abort scanning of assets if duplicate dirs found.
         log_debug('Checking for duplicated artwork directories...')
         duplicated_name_list = asset_get_duplicated_dir_list(launcher)
         if duplicated_name_list:
             duplicated_asset_srt = ', '.join(duplicated_name_list)
-            log_info('Duplicated asset dirs: {0}'.format(duplicated_asset_srt))
-            kodi_dialog_OK('Duplicated asset directories: {0}. '.format(duplicated_asset_srt) +
+            log_info('Duplicated asset dirs: {}'.format(duplicated_asset_srt))
+            kodi_dialog_OK('Duplicated asset directories: {}. '.format(duplicated_asset_srt) +
                            'Change asset directories before continuing.')
             return
         else:
@@ -9513,7 +9516,7 @@ def _roms_import_roms(self, launcherID):
         if scraper_strategy.unconfigured_name_list:
             unconfigured_asset_srt = ', '.join(scraper_strategy.unconfigured_name_list)
             kodi_dialog_OK(
-                'Assets directories not set: {0}. '.format(unconfigured_asset_srt) +
+                'Assets directories not set: {}. '.format(unconfigured_asset_srt) +
                 'Asset scanner will be disabled for this/those.')
 
         # --- Create a cache of assets ---
@@ -9703,12 +9706,9 @@ def _roms_import_roms(self, launcherID):
                 report_fobj.write('  File not in launcher ROM list. Processing it ...\n')
 
             # --- Ignore BIOS ROMs ---
-            # Name of bios is: '[BIOS] Rom name example (Rev A).zip'
-            if self.settings['scan_ignore_bios']:
-                BIOS_re = re.findall('\[BIOS\]', ROM.getBase())
-                if len(BIOS_re) > 0:
-                    log_info("BIOS detected. Skipping ROM '{0}'".format(ROM.path))
-                    continue
+            if romfilter.ROM_is_filtered(ROM.getBase()): 
+                log_debug('ROM filtered.')
+                continue
 
             # --- Create new ROM and process metadata and assets ---------------------------------
             romdata = fs_new_rom()

resources/platforms.py

@@ -28,6 +28,9 @@
 DEFAULT_PLAT_MOBYGAMES     = '0'
 DEFAULT_PLAT_SCREENSCRAPER = '0'
 DEFAULT_PLAT_GAMEFAQS      = '0'
+PLATFORM_MAME_LONG    = 'MAME'
+PLATFORM_MAME_SHORT   = 'arcade-mame'
+PLATFORM_MAME_COMPACT = 'mame'
 PLATFORM_UNKNOWN_LONG    = 'Unknown'
 PLATFORM_UNKNOWN_SHORT   = 'unknown'
 PLATFORM_UNKNOWN_COMPACT = 'unknown'
@@ -222,7 +225,7 @@ def __init__(self, name, shortname, compactname, aliasof = None,
         DAT_NOINTRO, 'Magnavox - Odyssey2'),
 
     # --- MAME/Arcade ---
-    Platform('MAME', 'arcade-mame', 'mame', None, '23', '143', '75', '2', DAT_MAME),
+    Platform(PLATFORM_MAME_LONG, PLATFORM_MAME_SHORT, PLATFORM_MAME_COMPACT, None, '23', '143', '75', '2', DAT_MAME),
 
     # --- Mattel ---
     Platform('Mattel Intellivision', 'console-ivision', 'ivision', None, '32', '30', '115', '16',

resources/scrap.py

@@ -142,10 +142,54 @@
 # used to filter MAME machines.
 # This class is (will be) used in the ROM Scanner.
 class FilterROM(object):
-    def __init__(self, PATHS, settings):
+    def __init__(self, PATHS, settings, platform):
         log_debug('FilterROM.__init__() BEGIN...')
         self.PATHS = PATHS
         self.settings = settings
+        self.platform = platform
+        self.addon_dir = self.settings['scraper_aeloffline_addon_code_dir']
+
+        # If platform is MAME load the BIOS, Devices and Mechanical databases.
+        if self.platform == PLATFORM_MAME_LONG:
+            BIOS_path = os.path.join(self.addon_dir, 'data-AOS', 'MAME_BIOSes.json')
+            Devices_path = os.path.join(self.addon_dir, 'data-AOS', 'MAME_Devices.json')
+            Mechanical_path = os.path.join(self.addon_dir, 'data-AOS', 'MAME_Mechanical.json')
+            BIOS_list = self._load_JSON(BIOS_path)
+            Devices_list = self._load_JSON(Devices_path)
+            Mechanical_list = self._load_JSON(Mechanical_path)
+            # Convert lists to sets to execute efficiently 'x in y' operation.
+            self.BIOS_set = {i for i in BIOS_list}
+            self.Devices_set = {i for i in Devices_list}
+            self.Mechanical_set = {i for i in Mechanical_list}
+
+    def _load_JSON(self, filename):
+        log_debug('FilterROM::_load_JSON() Loading "{}"'.format(filename))
+        with open(filename) as file:
+            data = json.load(file)
+
+        return data
+
+    # Returns True if ROM is filtered, False otherwise.
+    def ROM_is_filtered(self, basename):
+        if self.platform == PLATFORM_MAME_LONG:
+            if basename in self.BIOS_set:
+                log_debug('FilterROM::ROM_is_filtered() Filtered MAME BIOS "{}"'.format(basename))
+                return True
+            if basename in self.Devices_set:
+                log_debug('FilterROM::ROM_is_filtered() Filtered MAME Device "{}"'.format(basename))
+                return True
+            if basename in self.Mechanical_set:
+                log_debug('FilterROM::ROM_is_filtered() Filtered MAME Mechanical "{}"'.format(basename))
+                return True
+        else:
+            # If it is not MAME it is No-Intro
+            # Name of bios is: '[BIOS] Rom name example (Rev A).zip'
+            BIOS_m = re.findall('\[BIOS\]', basename)
+            if BIOS_m:
+                log_debug('FilterROM::ROM_is_filtered() Filtered No-Intro BIOS "{}"'.format(basename))
+                return True
+
+        return False
 
 class ScraperFactory(object):
     def __init__(self, PATHS, settings):