Recalbox Forum

    • Register
    • Login
    • Search
    • Recent
    • Tags
    • recalbox.com
    • Gitlab repository
    • Documentation
    • Discord

    [SOFT] mamescraper - fast and simple scraper for MAME

    Community projects
    mame scraper
    5
    13
    9383
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • pdrb
      pdrb last edited by

      mamescraper is an open source fast and simple scraper that can scrap mame games information and images from 'mame.bigode.net' or 'adb.arcadeitalia.net' and generate a xml for use with EmulationStation.

      It should run on all platforms (Windows, Linux, Mac, etc...) that have Python 2 installed. The program uses only Python standard library for maximum compatibility.

      For Windows, a self contained executable file is available to download.

      mamescraper can:

      • Download images flyers and titles
      • Run with multiple workers to increase download/scrap speed
      • Read an existing xml file to scrap for missing games only

      Currently it supports two sources with drastically different scraping methodologies.

      The default source (bigode) is a lot faster because the scraper will download an entire mame database (1.1MB compressed) and scrap all the games information in one go. Besides that, this source uses a CDN and a very fast webserver to serve the images, resulting in faster responses and downloading speed overall.

      The adb source is a more traditional approach, for each game found, the scraper will do a http request to get the information needed and then will download the appropriate image.

      In my tests, I was able to scrap an entire mame 037b5 set (2241 roms) in less than two minutes using the default 'bigode' source and 10 workers threads.

      A useful use case is to run the scraper using the default 'bigode' source (since it's faster) and if a game is not found, just run the scraper again in 'append' mode using 'adb' as source to scrap only the missing games.

      Download from releases.

      More info, usage and source code on GitHub.

      A huge thanks to AntoPISA creator of Progetto Snaps for the images and tons of mame resources.

      Also, a huge thanks to Motoschifo creator of Arcade Database for the awesome arcade database website.

      Enjoy 😉

      DjLeChuck paradadf 3 Replies Last reply Reply Quote 1
      • DjLeChuck
        DjLeChuck @pdrb last edited by

        @pdrb Oh-oh-ooooooooooh that's a good soft! I will give it a try tonight!

        1 Reply Last reply Reply Quote 0
        • paradadf
          paradadf Staff @pdrb last edited by

          @pdrb cool, one more scraper! I'll take a look soon. Thanks for sharing your work!

          If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

          1 Reply Last reply Reply Quote 0
          • azvarel
            azvarel last edited by

            Great job!

            Is there a possibility to make mix image like uxs?

            Zodian Theme | #SNESProject

            1 Reply Last reply Reply Quote 0
            • DjLeChuck
              DjLeChuck last edited by

              Works great after a first fast test 🙂

              I will launch it on my complete romset.

              pdrb 1 Reply Last reply Reply Quote 0
              • pdrb
                pdrb @DjLeChuck last edited by pdrb

                @azvarel said in [SOFT] mamescraper - fast and simple scraper for MAME:

                Great job!

                Is there a possibility to make mix image like uxs?

                Sorry but I don't think it's worth it, since most of us just use flyers for arcade games...

                The scraper has a 'mixed' mode enabled by default, but in this case the 'mixed' mode is just for choosing the image type for download, in 'mixed' mode a flyer will be downloaded and if a flyer is not found, a title will be downloaded instead...

                @DjLeChuck said in [SOFT] mamescraper - fast and simple scraper for MAME:

                Works great after a first fast test 🙂

                I will launch it on my complete romset.

                Thanks for testing, let me know the results after you scrap your entire set 🙂

                1 Reply Last reply Reply Quote 0
                • paradadf
                  paradadf Staff @pdrb last edited by paradadf

                  @pdrb some feedback:
                  tested on windows...

                  • if you try to use -d path_to_games_dir, the software expects to find the mame_database.xml on that folder, but it is downloaded on the launch directory. So it should look for that xml file there 😉
                  mamescraper.exe -d mame -i flyer -s bigode -w 10
                  
                  Initializing the scraper...
                  
                  Scraping from: bigode
                  
                  Mame database xml not found
                   Downloading: 100%
                  Done!
                  
                  Generating list of images to download...
                  Traceback (most recent call last):
                    File "mamescraper.py", line 575, in <module>
                    File "mamescraper.py", line 508, in cli
                    File "mamescraper.py", line 188, in init_bigode
                  IOError: [Errno 2] No such file or directory: 'mame\\mame_database.xml'
                  Failed to execute script mamescraper
                  

                  But even after copying the xml file there, the scrape didn't work. Could it be something with the double backslash?

                  • There is no option to set the name of the folder where you download the images.

                  • If I lauch it from within the roms folder, it works as expected.

                  If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                  pdrb 1 Reply Last reply Reply Quote 0
                  • pdrb
                    pdrb @paradadf last edited by

                    @paradadf issues fixed 🙂

                    The problem was:

                    1. The romlist to scrap was being generated based on the current dir
                    2. The zip function was extracting in the current dir

                    I fixed those and added an option ('-e') to set the directory name to download the images.

                    Thanks for the feedback!

                    1 Reply Last reply Reply Quote 0
                    • DjLeChuck
                      DjLeChuck last edited by

                      On windows, the --append parameter did not works. I haven't writing down the error sorry... I will give it tonight.

                      pdrb 1 Reply Last reply Reply Quote 0
                      • pdrb
                        pdrb @DjLeChuck last edited by

                        @DjLeChuck strange...

                        I did some tests with append mode on my Windows 7 machine and it worked fine.

                        I am curious to know how you are running the scraper and the error message...

                        1 Reply Last reply Reply Quote 0
                        • saulob
                          saulob last edited by

                          How about using images/screenshot from the game not the flyer? How can I do that?

                          Thanks.

                          pdrb 1 Reply Last reply Reply Quote 0
                          • pdrb
                            pdrb @saulob last edited by

                            @saulob You can only use titles or flyers images of the games.

                            Sorry for the late reply 🙂

                            1 Reply Last reply Reply Quote 0
                            • pdrb
                              pdrb last edited by

                              I just updated the script to reflect the changes made last month on Arcade Database api.

                              Both sources are working fine now.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post

                              Want to support us ?

                              74
                              Online

                              98.9k
                              Users

                              28.1k
                              Topics

                              187.1k
                              Posts

                              Copyright © 2021 recalbox.com