[SOFT] mamescraper - fast and simple scraper for MAME

[pdrb]

mamescraper is an open source fast and simple scraper that can scrap mame games information and images from 'mame.bigode.net' or 'adb.arcadeitalia.net' and generate a xml for use with EmulationStation.

It should run on all platforms (Windows, Linux, Mac, etc...) that have Python 2 installed. The program uses only Python standard library for maximum compatibility.

For Windows, a self contained executable file is available to download.

mamescraper can:

• Download images flyers and titles
• Run with multiple workers to increase download/scrap speed
• Read an existing xml file to scrap for missing games only

Currently it supports two sources with drastically different scraping methodologies.

The default source (bigode) is a lot faster because the scraper will download an entire mame database (1.1MB compressed) and scrap all the games information in one go. Besides that, this source uses a CDN and a very fast webserver to serve the images, resulting in faster responses and downloading speed overall.

The adb source is a more traditional approach, for each game found, the scraper will do a http request to get the information needed and then will download the appropriate image.

In my tests, I was able to scrap an entire mame 037b5 set (2241 roms) in less than two minutes using the default 'bigode' source and 10 workers threads.

A useful use case is to run the scraper using the default 'bigode' source (since it's faster) and if a game is not found, just run the scraper again in 'append' mode using 'adb' as source to scrap only the missing games.

Download from releases.

More info, usage and source code on GitHub.

A huge thanks to AntoPISA creator of Progetto Snaps for the images and tons of mame resources.

Also, a huge thanks to Motoschifo creator of Arcade Database for the awesome arcade database website.

Enjoy

[DjLeChuck]

@pdrb Oh-oh-ooooooooooh that's a good soft! I will give it a try tonight!

[paradadf]

@pdrb cool, one more scraper! I'll take a look soon. Thanks for sharing your work!

[azvarel]

Great job!

Is there a possibility to make mix image like uxs?

[DjLeChuck]

Works great after a first fast test

I will launch it on my complete romset.

[pdrb]

@azvarel said in [SOFT] mamescraper - fast and simple scraper for MAME:

Great job!

Is there a possibility to make mix image like uxs?

Sorry but I don't think it's worth it, since most of us just use flyers for arcade games...

The scraper has a 'mixed' mode enabled by default, but in this case the 'mixed' mode is just for choosing the image type for download, in 'mixed' mode a flyer will be downloaded and if a flyer is not found, a title will be downloaded instead...

@DjLeChuck said in [SOFT] mamescraper - fast and simple scraper for MAME:

Works great after a first fast test

I will launch it on my complete romset.

Thanks for testing, let me know the results after you scrap your entire set

[paradadf]

@pdrb some feedback:
tested on windows...

• if you try to use -d path_to_games_dir, the software expects to find the mame_database.xml on that folder, but it is downloaded on the launch directory. So it should look for that xml file there

mamescraper.exe -d mame -i flyer -s bigode -w 10

Initializing the scraper...

Scraping from: bigode

Mame database xml not found
 Downloading: 100%
Done!

Generating list of images to download...
Traceback (most recent call last):
  File "mamescraper.py", line 575, in <module>
  File "mamescraper.py", line 508, in cli
  File "mamescraper.py", line 188, in init_bigode
IOError: [Errno 2] No such file or directory: 'mame\\mame_database.xml'
Failed to execute script mamescraper

But even after copying the xml file there, the scrape didn't work. Could it be something with the double backslash?

• There is no option to set the name of the folder where you download the images.

• If I lauch it from within the roms folder, it works as expected.

[pdrb]

@paradadf issues fixed

The problem was:

1. The romlist to scrap was being generated based on the current dir
2. The zip function was extracting in the current dir

I fixed those and added an option ('-e') to set the directory name to download the images.

Thanks for the feedback!

[DjLeChuck]

On windows, the --append parameter did not works. I haven't writing down the error sorry... I will give it tonight.

[pdrb]

@DjLeChuck strange...

I did some tests with append mode on my Windows 7 machine and it worked fine.

I am curious to know how you are running the scraper and the error message...

[saulob]

How about using images/screenshot from the game not the flyer? How can I do that?

Thanks.

[pdrb]

@saulob You can only use titles or flyers images of the games.

Sorry for the late reply

[pdrb]

I just updated the script to reflect the changes made last month on Arcade Database api.

Both sources are working fine now.