GitHub guide - scraping with PC

[nickdj]

I'm having a problem getting this to work correctly. The instructions state: "scraper.exe -image_path="~/.emulationstation/downloaded_images/SYSTEM_NAME" -no_thumb=true -max_width=375" No matter what I do it always just creates a sub-folder called 'images' and a gameslist.txt file in the same folder. The contents of the backup_scrape folder are: A folder called 'gameslist' which contains a sub-folder for each type of rom (e.g. psx, snes, etc). In here the gameslist.txt file resides. Sub-folders for each type of rom which contain the scraped images. Does anyone have updated instructions on how to use scraper.exe so it stores the files in the correct structure? My only other alternative is to create a batch file which executes all the scrape commands and then runs a create/move for each!

[nickdj]

I think I've found a much better solution so will post the results and create a batch file

[nickdj]

I've also found an error in the backup/restore scrape process so I've updated it (issue incorrect path) https://github.com/digitalLumberjack/recalbox-os/wiki/Backup-your-scrape-(EN)

[nickdj]

OK, here is my solution which works a lot better. I've created a batch file to scrape all my roms on my PC. 1) Create a folder on your computer and add your rom directories into it (e.g. snes, gamegear, etc) 2) Add scraper.exe to this folder 3) Shift+right-click on the folder and select 'Open Command Window Here' 4) Enter the following commands replacing [rom-folder] with the correct one (e.g. snes) mkdir backup_scrape\gameslist[rom-folder] mkdir backup_scrape\downloaded_images[rom-folder] scraper.exe -rom_dir="[rom-folder]" -no_thumb=true -max_width=375 move gamelist.xml backup_scrape\gameslist[rom-folder] move images*.* backup_scrape\downloaded_images[rom-folder] 5) You will now have a folder called backup-scrape, copy it to your Recalbox/system folder. 6) Use the restore instructions found at https://github.com/digitalLumberjack/recalbox-os/wiki/Backup-your-scrape-(EN) I will run a full test later but I'm confident this will work. If it does I'll upload my batch file somewhere Hope this helps.

[nickdj]

Here is a link to the batch file I created. http://www.mediafire.com/view/hkzrjhtoel5iz4v/Read_Me.txt http://www.mediafire.com/view/fp999h9hsz5m9hv/scraper.bat http://www.mediafire.com/download/sd9gzcm8h4kfnt1/scraper.exe How to use 1) Create a folder on your computer and add your rom directories into it (e.g. snes, gamegear, etc) 2) Add scraper.exe to this folder 3) Add scraper.bat to the same folder 4) Double-click the scraper.bat file 5) Wait 5) You will now have a folder called 'backup-scrape', copy it to your Recalbox/system folder. 6) Use the restore instructions found at https://github.com/digitalLumberjack/recalbox-os/wiki/Backup-your-scrape-(EN) The above batch file does all folders other than the PSX and SCUMM folders currently. (Update @ 22/04/2015 @ 14:19 GMT: New update scrapes FBA for title screen more accurately. MAME scrape requires tweaking. Just need to check that Recalbox looks in the right location for the images)

[neko]

This is great news! The build-in scrapper is painfully slow, I just did scraped the MAME set alone and it took me almost 2 weeks. I am starting to scrap the FBA and console sets by using your batch file, will report back later

[nickdj]

I'm going to do a complete scrape of the 4000 odd roms I have and see what works and doesn't. 99.99% of scrapes work well this way, just need to make sure the restore process works correctly. For scraping the FBA roms I used (" are speech quotes): scraper.exe -rom_dir="finalburnalphalibretro" -image_path="~/.emulationstation/downloaded_images/finalburnalphalibretro" -mame=true -mame_img="t,c,s" -no_thumb=true -max_width=375 -use_filename=true This finds nearly all the images and game data and uses the title screen image by default. It would take hours to scrape 1000 snes roms via Recalbox and via this method it takes about 10 minutes and appears to be much more accurate.

[neko]

Looks like Virtual Boy is currently not supported by scraper.exe? Got a lot of error when I try to scrape.

[nickdj]

This should be the final version. http://www.mediafire.com/view/u01vt7tm7z4uqv6/scraperv2.bat I noticed there were a few errors in my batch file so I've not only stripped it down to make it more streamline but also made sure the folder names are correct. Scraping by PC is a LOT quicker, however, you will still find some roms which are not picked up. You can scrape with Recalbox for the ones missing. Note: PSX, Virtualboy, and SCUMM roms are not scanned. scraper.exe doesn't appear to like them. I've left in the famicomdisksystem scrape, however, I've not managed to get it to recognize any of the roms I have. If anyone finds any issues with this batch file let me know. You are free to copy and/or amend this file (but some thanks would e nice). Enjoy!

[neko]

YES!! I have tested a few console systems with the latest batch file and it appears to works perfectly. Thank you nickdj, this has made my life much easier. ;-D

[nickdj]

I would like to hear others experience from using this batch file. If it works and everyone is happy with it I'll create a how-to on GitHub.

[ekstreme]

It works great for me. The only thing that I dislike is that when using the batch scraper ROM suffixes eg (World) (Rev A) are kept and displayed, even if the ROM is renamed first. Scraping within Recalbox does behave like this.

[rockaddicted]

Hi and thank you for the work about the scrapping from a PC. If you could update the wiki with your method, that would be great =).

[0zzy]

hi all, i have followed this guide and firt of all this one https://github.com/digitalLumberjack/recalbox-os/wiki/Scrape-your-games-with-your-PC-%28EN%29 but when i reboot the system, xml files are empty and no game details or images are shown in the emulator menu. what’s wrong?

[acris]

Sorry for my english try this via putty :

killall emulationstation

upload your gamelist and pictures after via putty : nohup /etc/init.d/S31emulationstation restart & reboot rpi

[0zzy]

Thanks Acris, this solved for me! I didn't think to kill ES process!

[nickdj]

After scraping with a PC, uploading it to Recalbox, and then copying to the correct location you will need to reboot Recalbox.  I am assuming after scraping there is a corresponding gameslist.xml file?  This shouldn't be empty although I'm aware descriptions are missing from FBA roms. The gameslist.xml file should not be empty as the scraper only creates this file if it finds roms.  Which roms/folder isn't working for you?

[0zzy]

hi nickdj, after scraping with PC and uploading the file in the right place, and after rebooting the xml file are empty again. As Acris suggests, i had to kill Emulationstation process before uplodaing xml, after that i rebboted and all goes right.

[nickdj]

That's very interesting.  I've not had this issue to be honest, however, I'm planning to have another go at the scraper especially as I'm thinking of creating a new theme which uses larger images.   What I may do is include the 'kill ES' command as part of the process.  I take it this is only run after you've uploaded the files to the 'system' folder but before you copy them?  If so that shouldn't be too hard.

[0zzy]

Yes, ES process must be killed before copying xml to the right ./emulationstation/ ecc folder. that's strange..more strange why happen to me only!