Scraping with PC - Batch process for you to try!

[nickdj]

I'm hoping these two batch files I've created will help those who wish to scrape all the rom files on their computer and copy the results to Recalbox. This could save you hours and more accurate. Of course you can still scrape the ones missed afterwards Links: http://www.mediafire.com/view/hkzrjhtoel5iz4v/Read_Me.txt http://www.mediafire.com/download/sd9gzcm8h4kfnt1/scraper.exe http://www.mediafire.com/view/u01vt7tm7z4uqv6/scraperv2.bat http://www.mediafire.com/view/4kjkcgherbd0oye/scraperv2-deluxe.bat

There are now two variations: scraperv2.bat - Scrapes all rom folders excluding psx, scumm, and virtualboy and puts the results in a folder called 'backup_scrape' scraperv2-deluxe.bat = Does all the above but checks data on roms with a missing hash, runs two workers rather than one, and once completed automatically copies the backup_scrape folder to the Recalbox\system folder! ----- 1) Create a folder on your computer and add your rom directories into it (e.g. snes, gamegear, etc) 2) Add scraper.exe to this folder 3) Add scraperv2.bat and/or scraperv2-deluxe.bat to the same folder 4) Double-click the scraperv2.bat or scraperv2-deluxe.bat file 5) Wait 6) If you did not run scraperv2-deluxe.bat copy the folder 'backup_scrape' to your Recalbox/system folder. 7) Get root access on terminal. Instructions can be found at: https://github.com/digitalLumberjack/recalbox-os/wiki/Root-access-on-terminal-(EN) Type/copy the follow and press return cp -r /recalbox/share/system/backup_scrape/downloaded_images /root/.emulationstation/ && cp -r /recalbox/share/system/backup_scrape/gamelists /root/.emulationstation/ ----- Let me know if you have any issues. Nick

[hans-brokking]

Hey Nick, Thank you very much for your work! it makes life a lot easier. Ik have tried it with Nes and it works perfectly!! I ran out off time to try the rest but I noticed that FBA didn't find anything... I still have to try FBA Libreto. Does this scraper work for FBA?

[nickdj]

FBA scraping works for me. I have 1944, Gould and Ghosts, and SF3 no problem here. As long as the folders you are searching are named exactly like they are on Recalbox then you should be fine. It looks for finalburnalpha & finalburnalphalibretro folder and put the output in a folder called fba & fbalibretro (the default in Emulationstation) Let me know if you experience any problems.

[hans-brokking]

ok,thanks for the reply. I will try again tomorow. The games work in FBA so the roms and folder names must be correct.

[neko]

Hi, just managed to get my Raspberry Pi 2 a few days ago, I spend my holiday to rebuild everything from zero, so I come back to share my results. This time, I got around 15000 ROMs on my SDCard, all ran through the scrapev2-deluxe.bat you supplied, done in a few hours without much problems. I must say that the scrapping speed and overall quality is very good, and I am very happy with what I get at this moment.   However I still have a few questions and see if you can help: 1. I noticed that the boot up time increased significantly after the scrape. The loading screen holds for around 2 minutes, is that normal? 2. About the scrapping results of mame and fba games, V1 gives good picture quality but lacks of description. V2 gives good descriptions but the screenshots looks pretty bad. Is there anyway to improve the situation by adjusting the parameters of the batch files? 3. It looks like the scrape.exe does not work with FDS, MSX, SG-1000, MegaCD and PCEngine CD. Does anyone have some luck with these systems?

[nickdj]

Thanks for the feedback.  I will have another look at the scrape results.  The 'scraper.exe' file allows a different parameter for image quality but the description should be there.  I will have a look at it later. I've noticed boot times increase the more roms you have.  Not got as far as 2 minutes but I have about 4000k roms and it takes about 30 seconds now.   I've been working on a new theme for recalbox, I'll be posting a pre-view in a bit.

[nickdj]

I'm testing the FBA scraper and for some reason it's not picking up any descriptions.  The descriptions work within Recalbox though.  Tried to use a newer version of the scraper.exe but getting the same results.  

[airchrysalis]

Hi Nick, Did you ever figure out how to manually scrape and get it to save after shutdown? I'm trying to put metadata for folders containing psx games with multiple .iso files but the data is always lost when I shutdown. Do you have any suggestions?

[krantz]

To save manually scraped metadata, you have to quit EmulationStation NOT using the quit menu, but pressing F4 on the keyboard.

[nickdj]

I'm struggling with one final issue with the scraper and that's FBA games do not have a description.  I've tried all sorts to pull down this information but it's not working. Has anyone managed to scrape FBA games via Windows Commandline and managed to get a description?

[rockaddicted]

Nope no descriptions about the arcade's games. Certainly a problem from the games's database. We are working on scrapping database to add natively to recalbox in the futur. (take so many time... so be patient ^^) If someone want add the descriptions manually, tell us. We'll add them to the database https://github.com/digitalLumberjack/recalbox-gamelists (for information, this database is based on the set games no-intro + arcade)

[0zzy]

hi all, i have followed this guide and firt of all this one https://github.com/digitalLumberjack/recalbox-os/wiki/Scrape-your-games-with-your-PC-(EN) but when i reboot the system, xml files are empty and no game details or images are shown in the emulator menu. what's wrong?

[rberthao]

This process works great, thanks!

[liquidsnakeblue]

Is there any way to modify the batch file to DELETE roms it does not find metadata for?  If a rom's hash cannot be matched to the database it is more than likely a modified rom and I do not want it.  This way after the process is complete only good roms with matching metadata will show up in Recalbox. Liquidsnakeblue

[randomninjaatk]

I would also be a fan of having the option to have the batch file to possible either move or delete roms that do not match up in the database via the hash. This would allow to have the cleanest looking library and hopefully a very valid set of roms. I just found this tool today and it's amazing! Thanks for all your work on it!

[deano66]

Hi I'm new to the forums ! I have been playing around with Recalbox over the past few months on and off and think it is awesome I have built a number of different arcades using Mala and Hyperspin and hope to create a mini sized machine using Recalbox. I have had no luck with scraping images using all the methods above I can use the internal scraper and it saves and displays the images but it uses the same image for a number of different roms and is slow. If I use the above methods on a pc I get near all the images and they are accurate and quick but I can not see them on recalbox when I load it up. Sometimes a few systems will show(I have tested it with a fresh install maybe 20 times) and at other times none show or the box were the images should be shows but with nothing in it any help would be very much appreciated as I maybe doing something wrong but I have followed the tutorial as described and still have the same problem. cheers and thank you in advance

[darkbrokenheartz]

I have a pretty foolproof and simple method that does not require to use te external pc.   Source https://github.com/sselph/scraper     Showing simplified method to quickly scrape artwork for Recalbox     Step 1.  Connect to your raspi.  On windows you can use putty  and just open a connection to your ip and use root for username  recalboxroot for password.   Or from a terminal client >ssh root@yourip   Step 2. Paste the following command > wget https://raw.githubusercontent.com/substring/fullscrape/master/fullscrape.sh && chmod u+x ./fullscrape.sh   Step 3.  Initiate an individual scrape request system by system For example to do SNES  see below > ./fullscrape.sh snes -no_thumb=true -max_width=375   To do nes > ./fullscrape.sh nes -no_thumb=true -max_width=375   Using the rom folder name as reference. Just a list of examples of systems you can use are   cavestory fba fbalibretro fds gamegear gb gba gbc gw lutro lynx mame mastersystem megadrive moonlight msx n64 neogeo ngp pcengine prboom psx scummvm sega32x segacd sg1000 snes vectrex virtualboy wswan     If you want to update or only add images that have not already been scraped you can do   > ./fullscrape.sh -u  -no_thumb=true -max_width=375     Things to keep in mind. To run the ./fullscrape.sh script you must be in the directory that you originally downloaded it to in the first step.   Image art is stored in the directory /root/.emulationstation/downloaded_images/ the gamelist.xml file is stored in the directory /root/.emulationstation/gamelists each in their respective named folder   if you run ./fullscrape.sh without the no thumb width flag it will download unvisible art for certain systems and you will need to delete that before scraping again as it will not overwrite correctly.  (This could be added as a pull request to github but may be a security feature)   WARNING : fba_libretro has a little bug, that will be corrected in 4.0.0 : scraping can’t be displayed. This can be manually solved if you really really are desperate …