Recalbox Forum

    • Register
    • Login
    • Search
    • Recent
    • Tags
    • recalbox.com
    • Gitlab repository
    • Documentation
    • Discord

    Scraping with PC - Batch process for you to try!

    Recalbox General
    scraping process try
    12
    17
    7796
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • nickdj
      nickdj last edited by

      I'm hoping these two batch files I've created will help those who wish to scrape all the rom files on their computer and copy the results to Recalbox. This could save you hours and more accurate. Of course you can still scrape the ones missed afterwards 🙂 Links: http://www.mediafire.com/view/hkzrjhtoel5iz4v/Read_Me.txt http://www.mediafire.com/download/sd9gzcm8h4kfnt1/scraper.exe http://www.mediafire.com/view/u01vt7tm7z4uqv6/scraperv2.bat http://www.mediafire.com/view/4kjkcgherbd0oye/scraperv2-deluxe.bat

      There are now two variations: scraperv2.bat - Scrapes all rom folders excluding psx, scumm, and virtualboy and puts the results in a folder called 'backup_scrape' scraperv2-deluxe.bat = Does all the above but checks data on roms with a missing hash, runs two workers rather than one, and once completed automatically copies the backup_scrape folder to the Recalbox\system folder! ----- 1) Create a folder on your computer and add your rom directories into it (e.g. snes, gamegear, etc) 2) Add scraper.exe to this folder 3) Add scraperv2.bat and/or scraperv2-deluxe.bat to the same folder 4) Double-click the scraperv2.bat or scraperv2-deluxe.bat file 5) Wait 6) If you did not run scraperv2-deluxe.bat copy the folder 'backup_scrape' to your Recalbox/system folder. 7) Get root access on terminal. Instructions can be found at: https://github.com/digitalLumberjack/recalbox-os/wiki/Root-access-on-terminal-(EN) 😎 Type/copy the follow and press return cp -r /recalbox/share/system/backup_scrape/downloaded_images /root/.emulationstation/ && cp -r /recalbox/share/system/backup_scrape/gamelists /root/.emulationstation/ ----- Let me know if you have any issues. Nick

      RPi 2 Not overclocked Edimax EW-7811UN LogiLink BT0007A USB bluetooth V2.1 PS3 Controller RecalBox v3.2.8

      1 Reply Last reply Reply Quote 0
      • hans-brokking
        hans-brokking last edited by

        Hey Nick, Thank you very much for your work! it makes life a lot easier. Ik have tried it with Nes and it works perfectly!! I ran out off time to try the rest but I noticed that FBA didn't find anything... I still have to try FBA Libreto. Does this scraper work for FBA?

        1 Reply Last reply Reply Quote 0
        • nickdj
          nickdj last edited by

          FBA scraping works for me. I have 1944, Gould and Ghosts, and SF3 no problem here. As long as the folders you are searching are named exactly like they are on Recalbox then you should be fine. It looks for finalburnalpha & finalburnalphalibretro folder and put the output in a folder called fba & fbalibretro (the default in Emulationstation) Let me know if you experience any problems. 🙂

          RPi 2 Not overclocked Edimax EW-7811UN LogiLink BT0007A USB bluetooth V2.1 PS3 Controller RecalBox v3.2.8

          1 Reply Last reply Reply Quote 0
          • hans-brokking
            hans-brokking last edited by

            ok,thanks for the reply. I will try again tomorow. The games work in FBA so the roms and folder names must be correct.

            1 Reply Last reply Reply Quote 0
            • neko
              neko last edited by

              Hi, just managed to get my Raspberry Pi 2 a few days ago, I spend my holiday to rebuild everything from zero, so I come back to share my results. This time, I got around 15000 ROMs on my SDCard, all ran through the scrapev2-deluxe.bat you supplied, done in a few hours without much problems. I must say that the scrapping speed and overall quality is very good, and I am very happy with what I get at this moment.   However I still have a few questions and see if you can help: 1. I noticed that the boot up time increased significantly after the scrape. The loading screen holds for around 2 minutes, is that normal? 2. About the scrapping results of mame and fba games, V1 gives good picture quality but lacks of description. V2 gives good descriptions but the screenshots looks pretty bad. Is there anyway to improve the situation by adjusting the parameters of the batch files? 3. It looks like the scrape.exe does not work with FDS, MSX, SG-1000, MegaCD and PCEngine CD. Does anyone have some luck with these systems?

              1 Reply Last reply Reply Quote 0
              • nickdj
                nickdj last edited by

                Thanks for the feedback.  I will have another look at the scrape results.  The 'scraper.exe' file allows a different parameter for image quality but the description should be there.  I will have a look at it later. I've noticed boot times increase the more roms you have.  Not got as far as 2 minutes but I have about 4000k roms and it takes about 30 seconds now.   I've been working on a new theme for recalbox, I'll be posting a pre-view in a bit.

                RPi 2 Not overclocked Edimax EW-7811UN LogiLink BT0007A USB bluetooth V2.1 PS3 Controller RecalBox v3.2.8

                1 Reply Last reply Reply Quote 0
                • nickdj
                  nickdj last edited by

                  I'm testing the FBA scraper and for some reason it's not picking up any descriptions.  The descriptions work within Recalbox though.  Tried to use a newer version of the scraper.exe but getting the same results.  😞

                  RPi 2 Not overclocked Edimax EW-7811UN LogiLink BT0007A USB bluetooth V2.1 PS3 Controller RecalBox v3.2.8

                  1 Reply Last reply Reply Quote 0
                  • airchrysalis
                    airchrysalis last edited by

                    Hi Nick, Did you ever figure out how to manually scrape and get it to save after shutdown? I'm trying to put metadata for folders containing psx games with multiple .iso files but the data is always lost when I shutdown. Do you have any suggestions?

                    1 Reply Last reply Reply Quote 0
                    • krantz
                      krantz last edited by

                      To save manually scraped metadata, you have to quit EmulationStation NOT using the quit menu, but pressing F4 on the keyboard.

                      1 Reply Last reply Reply Quote 0
                      • nickdj
                        nickdj last edited by

                        I'm struggling with one final issue with the scraper and that's FBA games do not have a description.  I've tried all sorts to pull down this information but it's not working. Has anyone managed to scrape FBA games via Windows Commandline and managed to get a description?

                        RPi 2 Not overclocked Edimax EW-7811UN LogiLink BT0007A USB bluetooth V2.1 PS3 Controller RecalBox v3.2.8

                        1 Reply Last reply Reply Quote 0
                        • rockaddicted
                          rockaddicted last edited by

                          Nope no descriptions about the arcade's games. Certainly a problem from the games's database. We are working on scrapping database to add natively to recalbox in the futur. (take so many time... so be patient ^^) If someone want add the descriptions manually, tell us. We'll add them to the database 😃 https://github.com/digitalLumberjack/recalbox-gamelists (for information, this database is based on the set games no-intro + arcade)

                          Recalbox V4.1.0 - RPI3 OC - Xbox360 wireless
                          Wiki ENG : https://github.com/recalbox/recalbox-os/wiki/Mini-How-To-(EN)
                          Wiki FR : https://github.com/recalbox/recalbox-os/wiki/Mini-How-To-(FR)

                          Don't forget to upvote messages if it has been useful ;)

                          1 Reply Last reply Reply Quote 0
                          • 0zzy
                            0zzy last edited by

                            hi all, i have followed this guide and firt of all this one https://github.com/digitalLumberjack/recalbox-os/wiki/Scrape-your-games-with-your-PC-(EN) but when i reboot the system, xml files are empty and no game details or images are shown in the emulator menu. what's wrong?

                            1 Reply Last reply Reply Quote 0
                            • rberthao
                              rberthao last edited by

                              This process works great, thanks!

                              1 Reply Last reply Reply Quote 0
                              • liquidsnakeblue
                                liquidsnakeblue last edited by

                                Is there any way to modify the batch file to DELETE roms it does not find metadata for?  If a rom's hash cannot be matched to the database it is more than likely a modified rom and I do not want it.  This way after the process is complete only good roms with matching metadata will show up in Recalbox. Liquidsnakeblue

                                1 Reply Last reply Reply Quote 0
                                • randomninjaatk
                                  randomninjaatk last edited by

                                  I would also be a fan of having the option to have the batch file to possible either move or delete roms that do not match up in the database via the hash. This would allow to have the cleanest looking library and hopefully a very valid set of roms. I just found this tool today and it's amazing! Thanks for all your work on it!

                                  1 Reply Last reply Reply Quote 0
                                  • deano66
                                    deano66 last edited by

                                    Hi I'm new to the forums ! I have been playing around with Recalbox over the past few months on and off and think it is awesome 🙂 I have built a number of different arcades using Mala and Hyperspin and hope to create a mini sized machine using Recalbox. I have had no luck with scraping images using all the methods above 😞 I can use the internal scraper and it saves and displays the images but it uses the same image for a number of different roms and is slow. If I use the above methods on a pc I get near all the images and they are accurate and quick but I can not see them on recalbox when I load it up. Sometimes a few systems will show(I have tested it with a fresh install maybe 20 times) and at other times none show or the box were the images should be shows but with nothing in it 😞 any help would be very much appreciated as I maybe doing something wrong but I have followed the tutorial as described and still have the same problem. cheers and thank you in advance

                                    1 Reply Last reply Reply Quote 0
                                    • darkbrokenheartz
                                      darkbrokenheartz last edited by

                                      I have a pretty foolproof and simple method that does not require to use te external pc.   Source https://github.com/sselph/scraper     Showing simplified method to quickly scrape artwork for Recalbox     Step 1.  Connect to your raspi.  On windows you can use putty  and just open a connection to your ip and use root for username  recalboxroot for password.   Or from a terminal client >ssh root@yourip   Step 2. Paste the following command > wget https://raw.githubusercontent.com/substring/fullscrape/master/fullscrape.sh && chmod u+x ./fullscrape.sh   Step 3.  Initiate an individual scrape request system by system For example to do SNES  see below > ./fullscrape.sh snes -no_thumb=true -max_width=375   To do nes > ./fullscrape.sh nes -no_thumb=true -max_width=375   Using the rom folder name as reference. Just a list of examples of systems you can use are   cavestory fba fbalibretro fds gamegear gb gba gbc gw lutro lynx mame mastersystem megadrive moonlight msx n64 neogeo ngp pcengine prboom psx scummvm sega32x segacd sg1000 snes vectrex virtualboy wswan     If you want to update or only add images that have not already been scraped you can do   > ./fullscrape.sh -u  -no_thumb=true -max_width=375     Things to keep in mind. To run the ./fullscrape.sh script you must be in the directory that you originally downloaded it to in the first step.   Image art is stored in the directory /root/.emulationstation/downloaded_images/ the gamelist.xml file is stored in the directory /root/.emulationstation/gamelists each in their respective named folder   if you run ./fullscrape.sh without the no thumb width flag it will download unvisible art for certain systems and you will need to delete that before scraping again as it will not overwrite correctly.  (This could be added as a pull request to github but may be a security feature)   WARNING : fba_libretro has a little bug, that will be corrected in 4.0.0 : scraping can’t be displayed. This can be manually solved if you really really are desperate …

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post

                                      Want to support us ?

                                      70
                                      Online

                                      98.6k
                                      Users

                                      28.1k
                                      Topics

                                      187.0k
                                      Posts

                                      Copyright © 2021 recalbox.com