Recalbox Forum

    • Register
    • Login
    • Search
    • Recent
    • Tags
    • recalbox.com
    • Gitlab repository
    • Documentation
    • Discord

    [BATCH] Scrape your roms on your PC - fastscraper

    Community projects
    multiconsole scraper fastscraper
    37
    221
    240609
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • paradadf
      paradadf Staff last edited by paradadf

      The fastscraper is an useful batch script to be used with sselph scraper on Windows. With it, you can scrape many systems (some listed here) directly from your Windows PC (really fast!) according to the format needed in recalbox 4.
      You just need to put fastscraper.bat and scraper.exe on the root folder of your roms.
      Folder structure
      And execute fastscraper.bat. For multiple systems you can just type a space between the names or type all to scan all folders, and you can select if you want to update a gamelist or generate a new one. Fast and simple!
      fastscraper console
      Finally you would need to transfer the downloaded_images folder and gamelist.xml to your SD Card.

      You must stop EmulationStation before copying the gamelist.xml with the command: /etc/init.d/S31emulationstation stop and start it again after the copy is finished with /etc/init.d/S31emulationstation start.

      That's all!

      Download the latest version here!

      Update 08.01.2017:
      You can open a folder browser and navigate directly to the network share of your recalbox and scrape from it! This is, of course, slower than doing it on your computer, it can be useful though.

      Update 10.01.2017:
      macOS and Linux version available here.

      Update 27.01.2017:
      Automatic download/update of scraper.exe

      Update 25.04.2017:
      Mix images for consoles arrived and became the default.

      If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

      levendis32 1 Reply Last reply Reply Quote 2
      • nickdj
        nickdj last edited by

        Works great.  I've been playing with various tools (including the batch files I created many moons ago) and like the simplicity of this one.

        RPi 2 Not overclocked Edimax EW-7811UN LogiLink BT0007A USB bluetooth V2.1 PS3 Controller RecalBox v3.2.8

        1 Reply Last reply Reply Quote 0
        • voljega
          voljega Banned last edited by

          Looks simple enough ! A few questions : - what is the size of downloaded images ? - what criteria does it use to find a game ? hash code of the file ? full name of the rom file ? partial full name when no match is found for full name ?

          1 Reply Last reply Reply Quote 0
          • paradadf
            paradadf Staff last edited by paradadf

            @Voljega If you open the batch file with a text editor and go to the main code section you can see (and change if you need to) all parameters. -max_width=375 defines the image size. All images are resized (proportionally) to 375 pixels width. I know that sselph scraper uses its own hash database based on thegamesdb.net as default (to be found here (hash.csv)) and I enabled the use of the OpenVGDB if the hash isn't in hash.csv by setting the flag to -use_ovgdb=true, so that more games are found. How exactly the hashing works, I can't say, sorry.  For the tests I did, if you change the name of the rom, the scraper still finds a match on the database. For mame, fba, fba_libretro and neogeo, mamedb is always used.

            If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

            1 Reply Last reply Reply Quote 0
            • sselph
              sselph last edited by

              Sorry for the late response, I only periodically look for mentions of the scraper. The scraper uses hashing for most systems and attempts to do a sha1 hash of the original rom data. To do this, it pa**es and/or strips out additional headers and flips bits around to get a consistent encoding.

              1 Reply Last reply Reply Quote 1
              • screech
                screech last edited by

                or the Universal_XML_Scraper which scrape from Windows but only in french ?

                Just for information, that's not true ^^ Universal XML Scraper scrape also in english if you choose english langage (and it's automatique now ^^) 😉

                Clean your Romset with Universal ROM Cleaner
                Scrape your Romset with Universal XML Scraper

                1 Reply Last reply Reply Quote 0
                • paradadf
                  paradadf Staff last edited by paradadf

                  @Screech Sorry for having written that if it isn't true... the last time I tried, everything scraped was in french. I don't know if you changed it in the meantime 😉 As there are always some improvements on sselph's side, I want to share with you a new version of fastscraper. Mainly and the most important, now you have the option to choose between scraping NeoGeo in mame mode (using mamedb.com as a source) or using it in non-mame mode (thegamesdb.net). That feature was implemented on v.1.1.4.

                  fastscraper ver 01.07.2016

                  Every time NeoGeo is selected, you get the option to choose between those two modes. I hope you find it useful!

                  Download fastscraper ver. 01.07.2016

                  If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                  1 Reply Last reply Reply Quote 0
                  • screech
                    screech last edited by

                    @paradadf No problemo ^^ it's pretty new (the 'automatic english setup'. You can do it before the new version, but you have to edit the config file manually. Now you can just change your language to change the scraped info 😉 )

                    Clean your Romset with Universal ROM Cleaner
                    Scrape your Romset with Universal XML Scraper

                    1 Reply Last reply Reply Quote 0
                    • paradadf
                      paradadf Staff last edited by paradadf

                      As ScreenScraper.fr has been newly added as a source in sselph's scraper, I updated the batch to be able to easily use it. The support for SS is given starting with the scraper's version v1.1.10. Currently only consoles are supported, but you'll probably still have some problems with cd based consoles and/or computers. Arcade (mame, fba, fba_libretro) is with this release still not supported either. mamedb and tgdb are of course working as before.

                      fastscraper console

                      Starting with this version of the batch, I created on top of the file (just open it with any text editor) a section where you can change parameters such as:

                      • Change the parameters "b,s" as you prefer.
                      Comma separated order to prefer images, s=snapshot, b=boxart, f=fanart, a=banner, l=logo, 3b=3D boxart. (default "b")
                      		set consoleIMG=-console_img="b,s"
                      
                      • Change between jpg or png image format
                      jpg or png, the format to write the images. (default "jpg")
                      		set imgFormat=-img_format="jpg"
                      
                      • Choose the data sources (if you activate the three of them, currently the hierarchy is gdb > ss > ovgdb)
                      Use the hash.csv and theGamesDB metadata. (default true)
                      		set useGDB=-use_gdb=false
                      
                      Use the OpenVGDB if the hash isn´t in hash.csv.
                      	       set useOVGDB=-use_ovgdb=false
                      

                      SS is the only one I activated by default. Just change false for true as you prefer.

                      Use the ScreenScraper.fr as a datasource.
                      		set useSS=-use_ss=true
                      
                      • Chose the language (mainly english and french data currently) and region for the data/media scraped
                      The order to choose for language if there is more than one for a value. (en, fr, es, de, pt) (default "en")
                      		set langSS=-lang="en,es,pt,de,fr"
                      
                      The order to choose for region if there is more than one for a value. (us, eu, jp, fr, xx) (default "us,eu,jp,fr,xx")
                      		set regionSS=-region="us,eu,jp,fr,xx"
                      

                      Download fastscraper ver. 04.10.2016

                      If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                      1 Reply Last reply Reply Quote -1
                      • Substring
                        Substring last edited by

                        OMG, it's not autoit ! Hey @screech i'm sure this runs faster than UXS 😛

                        Former dev - Please reply with @substring so that i am notified when you answer me
                        Ex dev - Merci de me répondre en utilisant @substring pour que je sois notifé

                        1 Reply Last reply Reply Quote 0
                        • screech
                          screech last edited by

                          @Subs sure it run faster 😉 (and on Linux and MAC ^^) but no MIX template 😛 (for the moment ^^)

                          Clean your Romset with Universal ROM Cleaner
                          Scrape your Romset with Universal XML Scraper

                          Substring 1 Reply Last reply Reply Quote 0
                          • Substring
                            Substring @screech last edited by

                            @screech it's just a matter fo time 😛

                            Former dev - Please reply with @substring so that i am notified when you answer me
                            Ex dev - Merci de me répondre en utilisant @substring pour que je sois notifé

                            1 Reply Last reply Reply Quote 0
                            • screech
                              screech last edited by

                              @Subs I know Sselph make some try on MIX 😉

                              Clean your Romset with Universal ROM Cleaner
                              Scrape your Romset with Universal XML Scraper

                              Substring 1 Reply Last reply Reply Quote 0
                              • Substring
                                Substring @screech last edited by

                                @screech hahaha fear sselph, for your scraper will soon be overwhelmed 😄

                                Former dev - Please reply with @substring so that i am notified when you answer me
                                Ex dev - Merci de me répondre en utilisant @substring pour que je sois notifé

                                1 Reply Last reply Reply Quote 0
                                • screech
                                  screech last edited by screech

                                  @Subs Never ^^ I use AUTOIT !! and it's the FUTUR !!! (And wait for the V2 😉 you will see 😉 the great power of Autoit 😛 )

                                  (Sorry paradadf, we are trolling on your post :S )

                                  Clean your Romset with Universal ROM Cleaner
                                  Scrape your Romset with Universal XML Scraper

                                  paradadf 1 Reply Last reply Reply Quote 0
                                  • paradadf
                                    paradadf Staff @screech last edited by

                                    @screech no prob 😉

                                    If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                    1 Reply Last reply Reply Quote 0
                                    • The D
                                      The D Banned last edited by

                                      Wow! This thing is fast and powerful. I'm glad I stumbled upon it!

                                      One question- is there a way to prevent the addition of the regions to the display name?

                                      paradadf 1 Reply Last reply Reply Quote 0
                                      • paradadf
                                        paradadf Staff @The D last edited by

                                        @The-D try editing this to:
                                        -use_nointro_name=false

                                        If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                        1 Reply Last reply Reply Quote 0
                                        • paradadf
                                          paradadf Staff last edited by

                                          Little update:

                                          Now you can type all to scrape all the folders at once.
                                          fastscraper_console

                                          I added some extra extension so that, for example, ScummVM can now be scraped, assuming you followed this instructions.

                                          Please, don't forget to contribute to ScreenScraper and make the database bigger!
                                          PC DOS, PC Win3.xx, PC Win9X and PC Windows are newly added systems, so you can now scrape Cave Story and Doom games. (Thinking about doxbox and future ports ;))

                                          Download fastscraper ver. 24.10.2016

                                          If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                          1 Reply Last reply Reply Quote 0
                                          • Gonzakpo
                                            Gonzakpo last edited by

                                            Hi. Thanks for the script. I've just found a little error on it. If I change the parameter where downloaded images are stored, it just changes the value in the gamelist xml. It doesn't change the actual place where they are downloaded (the command is hardcoded to download always on downloaded_images).

                                            By the way, two days ago everything was working great. And now most of my roms are giving me "hash not found" although I verified the hashes and they have not changed. Well known games like 1942 are giving me "hash not found". Is something wrong with the scraper.exe ? I'm using the latest one.

                                            paradadf 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Want to support us ?

                                            66
                                            Online

                                            99.6k
                                            Users

                                            28.1k
                                            Topics

                                            187.1k
                                            Posts

                                            Copyright © 2021 recalbox.com