Recalbox Forum

    • Register
    • Login
    • Search
    • Recent
    • Tags
    • recalbox.com
    • Gitlab repository
    • Documentation
    • Discord

    [BATCH] Scrape your roms on your PC - fastscraper

    Community projects
    multiconsole scraper fastscraper
    37
    221
    240627
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • paradadf
      paradadf Staff last edited by paradadf

      @Voljega If you open the batch file with a text editor and go to the main code section you can see (and change if you need to) all parameters. -max_width=375 defines the image size. All images are resized (proportionally) to 375 pixels width. I know that sselph scraper uses its own hash database based on thegamesdb.net as default (to be found here (hash.csv)) and I enabled the use of the OpenVGDB if the hash isn't in hash.csv by setting the flag to -use_ovgdb=true, so that more games are found. How exactly the hashing works, I can't say, sorry.  For the tests I did, if you change the name of the rom, the scraper still finds a match on the database. For mame, fba, fba_libretro and neogeo, mamedb is always used.

      If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

      1 Reply Last reply Reply Quote 0
      • sselph
        sselph last edited by

        Sorry for the late response, I only periodically look for mentions of the scraper. The scraper uses hashing for most systems and attempts to do a sha1 hash of the original rom data. To do this, it pa**es and/or strips out additional headers and flips bits around to get a consistent encoding.

        1 Reply Last reply Reply Quote 1
        • screech
          screech last edited by

          or the Universal_XML_Scraper which scrape from Windows but only in french ?

          Just for information, that's not true ^^ Universal XML Scraper scrape also in english if you choose english langage (and it's automatique now ^^) 😉

          Clean your Romset with Universal ROM Cleaner
          Scrape your Romset with Universal XML Scraper

          1 Reply Last reply Reply Quote 0
          • paradadf
            paradadf Staff last edited by paradadf

            @Screech Sorry for having written that if it isn't true... the last time I tried, everything scraped was in french. I don't know if you changed it in the meantime 😉 As there are always some improvements on sselph's side, I want to share with you a new version of fastscraper. Mainly and the most important, now you have the option to choose between scraping NeoGeo in mame mode (using mamedb.com as a source) or using it in non-mame mode (thegamesdb.net). That feature was implemented on v.1.1.4.

            fastscraper ver 01.07.2016

            Every time NeoGeo is selected, you get the option to choose between those two modes. I hope you find it useful!

            Download fastscraper ver. 01.07.2016

            If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

            1 Reply Last reply Reply Quote 0
            • screech
              screech last edited by

              @paradadf No problemo ^^ it's pretty new (the 'automatic english setup'. You can do it before the new version, but you have to edit the config file manually. Now you can just change your language to change the scraped info 😉 )

              Clean your Romset with Universal ROM Cleaner
              Scrape your Romset with Universal XML Scraper

              1 Reply Last reply Reply Quote 0
              • paradadf
                paradadf Staff last edited by paradadf

                As ScreenScraper.fr has been newly added as a source in sselph's scraper, I updated the batch to be able to easily use it. The support for SS is given starting with the scraper's version v1.1.10. Currently only consoles are supported, but you'll probably still have some problems with cd based consoles and/or computers. Arcade (mame, fba, fba_libretro) is with this release still not supported either. mamedb and tgdb are of course working as before.

                fastscraper console

                Starting with this version of the batch, I created on top of the file (just open it with any text editor) a section where you can change parameters such as:

                • Change the parameters "b,s" as you prefer.
                Comma separated order to prefer images, s=snapshot, b=boxart, f=fanart, a=banner, l=logo, 3b=3D boxart. (default "b")
                		set consoleIMG=-console_img="b,s"
                
                • Change between jpg or png image format
                jpg or png, the format to write the images. (default "jpg")
                		set imgFormat=-img_format="jpg"
                
                • Choose the data sources (if you activate the three of them, currently the hierarchy is gdb > ss > ovgdb)
                Use the hash.csv and theGamesDB metadata. (default true)
                		set useGDB=-use_gdb=false
                
                Use the OpenVGDB if the hash isn´t in hash.csv.
                	       set useOVGDB=-use_ovgdb=false
                

                SS is the only one I activated by default. Just change false for true as you prefer.

                Use the ScreenScraper.fr as a datasource.
                		set useSS=-use_ss=true
                
                • Chose the language (mainly english and french data currently) and region for the data/media scraped
                The order to choose for language if there is more than one for a value. (en, fr, es, de, pt) (default "en")
                		set langSS=-lang="en,es,pt,de,fr"
                
                The order to choose for region if there is more than one for a value. (us, eu, jp, fr, xx) (default "us,eu,jp,fr,xx")
                		set regionSS=-region="us,eu,jp,fr,xx"
                

                Download fastscraper ver. 04.10.2016

                If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                1 Reply Last reply Reply Quote -1
                • Substring
                  Substring last edited by

                  OMG, it's not autoit ! Hey @screech i'm sure this runs faster than UXS 😛

                  Former dev - Please reply with @substring so that i am notified when you answer me
                  Ex dev - Merci de me répondre en utilisant @substring pour que je sois notifé

                  1 Reply Last reply Reply Quote 0
                  • screech
                    screech last edited by

                    @Subs sure it run faster 😉 (and on Linux and MAC ^^) but no MIX template 😛 (for the moment ^^)

                    Clean your Romset with Universal ROM Cleaner
                    Scrape your Romset with Universal XML Scraper

                    Substring 1 Reply Last reply Reply Quote 0
                    • Substring
                      Substring @screech last edited by

                      @screech it's just a matter fo time 😛

                      Former dev - Please reply with @substring so that i am notified when you answer me
                      Ex dev - Merci de me répondre en utilisant @substring pour que je sois notifé

                      1 Reply Last reply Reply Quote 0
                      • screech
                        screech last edited by

                        @Subs I know Sselph make some try on MIX 😉

                        Clean your Romset with Universal ROM Cleaner
                        Scrape your Romset with Universal XML Scraper

                        Substring 1 Reply Last reply Reply Quote 0
                        • Substring
                          Substring @screech last edited by

                          @screech hahaha fear sselph, for your scraper will soon be overwhelmed 😄

                          Former dev - Please reply with @substring so that i am notified when you answer me
                          Ex dev - Merci de me répondre en utilisant @substring pour que je sois notifé

                          1 Reply Last reply Reply Quote 0
                          • screech
                            screech last edited by screech

                            @Subs Never ^^ I use AUTOIT !! and it's the FUTUR !!! (And wait for the V2 😉 you will see 😉 the great power of Autoit 😛 )

                            (Sorry paradadf, we are trolling on your post :S )

                            Clean your Romset with Universal ROM Cleaner
                            Scrape your Romset with Universal XML Scraper

                            paradadf 1 Reply Last reply Reply Quote 0
                            • paradadf
                              paradadf Staff @screech last edited by

                              @screech no prob 😉

                              If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                              1 Reply Last reply Reply Quote 0
                              • The D
                                The D Banned last edited by

                                Wow! This thing is fast and powerful. I'm glad I stumbled upon it!

                                One question- is there a way to prevent the addition of the regions to the display name?

                                paradadf 1 Reply Last reply Reply Quote 0
                                • paradadf
                                  paradadf Staff @The D last edited by

                                  @The-D try editing this to:
                                  -use_nointro_name=false

                                  If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                  1 Reply Last reply Reply Quote 0
                                  • paradadf
                                    paradadf Staff last edited by

                                    Little update:

                                    Now you can type all to scrape all the folders at once.
                                    fastscraper_console

                                    I added some extra extension so that, for example, ScummVM can now be scraped, assuming you followed this instructions.

                                    Please, don't forget to contribute to ScreenScraper and make the database bigger!
                                    PC DOS, PC Win3.xx, PC Win9X and PC Windows are newly added systems, so you can now scrape Cave Story and Doom games. (Thinking about doxbox and future ports ;))

                                    Download fastscraper ver. 24.10.2016

                                    If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                    1 Reply Last reply Reply Quote 0
                                    • Gonzakpo
                                      Gonzakpo last edited by

                                      Hi. Thanks for the script. I've just found a little error on it. If I change the parameter where downloaded images are stored, it just changes the value in the gamelist xml. It doesn't change the actual place where they are downloaded (the command is hardcoded to download always on downloaded_images).

                                      By the way, two days ago everything was working great. And now most of my roms are giving me "hash not found" although I verified the hashes and they have not changed. Well known games like 1942 are giving me "hash not found". Is something wrong with the scraper.exe ? I'm using the latest one.

                                      paradadf 1 Reply Last reply Reply Quote 0
                                      • paradadf
                                        paradadf Staff @Gonzakpo last edited by paradadf

                                        @Gonzakpo you can change the folder name and location in the for-loop in the main part. If you need help, tell me where exactly you want to store it and I'll find the correct way (or die trying!).
                                        Mamedb is down since today so maybe by fba, mame and neogeo was the problem? Or every other system too? sselph's scraper hasn't enabled arcade systems yet.

                                        If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                        voljega Gonzakpo 2 Replies Last reply Reply Quote 0
                                        • voljega
                                          voljega Banned @paradadf last edited by

                                          @paradadf Does this script works for Atari2600 scrapping ? And what is the size of the download images ?

                                          paradadf 1 Reply Last reply Reply Quote 0
                                          • paradadf
                                            paradadf Staff @voljega last edited by

                                            @voljega it works and right now it is set to 400px width, you can change it on the parameters with any text editor.

                                            If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Want to support us ?

                                            76
                                            Online

                                            99.6k
                                            Users

                                            28.1k
                                            Topics

                                            187.1k
                                            Posts

                                            Copyright © 2021 recalbox.com