Recalbox Forum

    • Register
    • Login
    • Search
    • Recent
    • Tags
    • recalbox.com
    • Gitlab repository
    • Documentation
    • Discord

    [BATCH] Scrape your roms on your PC - fastscraper

    Community projects
    multiconsole scraper fastscraper
    37
    221
    240603
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • goyetus
      goyetus @paradadf last edited by

      @paradadf

      It Only happens at night. At mid-day is perfect, so I suppose is a server limit for Ip range.


      Creo que hablas Español, asi que tambien te lo dejo escrito en español 😛
      Creo que solo ocurre de noche, a medio dia funciona perfecto. creo que es una limitacion que esta aplicando el servidor a un rango de ips en concreto (yo estoy con movistar)

      1 Reply Last reply Reply Quote 0
      • pyrostic
        pyrostic last edited by pyrostic

        @paradadf
        I'm pretty NooB at this. How do I run commands from the RP3 in order to stop the emulation station before copying? I'm using a Windows PC.

        paradadf 1 Reply Last reply Reply Quote 0
        • paradadf
          paradadf Staff @pyrostic last edited by

          @pyrostic i strongly recommend you to read the wiki/manuals, as all the basic (and some advanced) festures and tricks are explained there.
          SSH:
          https://github.com/recalbox/recalbox-os/wiki/Root-access-on-terminal-(EN)
          WinSCP:
          https://github.com/recalbox/recalbox-os/wiki/Network-access-with-WinSCP-(EN)

          If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

          1 Reply Last reply Reply Quote 0
          • Keladir
            Keladir last edited by

            Ok. I got the scraper to scrape my ROM files and create the xml file and images folder. When I put my SD card in and open the main folder, all I have is a boot folder of around 50 MB (of a 32 GB card) that doesn't show any of my ROM files to be able to place the images folder and xml file in there. When I put the SD card into my raspberry pi zero and load up Retropie, all my ROMS are on there and I'm able to play them, but it doesn't show any of the image files or game data if I put the folder and xml onto the boot area I'm getting when I plug it into my PC.

            What am I doing wrong? Are the ROMS on a separate partition, or something?

            paradadf 1 Reply Last reply Reply Quote 0
            • paradadf
              paradadf Staff @Keladir last edited by

              @Keladir first of all, you are on the recalbox forum. for retropie specific stuff, you should use its forum.

              Now about your "issue"... Windows PCs can't read the extra partitions, so you need special software to access them. And as clear as I was on my instructions on the first post, you need to place both, gamelist.xml and downloaded_images folder next to your roms. So I just don't get why would you place it anywhere else. I doesn't make any sense.

              If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

              Keladir 1 Reply Last reply Reply Quote 0
              • Keladir
                Keladir @paradadf last edited by

                @paradadf I opened it up on Retropie's file explorer to poke around, and when I imported my ROMS via USB thumb drive, it copied over the images folder and xml file into the same directory as all the roms, "next to" them as you say, yet they're still not showing up when I load into a console's game list.

                Your initial post doesn't say where on the SD card to transfer the XML and images folder to.

                paradadf 1 Reply Last reply Reply Quote 0
                • paradadf
                  paradadf Staff @Keladir last edited by

                  @Keladir again, this is the recalbox forum, if you have problems transfering the files, make your question on the retropie forum.

                  Your initial post doesn't say where on the SD card to transfer the XML and images folder to.

                  Next to your roms is a unique location. Wherever the correct place were you copied your roms for retropie, that's the place! I don't know how retropie works, but my initial post also states that you must kill emulationstation before trying to edit/copy/transfer/touch the gamelist.xml. And that is valid for emulationstation in general.

                  Sorry but either you don't want to understand, or you are not paying attention to what you read.

                  If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                  1 Reply Last reply Reply Quote 0
                  • airart
                    airart last edited by

                    hi, I copied the roms of my PC, but I get this error
                    alt text

                    I apologize for my English, but in Italian forum I did not have answers

                    paradadf 1 Reply Last reply Reply Quote 0
                    • paradadf
                      paradadf Staff @airart last edited by

                      @airart mamedb is down and thus, unscrapable. I've seen that sselph just added a mirror for it and enabled SS as source for mame as well. He just hasn't compiled and released a new version with all those changes yet so I can test and upgrade the batch.

                      If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                      1 Reply Last reply Reply Quote 0
                      • paradadf
                        paradadf Staff last edited by

                        fastscraper has been upgraded and is fully functional again.
                        New feature: you can open a folder browser and choose the roms folder of your recalbox directly over the network to scrape from it!
                        I have set ScreenScraper as the only source because the quality is superior to any other. You can expand or modify it easily if you wish. Enjoy!

                        If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                        1 Reply Last reply Reply Quote 0
                        • kevinnash
                          kevinnash last edited by

                          Hi,

                          How do we download it on github ?

                          I don't see any download link there.

                          1 Reply Last reply Reply Quote 0
                          • paradadf
                            paradadf Staff last edited by

                            You can select RAW and copy the plain text to a txt file and rename the extension to .bat

                            If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                            1 Reply Last reply Reply Quote 0
                            • maximus37
                              maximus37 last edited by paradadf

                              This tool is fantastic!! Nice work!!

                              One problem though, I cannot scrape NeoGeo roms, I get the following output:

                              2017/01/10 10:06:56 arcade  srcs: ss
                              2017/01/10 10:06:56 console srcs:
                              2017/01/10 10:06:56 INFO: Starting: C:\Users[...]\Downloads\neogeo\3countb.zip
                              2017/01/10 10:06:56 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bba.zip
                              2017/01/10 10:06:56 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bbh.zip
                              2017/01/10 10:06:56 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bb.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\2020bb.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bb.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users\\Downloads\neogeo\2020bbh.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bbh.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\3countb.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\3countb.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\2020bba.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bba.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\2020bb.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bb.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\2020bbh.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bbh.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\3countb.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\3countb.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\2020bba.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\2020bba.zip
                              2017/01/10 10:06:58 ERR: error processing C:\Users[...]\Downloads\neogeo\2020bb.zip: image: unknown format
                              2017/01/10 10:06:58 INFO: Starting: C:\Users[...]\Downloads\neogeo\alpham2.zip
                              panic: runtime error: invalid memory address or nil pointer dereference
                              [signal 0xc0000005 code=0x0 addr=0x14 pc=0x408fe7]
                              
                              goroutine 22 [running]:
                              panic(0x7e9580, 0x113a0030)
                                      /usr/local/go/src/runtime/panic.go:500 +0x331
                              main.crawlROMs.func2(0x11472820, 0x114aa900, 0x0, 0x11472810, 0x114755a0, 0x11475520)
                                      /home/sselph/go/src/github.com/sselph/scraper/scraper.go:295 +0x527
                              created by main.crawlROMs
                                      /home/sselph/go/src/github.com/sselph/scraper/scraper.go:328 +0xbd7
                              
                              Start    : 10:06:56.62
                              Finish   : 10:06:58.67
                                       ---------------
                              Duration : 00:00:02,05
                              
                              1 Reply Last reply Reply Quote 0
                              • acris
                                acris last edited by

                                Hello @maximus37
                                the server for arcade is down.

                                maximus37 1 Reply Last reply Reply Quote 0
                                • maximus37
                                  maximus37 @acris last edited by

                                  @acris Oh, well that explains it 🙂

                                  How can I check the status of it? How long has it been down? Is there a site I can keep an eye on?

                                  Thanks!

                                  paradadf 1 Reply Last reply Reply Quote 0
                                  • paradadf
                                    paradadf Staff @maximus37 last edited by

                                    @maximus37 @acris The problem with NEOGEO right now (using SS as source) is explained here: https://github.com/sselph/scraper/issues/128
                                    So most likely sselph will need to compile a new version with the corrections to allow us to scrape it.

                                    If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                    maximus37 1 Reply Last reply Reply Quote 0
                                    • maximus37
                                      maximus37 @paradadf last edited by

                                      @paradadf Thanks for the link!

                                      maximus37 1 Reply Last reply Reply Quote 0
                                      • maximus37
                                        maximus37 @maximus37 last edited by

                                        @paradadf Hi, I noticed the issue was fixed? Do I simply download the latest version of fastscraper? Do I need to include any options for the scraping?

                                        paradadf 1 Reply Last reply Reply Quote 0
                                        • paradadf
                                          paradadf Staff @maximus37 last edited by paradadf

                                          @maximus37 it wasn't a problem from sselph's scraper, but from the default used database (SS). So the version you have should work. But it is always better to use the latest 😉

                                          About the options, on top of the batch you can modify the different flags to your taste, change sources, type of images, etc. So is up to you which configuration you prefer.

                                          If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                          1 Reply Last reply Reply Quote 0
                                          • airart
                                            airart last edited by

                                            I try to download or copy fastscraper.bat new version but server is down

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Want to support us ?

                                            72
                                            Online

                                            99.6k
                                            Users

                                            28.1k
                                            Topics

                                            187.1k
                                            Posts

                                            Copyright © 2021 recalbox.com