Recalbox Forum

    • Register
    • Login
    • Search
    • Recent
    • Tags
    • recalbox.com
    • Gitlab repository
    • Documentation
    • Discord

    [SOFT] Universal XML Scraper - Windows Scraper for your Recalbox

    Recalbox General
    [soft] universal xml scraper windows
    19
    61
    56725
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • tubarao2007
      tubarao2007 last edited by

      Thats ok... I got the No-Intro e all is good. Thanks a lot...
      One more question: I have a lot of clones rom in mame and snes. There is a software to find and delete duplicate roms?

      1 Reply Last reply Reply Quote 0
      • screech
        screech last edited by

        Look at my signature πŸ˜‰

        Clean your Romset with Universal ROM Cleaner

        Clean your Romset with Universal ROM Cleaner
        Scrape your Romset with Universal XML Scraper

        1 Reply Last reply Reply Quote 0
        • ogroto
          ogroto last edited by

          Hi screech, your tool is amazing brother, keep it up the good work!
          Just some constructive criticism:
          The tool takes a lot of time , at least when i scrap 700 roms or so.
          Also i couldn't scrap roms that were already scraped from the oficial recalbox scraper, i don't know why,but it just skips them and does not download anything at all. If you fix this then this could be a heaven app πŸ˜ƒ
          regards

          1 Reply Last reply Reply Quote 0
          • screech
            screech last edited by

            Hi @ogroto

            I'm working on a Version 2 πŸ˜‰ Much more optimised and stable (with new XML component)
            So I hope it become faster than this version πŸ˜‰

            For your problem, if you want to "rescrape" the game already scraped by recalbox inner scraper, 2 solutions :

            • In Edit/Config menu choose "New XML File"
            • Or simply delete your old gamelist.xml on your recalbox.

            Clean your Romset with Universal ROM Cleaner
            Scrape your Romset with Universal XML Scraper

            1 Reply Last reply Reply Quote 0
            • ogroto
              ogroto last edited by

              Hello! i think i have already done both things. And still it does not download them. Both are 700+ roms snes and megadrive.

              1 Reply Last reply Reply Quote 0
              • screech
                screech last edited by

                Have you deleted Image in download_image directory too ?
                If a picture exist, it didn't download it again to spare some time πŸ˜‰

                Clean your Romset with Universal ROM Cleaner
                Scrape your Romset with Universal XML Scraper

                ogroto 1 Reply Last reply Reply Quote 0
                • ogroto
                  ogroto @screech last edited by ogroto

                  @screech Yes i deleted both, gamelist.xml in megadrive roms folder and also the complete download_image directory. I'm scraping rigth now will tell you soon how it goes. Thanks.
                  Edit: So, after 90 mins, the scraper only downloaded 3 images, which are the latest roms i added to the list. It created the xmls but only 3 images downloaded.
                  Edit2: ok so i think i found the issue, the scraper seems to not like .zip roms, at least for megadrive and snes. I use them compressed because i use a small sd 16gb card, and when it tries to scrap them zipped it doesn't work, but if i uncompress some rom the program works ok
                  Edit3: Ok so, there was a problem with my roms, half of them are not any of the correct CRCs that are listed in screenscraper.fr, which is really odd and also makes me redownload but found the issue. By the way, the software works a lot faster if you do everything in your pc HDD and then transfer the files manually to the Raspberry Pi. Screenscraper.fr takes a lot of time to load for me (i'm from Argentina) so maybe that is why also the app is slow. It is taking time but looking great so far! I'm really happy thanks again

                  1 Reply Last reply Reply Quote 0
                  • Tick Lpsk
                    Tick Lpsk last edited by

                    another question screech
                    when i scrape Neo Geo game (neogeo folder)
                    it end up scrape game as MAME (as shown on software )
                    and it's scrape just screenshot , no cartridge images
                    but when i rename folder to Neo-Geo then scrape
                    the program ask me what type of profile
                    then i choose Neo Geo AES
                    then it's work fine everything is include in one image
                    the question is why neogeo folder is now work foe neo geo game?
                    thanks

                    1 Reply Last reply Reply Quote 0
                    • supernature2k
                      supernature2k last edited by supernature2k

                      Hi @Tick-Lpsk , it is because the neogeo roms ARE mame games!
                      As screech's soft check the db based on hash, it find the hashes in mame games.
                      To sum up, you have the MVS version of games, not AES.

                      A workaround for this is to go to Edit>Settings and then Advanced at the bottom right.
                      From that advanced settings screen, set RechSYS value to 0. That way, UXS will always ask for the system. Tell the soft it's AES and here you go!

                      EDIT: In fact you also need to change the RechAPI in the same screen. Set it to 3|1|2.

                      Pi powered NES | Gameboy HD | RecalStation | RecalDrive
                      Upvote messages if it has been useful ;)

                      Tick Lpsk 1 Reply Last reply Reply Quote 0
                      • screech
                        screech last edited by

                        That's it ^^

                        thanks @supernature2k πŸ˜‰

                        Clean your Romset with Universal ROM Cleaner
                        Scrape your Romset with Universal XML Scraper

                        1 Reply Last reply Reply Quote 0
                        • Tick Lpsk
                          Tick Lpsk @supernature2k last edited by

                          @supernature2k wow!!!!
                          that's work man!
                          thank you supernature2k
                          also screech for great program

                          1 Reply Last reply Reply Quote 0
                          • daviduco
                            daviduco last edited by daviduco

                            Last night i scraped snes succesfully, i'm trying to scrape it now and it fails.

                            I can't scrape anything, it scrapes 600 roms in 2 minutes. Has anything changed? I've tried another 9 systems with exact same result.

                            [22:11:02] - Recuperation des informations des systemes
                            [22:11:03] - Fichier d'informations non valide
                            [22:11:03] - Selection du systeme
                            [22:11:03] - Recuperation des Profil
                            [22:11:03] - Recuperation des champs du profil
                            [22:11:03] - Nombre de roms trouvees : 547
                            [22:11:03] - Creation des parametres pour le Mix
                            [22:11:03] - ---------------------------------------------------------------------------------------
                            [22:11:03] - Calcul des Hash en : 0.24s
                            [22:11:03] - Recuperation des informations de la Rom no 1 (CRC + SYSTEM)
                            [22:11:04] - ()Rom non trouve : 90 Minutes - European Prime Goal (Europe).sfc (Hash : 1AD61BD0)
                            [22:11:04] - Fin de l'ecriture des informations de la Rom no 1 en 0.551s
                            [22:11:04] - ---------------------------------------------------------------------------------------
                            [22:11:04] - Calcul des Hash en : 0.24s
                            [22:11:04] - (
                            )Rom non trouve : Aaahh!!! Real Monsters (Europe).sfc (Hash : 946313FB)
                            [22:11:04] - Fin de l'ecriture des informations de la Rom no 2 en 0.299s
                            [22:11:04] - ---------------------------------------------------------------------------------------
                            [22:11:04] - Calcul des Hash en : 0.14s
                            [22:11:04] - ()Rom non trouve : ACME Animation Factory (Europe).sfc (Hash : AAD3A72D)
                            [22:11:04] - Fin de l'ecriture des informations de la Rom no 3 en 0.19s
                            [22:11:04] - ---------------------------------------------------------------------------------------
                            [22:11:04] - Calcul des Hash en : 0.14s
                            [22:11:04] - (
                            )Rom non trouve : ActRaiser (Europe).sfc (Hash : 09097B2B)
                            [22:11:04] - Fin de l'ecriture des informations de la Rom no 4 en 0.19s
                            [22:11:04] - ---------------------------------------------------------------------------------------
                            [22:11:05] - Calcul des Hash en : 0.22s
                            [22:11:05] - (*)Rom non trouve : Actraiser 2 (Europe) (En,Fr,De).sfc (Hash : 630E16A1)
                            [22:11:05] - Fin de l'ecriture des informations de la Rom no 5 en 0.272s
                            [22:11:05] - ---------------------------------------------------------------------------------------
                            [22:11:05] - Calcul des Hash en : 0.24s
                            [22:11:05] - ---------------------------------------------------------------------------------------
                            [22:11:05] - Debut de fusion des fichiers
                            [22:11:05] - Bilan

                            [22:11:05] - Scrape Annule
                            [22:11:05] - Roms trouvees :0 / 5
                            [22:11:05] - Roms ajoutes :0 / 5
                            [22:11:05] - Temps moyen par Rom :0.3s
                            [22:11:05] - Temps Total :0h 0m 2s

                            1 Reply Last reply Reply Quote 0
                            • screech
                              screech last edited by

                              Sorry. The server provider have problem. It will came back tomorrow.

                              Clean your Romset with Universal ROM Cleaner
                              Scrape your Romset with Universal XML Scraper

                              1 Reply Last reply Reply Quote 0
                              • thanosazlin
                                thanosazlin last edited by

                                just used it , works great. but, i only picked up 9 out of 206 N64 games? my .z64 files all have the name of the game no other special characters or extra text like (USA) or (U) [!] in the name of the roms. just wondering what might be causing them to not get picked up?

                                example of one of the roms in the _nintendo 64_missing.txt file

                                1080 Snowboarding.z64 08FE81C7 16 777 216 07:33.00 2004-03-28 FA27089C425DBAB99F19245C5C997613

                                paradadf 1 Reply Last reply Reply Quote 0
                                • paradadf
                                  paradadf Staff @thanosazlin last edited by

                                  @thanosazlin please, read the forum:
                                  https://forum.recalbox.com/topic/3842/how-to-screenscraper

                                  There you have the explanation why you fon't get roms scraped. Just a hint: the filename must be exact as the one in the database (without extension), when the CRC hash doesn't match.

                                  This is the thread for the version 2 btw:
                                  https://forum.recalbox.com/topic/4690/soft-universal-xml-scraper-v2

                                  If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                  1 Reply Last reply Reply Quote 0
                                  • azvarel
                                    azvarel last edited by

                                    Suggestion for UXS implementation: when the internet connection fails, instead of redoing the scraper starting again, downloading the images where it left off and then being able to mount the gamelist.xml

                                    My connection is 35mbps and the scrapper download is very slow, 40 scrapers for example here gives an estimate of 54 minutes 42 seconds.

                                    Zodian Theme | #SNESProject

                                    paradadf 1 Reply Last reply Reply Quote 0
                                    • paradadf
                                      paradadf Staff @azvarel last edited by

                                      @azvarel you can change the scrape mode in general configurations from new file to add new game (or something like that). There is no need to scrape everything again.
                                      About the speed... try contributing to screenscraper.fr to become more threads πŸ˜‰

                                      If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                                      azvarel 1 Reply Last reply Reply Quote 0
                                      • azvarel
                                        azvarel @paradadf last edited by

                                        @paradadf But if you drop the internet connection, it does not generate the gamelist.xml file, the program simply gives error and closes. Then this option is not valid.

                                        Zodian Theme | #SNESProject

                                        1 Reply Last reply Reply Quote 0
                                        • screech
                                          screech last edited by

                                          @azvarel normally there is a timeout function with retry... It try to download 3 times with 20 sec timeout for every file... And if it can't download, it will try the next rom... So it's strange you have a "crash"...

                                          40 roms in 54minutes is very longue... (more than a minute by rom :S )

                                          normally it's about 1 or 2 sec by roms... (even without multi-thread)

                                          Wich rom are you trying to scrape ?
                                          Exemple Big BIN or ISO files can take long time but it's not a "scrape" problem, it's an "Hash" problem...
                                          Your computeur need to calculate a value based on the file you scrape...
                                          If it's a big file and you have a pretty slow computer, it can take lot's of time...

                                          For these "type" of file, You need to consider .CUE or .GDI files.
                                          UXS will put you BIN in hidden without scrape them if you have a .CUE file with the same name πŸ˜‰

                                          Clean your Romset with Universal ROM Cleaner
                                          Scrape your Romset with Universal XML Scraper

                                          azvarel 2 Replies Last reply Reply Quote 0
                                          • azvarel
                                            azvarel @screech last edited by

                                            @screech my pc no is slow, is AM3+ with 16gb RAM, I'm doing this via wifi connection, it has some influence, I did not put the rpi3 on the router via ethernet cable. This time it was roms of CPS 2 that took time, even today I could not even do those of NEOGEO, simply the program UXS gave error and closed the application, I lost hours trying here and I decided to test the scraper atari 2600 and to my surprise, it worked. I tested initially with 16 roms, and the UXS gave an estimated term of 58 min and 28 sec, time that gives enough to watch an episode of a TV series. 😞

                                            Zodian Theme | #SNESProject

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Want to support us ?

                                            77
                                            Online

                                            99.6k
                                            Users

                                            28.1k
                                            Topics

                                            187.1k
                                            Posts

                                            Copyright Β© 2021 recalbox.com