Recalbox Forum

    • Register
    • Login
    • Search
    • Recent
    • Tags
    • recalbox.com
    • Gitlab repository
    • Documentation
    • Discord

    [SOFT] New command-line scraper

    Community projects
    [soft] command-line scraper
    6
    16
    7729
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Bkg2k
      Bkg2k Staff last edited by Bkg2k

      A new scraper is available HERE
      Command line only for now, no user interface. Bêta stage - Source code in C# not released yet.
      Fetch metadata from ScreenScraper.fr
      Having a ScreenScraper account is not mandatory, but is highly recommanded since unregistered users won't be able to scrap when server load is too high.

      Run on Windows (Latest .NET framework required. Should not have any issue on windows 7 and higher) and linux (mono-complete package required, tested on Ubuntu 17.04) and probably Mac OSX (Mac user feedback is welcome!)

      Key Features:

      • Speed! A few miliseconds to scrap a single rom (not including HTTP request time which may vary depending of your bandwidth and screenscraper's server load)
      • Native support of zip, rar, gzip and 7zip archives - Read file CRC without extracting files
      • 7 integrated image Mix (inpired by classics from UXS - Thanks Screech)
      • Can also fetch raw media, including video, and soon HyperSpin themes and pdf manuals
      • Maximum multithreading, depending of your ranking on ScreenScraper server (Get a ScreenSaver account!)
      • Local cache, allowing to test multiple configurations without waiting hours or even days.
      • Aimed at Recalbox (& Retropie - EmulationStation in general), but can be used to populate other front-end's meta-data also (Launchbox, ...)

      Documentation available in french and english inside the .7z archive.

      Post your feedback here or on IRC.

      Quick test w/o RTFM:

      Skraper.exe D:\roms\snes  "Super Nintendo"  -output=mix:4images -ssu=your_login -ssp=your_password
      

      Next beta will support rb:<recalbox folder> as shortcut to set target system. Works with all recalbox rom folders except lua games and cavestory.
      Replace "D:\roms\snes" with one of your own rom folder
      For an arcade system, add the option -dnia (or -donotinspectarchives)

      To go into details:

      Skraper.exe D:\roms\snes "Super Nintendo" -output=mix:4images -ssu=your_ss_login -ssp=your_ss_password -v
      

      In case of bug or unexpected error:

      Skraper.exe D:\roms\snes "Super Nintendo" -output=mix:4images -ssu=your_ss_login -ssp=your_ss_password -v -logtofile
      

      Then isolate the few line around the error in the log file and post them here.

      When scraping on the recalbox SD directly, add -killrecalboxes to shutdown and reboot your recalbox atfer the gamelist.xml has been saved.

      With the 4image mix, you'll get:
      alt text

      Support custom JSON Mix file. No documentation yet. Add -gmt to generate a JSON file template from an internal mix (4images in the example above) and play with it.
      Wanna know more? RTFM 🙂

      1 Reply Last reply Reply Quote 2
      • Bkg2k
        Bkg2k Staff last edited by

        Updated to Bêta 0.9.17315.101, available HERE

        Release note:

        • Added Recalbox folder as system identifiers. Use rb:<folder> as system shortcut. (ex: "Super Nintendo" => rb:snes)
        • Added -killrecalboxes to kill ES before saving gamelist.xml, then reboot recalbox.
        • Fix wrong command line parameters not raising any error.
        keltac 1 Reply Last reply Reply Quote 1
        • keltac
          keltac @Bkg2k last edited by

          @bkg2k

          J'ai voulu tester, et j'ai rien compris ...

          Je suis pas à l'aise dans ce domaine, alors j'attends vivement une interface, pour moi et ceux que cela pourra intéresser.

          En tout cas, merci.

          1 Reply Last reply Reply Quote 0
          • Bkg2k
            Bkg2k Staff last edited by

            Une interface est en cours, mais il faudra attendre encore quelques jours 🙂

            A Windows/Linux GUI will be available soon.

            1 Reply Last reply Reply Quote 0
            • Bkg2k
              Bkg2k Staff last edited by

              Hi all,

              The Skraper GUI is on the way. The GUI use the same underlying scraping library, but offer a more visual approach for better understanding. It includes a Wizard to ease configuration of all the stuff for RecalBox, Retropie, Launchbox and even a Generic retroemulation profile.

              Originaly designed in english, I need help to translate the whole project.
              I'll do the french (france) translation and I've already a translator for portugeese/BRASIL.

              I'm looking for translators, mainly for:

              • German (de-DE)
              • Spanish (es-ES or es-* if there are only a few difference with other spanish-speaking countries)
              • Italian (it-IT)
              • Dutch (nl-NL)
              • pt-PT since there are many differences with pt-BR

              Obviously, any other language translator is welcome!

              For now, I cannot manage right-to-left languages, maybe later.

              Translation will be done (and updated) through Excel .xlsx files, and translators will receive new bêta everytime a translation is updated.

              Thanks in advance!

              Here is a link to several SkraperUI screenshots, quite old now, but I'll update new picture later:
              Skraper Scrrenshots

              paradadf NeeeeB 2 Replies Last reply Reply Quote 0
              • paradadf
                paradadf Staff @Bkg2k last edited by

                @bkg2k I can do the spanish translation.

                One request I have (haven’t been able to test your scraper yet though) is to upload all known hashes to SS automatically. Specially for large/cd based systems... screenscraper only calculates the crc when the rom file is big, thus large roms metadata is a bit incomplete for such.

                Thanks for your work!

                If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                Bkg2k 1 Reply Last reply Reply Quote 0
                • NeeeeB
                  NeeeeB @Bkg2k last edited by

                  @bkg2k

                  Regarde si tu peux utiliser un fichier .po - .mo pour tes traductions.
                  Plus simple à gérer que par xml, et tu peux du coup utiliser ce site qui est hyper pratique :
                  https://www.poeditor.com

                  Vieux geek ^^
                  GameList Editor : https://github.com/NeeeeB/GameList_Editor

                  Bkg2k 1 Reply Last reply Reply Quote 0
                  • Bkg2k
                    Bkg2k Staff @NeeeeB last edited by

                    Le soft est en C#, donc les langues et les ressources associées sont gérée automatiquement par Visual Studio (et 2 plugin pour me mâcher le travail un peu plus).
                    L'import/export excel permet à tout un chacun de traduire et me permet à moi de réimporter les traductions en 2 clicks 😉

                    1 Reply Last reply Reply Quote 0
                    • Bkg2k
                      Bkg2k Staff @paradadf last edited by

                      @paradadf You're welcome for the spanish translation 🙂

                      About automatic MD5/SHA1, sure that's a good idea. I can add this automatically for files less than a few megabytes, and add an options to force MD5/SHA for larges files.
                      The only limitation is for files in zip/7z/rar, because the scraper do not inflate files, it takes the CRC given by the archive instead.

                      paradadf 1 Reply Last reply Reply Quote 0
                      • paradadf
                        paradadf Staff @Bkg2k last edited by

                        @bkg2k yeah, no need to inflate them. Usually, cd-based roms are unzipped anyway. It is just that I hate seeing those hashes missing. If you could add such a switch and have it deactivated by default, that’d be great.

                        If your question was answered, please mark it as solved: Topic Tools -> Ask a question + Topic Tools -> Mark as Solved

                        Bkg2k 1 Reply Last reply Reply Quote 0
                        • Bkg2k
                          Bkg2k Staff @paradadf last edited by

                          Hello all,

                          The new GUI skraper is in alpha stage, tested by a few people. It's almost complete and it should be released in bêta stage in a few days for Windows only. Linux & MacOS version will come later.
                          Also, the "old" console version will be replaced with a new one, using the new scraping library used in the GUI version: faster & more accurate.
                          Meaningwhile, you may want to check the development progress here: www.skraper.net

                          @paradadf : feel free to contact me on ScreenScraper if your translation proposal is still relevant 🙂

                          Also, Italian and German translated are welcome for both the application and the Web Site! (other languages too 😉 )

                          Substring 1 Reply Last reply Reply Quote 0
                          • Substring
                            Substring @Bkg2k last edited by Substring

                            @bkg2k you should consuder poeditor (which means your project must be open source). Teabslations are extremly fast there

                            • Consider Mono compatibility. People don't only use windows ...

                            Former dev - Please reply with @substring so that i am notified when you answer me
                            Ex dev - Merci de me répondre en utilisant @substring pour que je sois notifé

                            1 Reply Last reply Reply Quote 0
                            • Bkg2k
                              Bkg2k Staff last edited by

                              @Substring Yes the project will be open source (as soon as I find a clean solution to remove my ScreenScraper's developer identifiers from the project 🙂 ).

                              As I use Visual Studio with the appropriate plugin, languages resources are managed automatically in a fast, efficient and clean way.
                              Also, mono compatibility is planned, as the soft shoud be released on linux & macOS.

                              1 Reply Last reply Reply Quote 1
                              • Dan Donoghue
                                Dan Donoghue last edited by Dan Donoghue

                                @bkg2k said in [SOFT] New command-line scraper:

                                @Substring Yes the project will be open source (as soon as I find a clean solution to remove my ScreenScraper's developer identifiers from the project 🙂 ).

                                As I use Visual Studio with the appropriate plugin, languages resources are managed automatically in a fast, efficient and clean way.
                                Also, mono compatibility is planned, as the soft shoud be released on linux & macOS.

                                That's always the hard part, Because screenscraper needs the password in each request. That's why I stopped working on my tool, got it to a point where all the admins could use it but we could never release it to the wild without my dev login being visible.

                                Edit: Hi bkg2k Blade Hunter here, havn't been around SS much since moving house in Dec :).

                                1 Reply Last reply Reply Quote 0
                                • Dan Donoghue
                                  Dan Donoghue last edited by Dan Donoghue

                                  Really nice looking front end on it, I have one question though.

                                  For the image output folder I wanted to make it something like this: %ROMROOTFOLDER%%SYSTEM%\downloaded_images Is %SYSTEM% a variable it will recognize?

                                  Bkg2k 1 Reply Last reply Reply Quote 0
                                  • Bkg2k
                                    Bkg2k Staff @Dan Donoghue last edited by

                                    @dan-donoghue Currently, Skraper replace the following varables:
                                    %ROMROOTFOLDER% : The selected root folder you selected in the wizard.
                                    %FRONTENDROOTFOLDER% : Selected front-end path in the wizard (only for Retropie & Launchbox).
                                    %SCREENSCRAPERPLATFORM% : Internal ScreenScraper system name.

                                    There is no variable for recalbox (or any other frontend) that map recalbox system folders. Actually those folders are hardcoded in the wizard (for ES-based front-ends)

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post

                                    Want to support us ?

                                    88
                                    Online

                                    98.7k
                                    Users

                                    28.1k
                                    Topics

                                    187.0k
                                    Posts

                                    Copyright © 2021 recalbox.com