[SOFT] New command-line scraper

[Bkg2k]

A new scraper is available HERE
Command line only for now, no user interface. Bêta stage - Source code in C# not released yet.
Fetch metadata from ScreenScraper.fr
Having a ScreenScraper account is not mandatory, but is highly recommanded since unregistered users won't be able to scrap when server load is too high.

Run on Windows (Latest .NET framework required. Should not have any issue on windows 7 and higher) and linux (mono-complete package required, tested on Ubuntu 17.04) and probably Mac OSX (Mac user feedback is welcome!)

Key Features:

• Speed! A few miliseconds to scrap a single rom (not including HTTP request time which may vary depending of your bandwidth and screenscraper's server load)
• Native support of zip, rar, gzip and 7zip archives - Read file CRC without extracting files
• 7 integrated image Mix (inpired by classics from UXS - Thanks Screech)
• Can also fetch raw media, including video, and soon HyperSpin themes and pdf manuals
• Maximum multithreading, depending of your ranking on ScreenScraper server (Get a ScreenSaver account!)
• Local cache, allowing to test multiple configurations without waiting hours or even days.
• Aimed at Recalbox (& Retropie - EmulationStation in general), but can be used to populate other front-end's meta-data also (Launchbox, ...)

Documentation available in french and english inside the .7z archive.

Post your feedback here or on IRC.

Quick test w/o RTFM:

Skraper.exe D:\roms\snes  "Super Nintendo"  -output=mix:4images -ssu=your_login -ssp=your_password

Next beta will support rb:<recalbox folder> as shortcut to set target system. Works with all recalbox rom folders except lua games and cavestory.
Replace "D:\roms\snes" with one of your own rom folder
For an arcade system, add the option -dnia (or -donotinspectarchives)

To go into details:

Skraper.exe D:\roms\snes "Super Nintendo" -output=mix:4images -ssu=your_ss_login -ssp=your_ss_password -v

In case of bug or unexpected error:

Skraper.exe D:\roms\snes "Super Nintendo" -output=mix:4images -ssu=your_ss_login -ssp=your_ss_password -v -logtofile

Then isolate the few line around the error in the log file and post them here.

When scraping on the recalbox SD directly, add -killrecalboxes to shutdown and reboot your recalbox atfer the gamelist.xml has been saved.

With the 4image mix, you'll get:

Support custom JSON Mix file. No documentation yet. Add -gmt to generate a JSON file template from an internal mix (4images in the example above) and play with it.
Wanna know more? RTFM

[Bkg2k]

Updated to Bêta 0.9.17315.101, available HERE

Release note:

• Added Recalbox folder as system identifiers. Use rb:<folder> as system shortcut. (ex: "Super Nintendo" => rb:snes)
• Added -killrecalboxes to kill ES before saving gamelist.xml, then reboot recalbox.
• Fix wrong command line parameters not raising any error.

[keltac]

@bkg2k

J'ai voulu tester, et j'ai rien compris ...

Je suis pas à l'aise dans ce domaine, alors j'attends vivement une interface, pour moi et ceux que cela pourra intéresser.

En tout cas, merci.

[Bkg2k]

Une interface est en cours, mais il faudra attendre encore quelques jours

A Windows/Linux GUI will be available soon.

[Bkg2k]

Hi all,

The Skraper GUI is on the way. The GUI use the same underlying scraping library, but offer a more visual approach for better understanding. It includes a Wizard to ease configuration of all the stuff for RecalBox, Retropie, Launchbox and even a Generic retroemulation profile.

Originaly designed in english, I need help to translate the whole project.
I'll do the french (france) translation and I've already a translator for portugeese/BRASIL.

I'm looking for translators, mainly for:

• German (de-DE)
• Spanish (es-ES or es-* if there are only a few difference with other spanish-speaking countries)
• Italian (it-IT)
• Dutch (nl-NL)
• pt-PT since there are many differences with pt-BR

Obviously, any other language translator is welcome!

For now, I cannot manage right-to-left languages, maybe later.

Translation will be done (and updated) through Excel .xlsx files, and translators will receive new bêta everytime a translation is updated.

Thanks in advance!

Here is a link to several SkraperUI screenshots, quite old now, but I'll update new picture later:
Skraper Scrrenshots

[paradadf]

@bkg2k I can do the spanish translation.

One request I have (haven’t been able to test your scraper yet though) is to upload all known hashes to SS automatically. Specially for large/cd based systems... screenscraper only calculates the crc when the rom file is big, thus large roms metadata is a bit incomplete for such.

Thanks for your work!

[NeeeeB]

@bkg2k

Regarde si tu peux utiliser un fichier .po - .mo pour tes traductions.
Plus simple à gérer que par xml, et tu peux du coup utiliser ce site qui est hyper pratique :
https://www.poeditor.com

[Bkg2k]

Le soft est en C#, donc les langues et les ressources associées sont gérée automatiquement par Visual Studio (et 2 plugin pour me mâcher le travail un peu plus).
L'import/export excel permet à tout un chacun de traduire et me permet à moi de réimporter les traductions en 2 clicks

[Bkg2k]

@paradadf You're welcome for the spanish translation

About automatic MD5/SHA1, sure that's a good idea. I can add this automatically for files less than a few megabytes, and add an options to force MD5/SHA for larges files.
The only limitation is for files in zip/7z/rar, because the scraper do not inflate files, it takes the CRC given by the archive instead.

[paradadf]

@bkg2k yeah, no need to inflate them. Usually, cd-based roms are unzipped anyway. It is just that I hate seeing those hashes missing. If you could add such a switch and have it deactivated by default, that’d be great.

[Bkg2k]

Hello all,

The new GUI skraper is in alpha stage, tested by a few people. It's almost complete and it should be released in bêta stage in a few days for Windows only. Linux & MacOS version will come later.
Also, the "old" console version will be replaced with a new one, using the new scraping library used in the GUI version: faster & more accurate.
Meaningwhile, you may want to check the development progress here: www.skraper.net

@paradadf : feel free to contact me on ScreenScraper if your translation proposal is still relevant

Also, Italian and German translated are welcome for both the application and the Web Site! (other languages too )

[Substring]

@bkg2k you should consuder poeditor (which means your project must be open source). Teabslations are extremly fast there

• Consider Mono compatibility. People don't only use windows ...

[Bkg2k]

@Substring Yes the project will be open source (as soon as I find a clean solution to remove my ScreenScraper's developer identifiers from the project ).

As I use Visual Studio with the appropriate plugin, languages resources are managed automatically in a fast, efficient and clean way.
Also, mono compatibility is planned, as the soft shoud be released on linux & macOS.

[Dan Donoghue]

@bkg2k said in [SOFT] New command-line scraper:

@Substring Yes the project will be open source (as soon as I find a clean solution to remove my ScreenScraper's developer identifiers from the project ).

As I use Visual Studio with the appropriate plugin, languages resources are managed automatically in a fast, efficient and clean way.
Also, mono compatibility is planned, as the soft shoud be released on linux & macOS.

That's always the hard part, Because screenscraper needs the password in each request. That's why I stopped working on my tool, got it to a point where all the admins could use it but we could never release it to the wild without my dev login being visible.

Edit: Hi bkg2k Blade Hunter here, havn't been around SS much since moving house in Dec :).

[Dan Donoghue]

Really nice looking front end on it, I have one question though.

For the image output folder I wanted to make it something like this: %ROMROOTFOLDER%%SYSTEM%\downloaded_images Is %SYSTEM% a variable it will recognize?

[Bkg2k]

@dan-donoghue Currently, Skraper replace the following varables:
%ROMROOTFOLDER% : The selected root folder you selected in the wizard.
%FRONTENDROOTFOLDER% : Selected front-end path in the wizard (only for Retropie & Launchbox).
%SCREENSCRAPERPLATFORM% : Internal ScreenScraper system name.

There is no variable for recalbox (or any other frontend) that map recalbox system folders. Actually those folders are hardcoded in the wizard (for ES-based front-ends)