[BATCH] Scrape your roms on your PC - fastscraper

[DavidinCT]

@DavidinCT OK, I think this needed it's own post. I did get it WORKING and working in ES on Windows...paradadf you have been moved to a GOD status in my book !

FIRST CLOSE ES and MAKE SURE it's not running in the task manager.

Ok, When this script runs on Windows, it creates in the folder in the roms folder for each system. For example.

Roms\nes
Roms\nes\downloaded_images (Image store)
Roms\nes\gamelist.xml (game list and meta data)
and
Roms\nes_nes_missing.txt (roms that were not found)

In the Windows version,
There is a few folders where this data is stored. Starting in the root of the <user> folder by default.

C:\users<user name>.emulationstation
C:\users<user name>.emulationstation\roms
C:\users<user name>.emulationstation\downloaded_images
C:\Users<user name>.emulationstation\gamelists

In each of the ROM, Downloaded_images and gamelists, there are sub folders for each system, named accordingly, like nes, snes, atari2600, etc. So they should all line up.

Now looking over the NEW gamelist.xml that is in your ROM folder...here is an example...
<path>./Castlevania 64.n64</path>
<name>Castlevania (USA)</name>
<desc> (SNIPPED FOR EXAMPLE).</desc>
<image>./downloaded_images/Castlevania 64-image.jpg</image>

The problem is, the image store is and as this is n64 game...
C:\users<user name>.emulationstation\downloaded_images\n64

So you would need to change that to, again example
~/.emulationstation/downloaded_images/n64

With notepad++ you can do a easy search and replace. save the file, drop in the correct folder (.emulationstation\gamelists<system name>) and over write the old one, then CUT all the images out of the downloaded_images folder (in the rom folders, and move them to the official store .emulationstation\gamelists<system name>)

Once this is done restart ES and enjoy your new images.

Hope this helped someone !

[paradadf]

@DavidinCT I meant to just scrape using a real pc and then transfer it over. You could even scrape over the network if your roms are on a network share. Anyway, glad it worked. You must close ES before editing the gamelists, I thought that was obvious, sorry I didn't mentioned it.

[DavidinCT]

@paradadf I'm just a Windows user, so thanks for your help again. After moding your script to fit my needs. I didn't even know what a Recallbox was till this morning. Sorry about that.

The machine is a HTPC, and for the roms, I dropped in a SSD so I get performance on them, they are in the local machine. By just dropping things in the ROM folder, it saves me a hassle anyway, as my roms are NOT on the drive. I just dont like how I see the desktop when loading a rom from ES...sigh, nothing I can do about that one.

I did need to edit the XML for it to work (it's nice too, as I can edit the name MUCH easier than in ES),

Going through all the NES Roms for example with the built in thing (with 46 roms), it would take hours and time out 1/2 the time, so about 50% of them had no data (once it times out, it's done till you do a new search). With this script, it found 99% of them (a few not found ones but, those I can do manually or find a new rom) and only took about 45 seconds...

Loving it... In my tool box for life...

[paradadf]

@DavidinCT I only created the script, but the credit for the scraper goes to sselph.
Also, take a look for the Universal XML Scraper if you don't know it yet ;).

[DavidinCT]

@paradadf said in [BATCH] Scrape your roms on your PC - fastscraper:

@DavidinCT I only created the script, but the credit for the scraper goes to sselph.
Also, take a look for the Universal XML Scraper if you don't know it yet ;).

What is it and what does it do ? I'm using Notepad++ (Notepad on steroids one of my fav apps)

Thanks again for your help !

[paradadf]

@DavidinCT just do a simple search on this forum and find it out.

[levendis32]

@paradadf Hello I have move my ROMS to PC but I cant find the .bat file to download it from github.The link I have is this https://github.com/sselph/scraper/releases but where is the bat file I dont know. I know that this is a 1st class NEWBE question but please show me the obvious because I m STUCK

[paradadf]

@levendis32 you should always read the very first entry on any topic. That's where usually the instructions are

Just right click and download as...
https://raw.githubusercontent.com/paradadf/recaltools/master/fastscraper/fastscraper.bat

fastscraper automatically downloads sselph's scraper.

[gmgman]

I like your script, it's pretty effective. I have one request, for game naming: is it possible to extract game names from their file, without having the bracket stuff and the file ending in it? e.g. if my SNES rom is called "Ys V - Kefin, Lost Kingdom of Sand (J) [T+Eng 1.00 Aeon Genesis].zip" your script either make the game names like the file itself or takes the name from the Database which will turn into "Ys V: Ushinawareta Suna no Miyako Kefin", I'm not happy with either of those and would simply prefer "Ys V - Kefin, Lost Kingdom of Sand", substring from file name before the first bracket. maybe you can add an option to set the game name stripped from the filename without the stuff in the brackets and without the .zip etc. ending? just a suggestion...

[gmgman]

@paradadf looks like your script doesnt add file endings already, but I would prefer to not see the stuff in (...) and [...] in the game name

[paradadf]

@gmgman the name convention doesn't come from my script but from the scraper itself. But I can try to add that option for you. I'm pretty busy right now though, so it might take me a while.
The only problem I see is that you might end up with many duplicates if you don't clean up your romset properly.
I'll let you know when the feature is implemented.

[gmgman]

@paradadf thanks for the quick response! yeah, I try to have my rom collections clean (1g1r) so I don't worry about dupies. dupies also appear if i get the game names from the database, e.g. Illusion of Gaia (USA) and Illusion of Time (Europe) both end up as Illusion of Gaia if I use the DB naming option.

Thanks to everyone involved in the recal box project. this is so much better than retropie!

[gmgman]

@paradadf. I was just thinking, maybe it's wiser to keep square brackets [...] in the game name, or maybe let the user set which brackets they want to remove from the game name?

[paradadf]

@gmgman hahaha I don't know if ai can do that easily in a batch script, but I'll try.
Maybe it would be just easier to use notepad++ to delete specific strings all at once... lets see.

[gmgman]

@paradadf the most part which I find annoying in the game name is the country codes and the language codes (En,Fr,Es) etc.. sure I could use notepad++ and search for strings to replace in the gamelist.xml file but then there are so many different combinations depenting on the rom set and I would have to make sure to only edit where <name>..</name> tags are and not remove them in the filename tags etc. which will make it more complicated. well it's all just an idea, no need to rush on this idea tho, I stick to the file name for now because I rather see the titles as I named the rom files instead of what the DB offers.

[fullskull]

Holis...eem...yo er...tengo un problemo con el fastscraper. Yeske, lo pongo en la ruta de las roms y cuando ejecuto el .bat desde windows, salta una ventana de cmd y se cierra al segundo de abrirse. ¿Que puede estar pasando?

[paradadf]

@fullskull probablemente el contenido del batch que descargaste no sea el correcto. Abre el archivo con algun editor de texto, copia y pega el contenido RAW de github.

[fullskull]

@paradadf ya hice de nuevo el archivo bat y ahora si parece hacer otra cosa, pero la otra cosa que hace es mostrarme un error.
[img]http://i.imgur.com/0vH8AkK.jpg[/img]

[paradadf]

@fullskull tu instalacion de windows debe ser minima y le faltaran algunos datos del registro. De todas formas deberia ponerse en ingles y funcionar sin problemas.

[fullskull]

@paradadf si lo hago desde macintosh, ¿crees que me podria ir mejor? tambien tengo linux ubuntu pero no se como hacer para ejecutarlo desde ahi.