[BATCH] Scrape your roms on your PC - fastscraper

[DavidinCT]

@paradadf I have a Home theater PC that I am using this on, It's based on Windows 7 because it's for TV watching as well (Windows Media Center). So I needed Emulation Station for Windows. Scraping has been a headache so if this works I wanted to use it. And the PC I am using is FAR more powerfull than a Pi is, as it's designed to be able to play 4K video.

I don't know, I set to raw, copy to a txt file and drop in the root of the rom folder, renamed to bat, it does not work. I tried it 2 times

I dropped one that you linked directly above and it seems to work. but, it does not allow me to set my LOGON info for that site.

I hope I am not missing anything by not setting the logon info.

[paradadf]

@DavidinCT ok, well... lets try to help you with that new info

Creating the batch isn't difficult. Just be sure to copy the raw content of the file in github. Always use the latest version of the batch. The console will turn cyan if you did it right.

About the windows version of ES... I don't know where the gamelists goes. Maybe you'll need to modify the paths if the don't belong next to the roms itself like on recalbox.
About your direct access and links or something, I suppose that could be a problem. So in the worst case, scrape directly the real folder and modify your gamelist.xml afterwards.

So... what is the problem again?

[DavidinCT]

@paradadf Yes, I know a batch file is not too hard. With out looking over it everything in the file, I am not sure. I have one that DOES run that I cant log into but, it is creating XML and images so, what ever on this one...as long as it works, right ?

I've been playing with this for a few hours now. I figured out the folder layout in Windows, and there is a

\downloaded_images
and
\gamelists

Folders with sub folders named for each system and deleing these folders, and re-running ES, they get re-created on the drive, so now I know what ones mean anything.

I copied my 20gb of roms on to the drive (correct folder layout) for this test, so I can rule things out.

When I run the fastscraper, I see it creates a gamelist.XML and a downloaded_images folder in the rom folder. So I have just copying them into the correct folder as they are laid out (overwriting the old XML file).

Problem is once I open Emulation Station, No changes show in there. Even after a reboot.

I figured the Meta data on ES would be the EASY part...in fact it's been the hardest part. Over a week working with itand 1/2 my games don't have meta data.

I was very excited to see another meta data option....just wish it was working for me

[paradadf]

@DavidinCT you should open the xml files and check the paths to the images are right.

Wich version of fastscraper do you have? As I said before, always use the last one.

[DavidinCT]

@paradadf 4.10.2016 is the one that works...17.02.2017 does not work (something I have to play with)

[paradadf]

@DavidinCT waaay to old!

Right click here and download it as file:
https://raw.githubusercontent.com/paradadf/recaltools/master/fastscraper/fastscraper.bat

[DavidinCT]

@paradadf It gets to ...

G:\EmulationStation.emulationstation\roms>reg Query "HKLM\Hardware\Description
System\CentralProcessor\0" | find /I "x86" 1>nul && set arch=386 || set a
rch=amd64

And just hangs....

[paradadf]

@DavidinCT weird... it must be because you use a nuc or something similar.

Can you open the command prompt and give the output of:

reg Query "HKLM\Hardware\Description System\CentralProcessor\0"

In the mean time, you can erase that part and just use:
set arch=386
Or whatever platform your pc is.

[DavidinCT]

@paradadf

Under a command line...
reg Query "HKLM\Hardware\Description System\CentralProcessor\0"
ERROR: The system was unable to find the specified registry key or value.

Manually checking that key Closest you get to AMD64 is "Intel64 Family 6 Model 60 Stepping 3"

[paradadf]

@DavidinCT hmmm... these new computers have weird registry entries XD
Try using the 386 arch as I said. Any 64 bit OS would run 32 bits apps, so no problems.

[DavidinCT]

@paradadf Think I found out what PART of my problem was....

I was using notepad...and word wrap was on... when saving it modified the file and caused errors (have not done that in years), Notepad++ only going forward... this gets me backwards tho...

Running as an Admin....
ERROR: The system was unable to find the specified registry key or value.
Can't get OS's language. English will be used.

from this point it just hangs here. Is there anything that needs to be enabled for this script to work ?

This is a HTPC, this is a very stripped down feature machine, only setup for media viewing. That I wanted to use some of it's horsepower for some classic gaming (funny it runs Wii and WiiU games fine too..LOL)

Thanks again for your help, I could understand if you were done at this point...

[paradadf]

@DavidinCT the registry entry the script isn't finding tells what arch (32 or 64 bit) your OS has, so it knows which version of the scraper to downlod. Just erase that prt of the code and let:
set arch=386

That should solve the problem.

[DavidinCT]

@paradadf Think I got it working. I did try to the set arch and I got stuck on the NEXT part...downloading...and stuck again...sigh

So, I pulled out a LOT of stuff..

-CPU check (changed to amd64)
-Lang check (changed to EN)
-Removed check version, download and extract (that whole section, as it was getting stuck on each part)

Now running it, it asked me the questions and seems to be running now.

Crossing fingers this works... remember my folder layout is not exactly common for this and before nothing would show up in ES...

Trust me, I really hope this all works... If you want me to up the mod that I did, let me know as it could help others with funky problems.

[paradadf]

@DavidinCT it might be better to run it from a real pc the next time XD
Probably the PowerShell functions where causing problems too.
If you can share how the paths within the gamelist.xml files should look like, that's be helpful for someone. Otherwise, good luck!

[DavidinCT]

@paradadf Real system ? I told you this system was designed as a media PC and on Windows 7 (that even updates ARE disabled on as too many problems over the years), I don't even think PowerShell is installed (I believe it's an optional feature in 7). I was not kidding !

Note, this system is not used for anything else, Not even a browser is opened on it unless it's for a driver, Even the host file is broken down so every ad site is blocked.

Anyway, the Atari 2600 finished up (250 or so roms), still the XML file and the Images are in the ROM folder, NOT in the \downloaded_images or \gamelists.(there are separate root folders for it)

I tried this before, leaving the XML in the folder with the images but, it didn't do anything or moving the files to the correct directories didn't do anything.

Here is something, I don't know how ES on windows processes information. Can it be manually imported ? What can I do with the xml to get it in the system with the according folder layout ?

sigh, so close...

Edit: finished up, looking over the Atari 2600 games, OMG it found like 99% of them, The built in thing only found about 15 out of 250ish. I really hope I can get this working !!!!

Edit 2: Oh, well.... Ran ES, No data updates keeping the files where they are (as your script created). Anyone else running the Windows version of ES using this and how did they get the data/images working correctly ?

The data and images are there, how can I use them ??

[DavidinCT]

@DavidinCT OK, I think this needed it's own post. I did get it WORKING and working in ES on Windows...paradadf you have been moved to a GOD status in my book !

FIRST CLOSE ES and MAKE SURE it's not running in the task manager.

Ok, When this script runs on Windows, it creates in the folder in the roms folder for each system. For example.

Roms\nes
Roms\nes\downloaded_images (Image store)
Roms\nes\gamelist.xml (game list and meta data)
and
Roms\nes_nes_missing.txt (roms that were not found)

In the Windows version,
There is a few folders where this data is stored. Starting in the root of the <user> folder by default.

C:\users<user name>.emulationstation
C:\users<user name>.emulationstation\roms
C:\users<user name>.emulationstation\downloaded_images
C:\Users<user name>.emulationstation\gamelists

In each of the ROM, Downloaded_images and gamelists, there are sub folders for each system, named accordingly, like nes, snes, atari2600, etc. So they should all line up.

Now looking over the NEW gamelist.xml that is in your ROM folder...here is an example...
<path>./Castlevania 64.n64</path>
<name>Castlevania (USA)</name>
<desc> (SNIPPED FOR EXAMPLE).</desc>
<image>./downloaded_images/Castlevania 64-image.jpg</image>

The problem is, the image store is and as this is n64 game...
C:\users<user name>.emulationstation\downloaded_images\n64

So you would need to change that to, again example
~/.emulationstation/downloaded_images/n64

With notepad++ you can do a easy search and replace. save the file, drop in the correct folder (.emulationstation\gamelists<system name>) and over write the old one, then CUT all the images out of the downloaded_images folder (in the rom folders, and move them to the official store .emulationstation\gamelists<system name>)

Once this is done restart ES and enjoy your new images.

Hope this helped someone !

[paradadf]

@DavidinCT I meant to just scrape using a real pc and then transfer it over. You could even scrape over the network if your roms are on a network share. Anyway, glad it worked. You must close ES before editing the gamelists, I thought that was obvious, sorry I didn't mentioned it.

[DavidinCT]

@paradadf I'm just a Windows user, so thanks for your help again. After moding your script to fit my needs. I didn't even know what a Recallbox was till this morning. Sorry about that.

The machine is a HTPC, and for the roms, I dropped in a SSD so I get performance on them, they are in the local machine. By just dropping things in the ROM folder, it saves me a hassle anyway, as my roms are NOT on the drive. I just dont like how I see the desktop when loading a rom from ES...sigh, nothing I can do about that one.

I did need to edit the XML for it to work (it's nice too, as I can edit the name MUCH easier than in ES),

Going through all the NES Roms for example with the built in thing (with 46 roms), it would take hours and time out 1/2 the time, so about 50% of them had no data (once it times out, it's done till you do a new search). With this script, it found 99% of them (a few not found ones but, those I can do manually or find a new rom) and only took about 45 seconds...

Loving it... In my tool box for life...

[paradadf]

@DavidinCT I only created the script, but the credit for the scraper goes to sselph.
Also, take a look for the Universal XML Scraper if you don't know it yet ;).

[DavidinCT]

@paradadf said in [BATCH] Scrape your roms on your PC - fastscraper:

@DavidinCT I only created the script, but the credit for the scraper goes to sselph.
Also, take a look for the Universal XML Scraper if you don't know it yet ;).

What is it and what does it do ? I'm using Notepad++ (Notepad on steroids one of my fav apps)

Thanks again for your help !