[BATCH] Scrape your roms on your PC - fastscraper

[rsn8887]

Worked two days ago, now broken:
ERR: error processing H:\EmuRoms\Recalbox\recalbox\roms\gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip: 504 from $s%!(EXTRA string=http://mirror.screenscraper.fr/api/mediaJeu.php?crc=&devid=&devpassword=&jeuid=83014&maxwidth=400&md5=&media=ss&mediaformat=png&sha1=&softname=sselph%2Fscraper&systemeid=12)
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x14 pc=0x408fe7]

goroutine 31 [running]:
panic(0x7e9580, 0x1137e030)
/usr/local/go/src/runtime/panic.go:500 +0x331
main.crawlROMs.func2(0x120b29d0, 0x11dcb540, 0x0, 0x120b29c0, 0x1287e0c0, 0x1287e040)
/home/sselph/go/src/github.com/sselph/scraper/scraper.go:295 +0x527
created by main.crawlROMs
/home/sselph/go/src/github.com/sselph/scraper/scraper.go:328 +0xbd7

[paradadf]

@rsn8887 I read on the UXS thread that gba was having problems the last couple of days... Do you still have this problem? Does it happen with any system or only gba?
I've also encounter some single roms which had wrong data on SS and thus "broke" the scraping process.
So please,

• try to scrape gba again
• try scraping any other system to see if you still get an error
• if other systems work and if the problematic rom is one of the first ones, try removing it. (And give me the name)

I'n not on my computer right now, so I can't test.

Update: This is what I had read

https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/issues/142

[rsn8887]

@paradadf
Other systems show the same problem, but some don't. For example gb scraped fine completely through all roms, but gba and n64 didn't. I tried to remove a few roms where the error appears, but then it just happens at a later rom. Judging from that 504 error message, it seems to have something to do with mirror.screenscraper.fr . As I said it all worked perfectly on the weekend a few days ago.

[paradadf]

@rsn8887 sselph's scraper uses no mirror. It has something to do with the database and not with the scraper itself. There is nothing I can do. You can complain about it on IRC XD

Update: I just tried it and saw the mirror thing you mentioned. But no idea what that is! I'll talk with the people there and ask.

Update 2: some kind of dns problem. They already are aware of it and are working on it. So just be patient

[screech]

Normally it's OK now

[paradadf]

@screech thanks for the heads up!

[rsn8887]

Yup it works just fine again

[rsn8887]

Well, I was a bit fast now there are problems with N64 scraping. It gives "unknown image format" errors. Worked fine 2 hours ago.

"GDB" scraping works fine, so it is again some problem with "ss" (maybe ss ist just inherently unstable?)

[paradadf]

I just think they are having dns problems lately. Personally, I'd rather wait until it is working again (usually just a matter of hours) instead of using other sources. No other database can compete with SS in that sense. And gdb is definitely not more stable

[faxesystem]

I have the following problem. When setting 3dboxart as prefered cover art it get's downloaded when scraping for a single system. When I choose all 3dboxart gets scraped for gba but not for any other system. For all other console systems the 2dboxart gets downloaded. Can anyone confirm this?

Greetings,

Faxe

[paradadf]

@faxesystem have you changed consoleImg=-console_img= to "3b" (mameImg=-mame_img="3b" for neogeo)?
Only ScreenScraper retrieves 3d boxes, so you better don't change the source. And important... use .png for 3d boxes, otherwise you'll get a black background.

[DavidinCT]

I tried this and so far getting no where.... I have maybe 500 roms... The Emulationstation folder is in the user path (c:)but, the roms are located on another drive(g:), so it's kind of broken up trying to keep the ROMS off the drive. With this the system works perfectly fine (config files are set right). I dropped this batch file in the root of the rom folder.

The only problem when I run this, I just get a message of
Error: The system was unable to find the specified registry key or value, Cant get the PS's language. English will be used.
And a blinking _

it's been sitting there for over an hour like this, never asked me to do anything. Am I doing something wrong ? Or should I be doing something different for my setup ?

[paradadf]

@DavidinCT what do you mean by "the emulationstation folder"?
What is your OS's language?
What windows version do you have?

[DavidinCT]

@paradadf

Windows 7 x64

The install is at C:\users<user name>.emulationstation

In the config file, I set the roms and emulators to point to a G:\Emulationstation.Emulationstation

So, emulators and roms are not on the OS drive. Like this it does not work (as images and XML files are on the drive). I copied the ROMS over to the drive (as a temp thing)

I ended up downloading your one that points to screenscraper.fr and it seems to be working, I think. It's giving an error that API not available to NON-MEMBERS. The site is in French and I cant figure it out to setup an account.

I could not find a direct download on
https://github.com/paradadf/recaltools/tree/master/fastscraper

So, I copied the text and created a batch file, pasted it in...maybe some errors (I viewed as raw).

Sigh....

[paradadf]

@DavidinCT you have flags on the left corner of the site in order to change the language of screenscraper.

Anyway, I don't understand. You installed EmulationStation for Windows? Why? Are you a recalbox user? Or what do you intent to do?

(Yes, you can copy the raw code into an empty txt file and change its extension to bat. If you did it wrong, the script won't even start properly. I would say it is pretty evident if everything is right)

[DavidinCT]

@paradadf I have a Home theater PC that I am using this on, It's based on Windows 7 because it's for TV watching as well (Windows Media Center). So I needed Emulation Station for Windows. Scraping has been a headache so if this works I wanted to use it. And the PC I am using is FAR more powerfull than a Pi is, as it's designed to be able to play 4K video.

I don't know, I set to raw, copy to a txt file and drop in the root of the rom folder, renamed to bat, it does not work. I tried it 2 times

I dropped one that you linked directly above and it seems to work. but, it does not allow me to set my LOGON info for that site.

I hope I am not missing anything by not setting the logon info.

[paradadf]

@DavidinCT ok, well... lets try to help you with that new info

Creating the batch isn't difficult. Just be sure to copy the raw content of the file in github. Always use the latest version of the batch. The console will turn cyan if you did it right.

About the windows version of ES... I don't know where the gamelists goes. Maybe you'll need to modify the paths if the don't belong next to the roms itself like on recalbox.
About your direct access and links or something, I suppose that could be a problem. So in the worst case, scrape directly the real folder and modify your gamelist.xml afterwards.

So... what is the problem again?

[DavidinCT]

@paradadf Yes, I know a batch file is not too hard. With out looking over it everything in the file, I am not sure. I have one that DOES run that I cant log into but, it is creating XML and images so, what ever on this one...as long as it works, right ?

I've been playing with this for a few hours now. I figured out the folder layout in Windows, and there is a

\downloaded_images
and
\gamelists

Folders with sub folders named for each system and deleing these folders, and re-running ES, they get re-created on the drive, so now I know what ones mean anything.

I copied my 20gb of roms on to the drive (correct folder layout) for this test, so I can rule things out.

When I run the fastscraper, I see it creates a gamelist.XML and a downloaded_images folder in the rom folder. So I have just copying them into the correct folder as they are laid out (overwriting the old XML file).

Problem is once I open Emulation Station, No changes show in there. Even after a reboot.

I figured the Meta data on ES would be the EASY part...in fact it's been the hardest part. Over a week working with itand 1/2 my games don't have meta data.

I was very excited to see another meta data option....just wish it was working for me

[paradadf]

@DavidinCT you should open the xml files and check the paths to the images are right.

Wich version of fastscraper do you have? As I said before, always use the last one.

[DavidinCT]

@paradadf 4.10.2016 is the one that works...17.02.2017 does not work (something I have to play with)