[SOFT] Universal XML Scraper V2

[azvarel]

@acris The same thing happened to me now!

[azvarel]

@screech
It's impossible to do scrap mix 3 img I lost the day and has not yet made 1651 roms of NES and is in 1/3 of what is missing and still says that it finishes in 6 hours

[acris]

@screech
je ne comprends pas, j'ai enfin réussi à scapper mes roms advmame (parent only avec neogeo) en choissant mame, cependant les roms neogeo ont pas été scrappé (logique)
je recommence en choisissant ajouter les roms manquantes dans la config général puis je choisis neogeo et là
il trouve rien enfin si les images c'est bon mais au niveau des noms rien, et il a supprimé ce qui était déjà existant.
Logiquement il aurait dû laisser le gamelist.xml et ajouté uniquement ce que je lui demandai

<game id="" source="ScreenScraper">
	<path>./005.zip</path>
	<name> (CLONE, BETA, DEMO, PROTO, TRAD, HACK, UNL)</name>
	<desc/>
	<image>./downloaded_images/005-image.png</image>
	<image/>
	<rating>0</rating>
	<releasedate>00000000T000000</releasedate>
	<developer/>
	<publisher/>
	<genre/>
	<players/>
	<region/>
	<hidden/>
</game>

[Pepe]

@screech, @cotedeporc From what I have been able to verify, modifying the parameters of pngquant in config.xml of the Foolscreen mix does not work. It seems as if the foolscreen mix does not get to use pngquant or even it may not be using by UXS. I have modified the settings in the config.xml of each foolscreen mix, I have scraped and just like before the change, there has been no compression in the png. I have passed pngquant by the result with the same parameters that I put in the config xml and compresses the images leaving them to 20%-25% of its weight. It seems that Foolscreen or UXS are not doing all their work well.

@cotedeporc On the other hand, in the animated gifs you show, there are mix images that do not exist in UXS: Vectrex, PSP, NGPC etc ...

[screech]

@acris
De ce que je vois tu as du cocher "ajout des roms manquantes"
ça te créer ce genre "d'entrée" vide dans le fichier XML
Du coup quand tu passe en "Mise à jour" il ne rescrape pas ces roms (qui existe déjà dans ton fichier XML mais vide)
Reessaye sans cocher cette option dans ton scrape de départ. (En toute logique, les rom NeoGeo sont des enfants des roms Mame... donc il devrait les trouver aussi :S)

Tiens moi au courant.

@Pepe
Where did you make your modification ?
You need to do it in the ZIP file.
(For exemple, open one MIX profil with winzip, open the config file, do your modification, save, and update the zip file)
If you do this directly in the MIX folder (not in the zip) every time you open UXS, it will update the Folder by unziping the ZIP file. So you loose you modification.

I did some try, and pngquant work well... so it must work this way

[Pepe]

@screech The config.xml files that I have modified, have been in the compressed file of the mix, I have decompressed them, modified the config.xml and compressed it again but foolscreen does not compress the png. I say that I do not know if it is a foolscreen or UXS issue for that reason I mentioned @cotedeporc and you.

[screech]

If it doesn't work, I think it's a UXS problem :S

Can you copy past your modification ?

normally, the standard thing is :
<Compression>
<use>yes</use>
<soft>pngquant.exe</soft>
<parameter>--force --verbose --ordered --speed=1 --quality=30-90 --ext .png</parameter>
</Compression>

So, UXS transform that in :
xxxPathtoUXSxxx\Ressources\pngquant.exe --force --verbose --ordered --speed=1 --quality=30-90 --ext .png xxxfiletocompressxxx

If you take a look at the logx.txt in the log folder, you must see a line like :
[12:05:44] - > PNGQuant command: "C:\Developpement\Github\BIN\Ressources\pngquant.exe" --force --verbose --ordered --speed=1 --quality=30-90 --ext .png "C:\Developpement\Github\BIN\TEMP\MIX3\MIX\screenTEMP.png"
[12:05:45] - PNG Optimization (PNGQuant): C:\Developpement\Github\BIN\TEMP\MIX3\MIX\screenTEMP.png(416 KB -> 110 KB)

(I just did it to have an exemple ^^)

[cotedeporc]

Hello,

I think, but I can't verify it yet, that I disactivate the PNG compression for the FoolScreen Mix.

If you put, as @screech says:
<Compression>
<use>yes</use>
it will work.

[Pepe]

@cotedeporc, @screech I apologize for my poor eyesight. In effect compression is disabled in foolscreen Mixes. Corresponding with what I noticed that foolscreen was not compressing the png. I do not understand how I escaped that detail when I modified the compression factor in the mix. I am modifying the mix again to enable compression, when I have been able to scrape again, I will comment the result, which I hope is the correct one.

[Pepe]

@Pepe said in [SOFT] Universal XML Scraper V2:

@cotedeporc On the other hand, in the animated gifs you show, there are mix images that do not exist in UXS: Vectrex, PSP, NGPC etc ...

[cotedeporc]

@Pepe,

If I'm right, I send to @screech, to add this missings Mix to UXS (via GitHub).

@screech,
Is it normal that the new Mixs are not available? It needs a new version?
Maybe I post them in the wrong repo?

[screech]

Oups !! My fault...

I had added the MIX, but haven't updated the list and preview...

It's done

[screech]

@cotedeporc Je me suis permis de modifier "légèrement" tes descriptions en y ajoutant un lien sur le sujet de ton Thème

Du coup quand on dl un des mix, dans la fenêtre de sélection, il y a un bouton "Link" en plus qui renvoi sur le fofo

J'ai aussi remis la "source d'inspiration Kent79" présente dans les previews mais plus dans la description... Meme si aujourd'hui ça en est loin, ça fait pas de mal de cité l'inspiration ^^ (l'image de la borne au minimum est de lui ^^)

(ça m'a d'ailleurs permis de voir qu'il y a un bug sur le module de download des MIX : pas de description et l'image empty est identique à la full... faut que je corrige ça :S )

[cotedeporc]

@screech

Merci pour les modifs/ajout!

[Pepe]

@cotedeporc, @screech, tested and working everything correctly, thank you guys.

[Pepe]

@screech said in [SOFT] Universal XML Scraper V2:

As you see you can put your own parameter and even choose an other software like optipng as post action

@screech This comment makes me think that I could do a second pass on the images in this case with optipng. Have I understood correctly?
If so, what would be the way? Repeating the compression section in config.xml?, or just adding another compressor inside the compression section?

[screech]

Sorry @Pepe It's only one post traitment...

But you can choose... pngquant OR optipng OR what you want

But I think you can do pretty the same thing with optipng and pngquant... Only Algorythm are different. (I choose pngquant because it is a little faster than optipng)... But it's not my speciality ^^

If you found a nice "configuration" for one or the other, let me know. Maybe I can change the standard.

[Pepe]

@screech Thanks.
It's better pngquant, especially in terms of speed and color reduction, but a combination of two would be perfect for what I'm looking for.
Pngquant does not eliminate / compresses metada what can be between 1% -5% of final weight reduction that added to the reduction of size obtained with pngquant already leaves an interesting file. The possibility of performing a second compression with optipng at the scraping time from uxs and not having to do it manually later, at least for me, would be helpful, it would be less painful to have all systems scraped with 1920x1080 images and an acceptable file size . Your comment made me see the light, but I see that I have misunderstood it, a pity, I will continue doing it manually.

[screech]

I'm just thinking :

Maybe you can make a BATCH file

something like :
pngquant --options %1
optipng --options %1

And put your batch as the post traitment

never tested that but it may work

[Pepe]

@screech It's a good idea, I'll try it. Thanks again.