perfectplex PerfectPlex's File Surgery.

[Perfectplex]

This thread is dedicated to my work with hex editing, mainly yobj files but its basically my little experimentation lab for things I play around with prodding and poking at files to see what can be achieved with a bit of trial and error.

I have been working for some time trying to understand the layout of the yobj files and the files therein.

I am simply trying to back engineer the yobj first then rebuild the hex by grabbing an understanding of the references.

The basic aim is to be able to build yobj files from scratch but for that we need to break down and understand each part and what needs to be modified when rebuilding one.

What do these numbers on the bottom of the header files mean?

 

Any idea what these 8 bytes are for and what they are referencing?

 

Can anybody decode the pofo part?

 

Also any help on filling this list would also be very welcomed. 

General snooping into the files which may help others.

Examples of X-Rey Info

 Breaking down the very top of the file.

Breaking down the header portion. 

Here is a simple 4 vertice plane object.

A much larger object and finding the references.

Material calling

Restructuring a material build, step by step manually.

and finished, notice they now call the direct offsets to each material.

 

Edited September 11, 2016 by Perfectplex

[kim666]

@Keshire started working on yobj files last year.  

https://github.com/Keshire/WWE2K15-Tools/blob/master/structures/WWE2K15-JBOY.hsl

[Perfectplex]

Thanks for the suggestion kim666 but it doesn't appear to be what I am looking for, that appears to be some form of script which is an abandoned WIP with little information.

The structure is yobj start chunk > headerinfo > objectinfo > skeleton > material assignment > end pofo chunk.

What I am trying to do is be able to 'cut out' objects from one yobj character and create its own individual file by rebuilding the hex code.

Example, I want to take a kneepad from The Rock, I want to be able to find the object in the hex code only relating to that item and cut off all the other body parts and rebuild it as its own individual yobj file.

Then if possible be able to increase the vert count to a set number of verts after modifying it (lets say it adds a few more verts) and be able to reinject the obj file without X-Rey giving me the error about the vert count being too high.

Or be able to open up one yobj object, inject a new obj with more verts and import a new header allowing for the larger vert count.

Edited July 28, 2016 by Perfectplex

[4 Life]

Would be awesome to know this stuff. Could make importing objects from other games work, if they were merged with a 2K16 object for example.

[Perfectplex]

My reasoning is this, I worked out how to rebuild a pach from scratch by creating a new pach then rearranging the top of the hex file of the pac into Hex 'alphabetical' order then reinjecting each file one by one from the top it would create a perfectly sized pach file to inject into a dummy pac without losing any functioning (physics etc).

I also found that you can change the offsets manually and pad out the file with 0 bytes to be able to fit new files directly in the gap and changing the offsets in the header.

You can read more in this incomplete tutorial I was writing.

Pac Hex editing:
-----------------
0000 (0)        45 50 4B 38 = EPK8 (file type label)
        1C 00 00 00 = Number of Pach's
        00 D0 A6 03 (61,263,872) = + 4000 (Signature Location)
        07 00 00 00 = 

0800 (2,048)    Tag location for the PACH ID
        45 4D 44 20 = PACH type
        04 00 01 00 = 
        00 00 00 00 = 
        XX XX XX XX XX XX XX XX = Pach ID label (eg: 000205402)
        00 00 00 00 = 
        D0 A6 03 00 = Signature (strange format)

4000 (16,384)    PACH start

XXXX        Signature, 800 bytes, pad with zeros.

######################################################
Pach Hex editing:
------------------
50 41 43 48 = Pach
03 00 00 00 = Number of files

Name    Starting offset    Size        Starting offset    Size
-----------------------------------------------------------------------------
00    00 00 00 00    1C E6 12 01    0        18,015,772
0A    1C E6 12 01    10 C8 27 02    18,015,772    36,161,552
0C    2C AE 3A 03    F0 41 6A 00    54,177,324    6,963,696

Now we want to add a file. This file will be called CA/00CA/000000CA.

OK first we must make the pach read the new file.

Change the number of files line to 04 00 00 00.
Now we need to add the 3 lots of 4 bytes (XXXX XXXX  XXXX XXXX  XXXX XXXX).
So head to the end of the last byte before the first file is read (*JBOY).
Insert not overwrite!
Now add CA 00 00 00 for the file name
Now we need to add the previous file size to the previous offset for the new offset to start on.

54,177,324 + 6,963,696 = 61,141,020
61,141,020 in Hex is 3A4F01C Now we need to reverse the bytes.
1C F0 A4 03
So we now have CA 00 00 00   1C F0 A4 03

Now we need to just add the file size.

Open up 00CA with Hex Editor and click on the end (after the very last byte) and check the hex and decimal in the boxes near the top of HexEdit for the file size.
We see that the hex is 1 DBC0 in the Hex box and 121,792 in the Decimal box.

Head back to the pach tab we are making.

So with reversed bytes this is C0 DB 01 00
Now we have
CA 00 00 00   1C F0 A4 03   C0 DB 01 00

Now Check that the last files size does not exceed the end offset.
Add the size of the last file onto the current running Size total.
1C F0 A4 03 + C0 DB 01 00 = 3A6 CBDC
or 61,141,020 + 121,792 = 61,262,812

However we have to take into account the bytes BEFORE the first file.
So before the JBOY (in this example) we have the following

50 41 43 48   04 00 00 00   00 00 00 00   00 00 00 00
1C E6 12 01   0A 00 00 00   1C E6 12 01   10 C8 27 02
0C 00 00 00   2C AE 3A 03   F0 41 6A 00   CA 00 00 00
1C F0 A4 03   DB C0 01 00

Which is 38 HEX bytes (56 in decimal).
So now we add this number to the summed up offset to find the Actual End Offset.
Hex : 3A6 CBDC + 38 = 3A6 CC14
Dec : 61,262,812 + 56 = 61,262,868

So pad the file out with zeros (00 00 00 00) until the empty end (after the last byte) is exactly on 3A6 CC14 (61,262,868).

This creates the exact byte space required.
Now you can inject the file into place via X-PacKer.

######################################################

Now we are going to insert a file between other files.

The file we are inserting is 3B and we need to insert it before the CA file we just installed.

As we did last time the first thing we need to do is increase the number of files in the pach to be read.
Therefore we increase from 04 to 05 in the line 04 00 00 00.

Now we need to insert the 3 lots of 4 bytes after the 0C file and before the CA file.
So lets take a look where it goes

0C 00 00 00   2C AE 3A 03   F0 41 6A 00   XX XX XX XX   
XX XX XX XX   XX XX XX XX   CA 00 00 00   1C F0 A4 03   
C0 DB 01 00

So first is the file name 0C 00 00 00
Next is the offset : 
Because we are inserting a file this now upsets all the offsets for the placements for each file that comes after it IF we choose to insert the Hex code in order.
*

Now we need the file size.
As before we open up the file we are inserting with hex editor and go to the very end after the last byte and read the byte we are on for the file size, in this case it is 144 in Hex and 324 in Decimal.
So go back to the PACH tab and in the file size bytes insert this as 44 01 00 00.
This has now upset the offset of any files thereafter, in this case just the CA file.
So we have to now adjust the offset of the CA file by working out the new offset.
So we take the offset of the inserted file and add its file size.
Highlight the offset and choose calculate using then the plus button, highlight the size and hit calculate using again then equals for the result.
This gives us 3A4 F160 which means the new offset for CA is 06 F1 4A 03.
So now we have:
3B 00 00 00   1C F0 A4 03   44 01 00 00
CA 00 00 00   60 F1 A4 03   C0 DB 01 00

To jump to this in the address you will need to add the bytes count before the first file.
We have used 44 bytes (68 in Dec).
Highlight the offset in 3B which is 1C F0 A4 03 and hit calculate using and add 44 (hex) then press equals to show the actual offset and hit GO button.

We are now at 3A4 F060 (61,141,088).
We can now post the new code in here or zero it out.
If you choose zeros you will have to keep adding 00's until you fill the exact ammount of bytes for insertion.
Easier however is to highlight the entire hex of the 003B file and copy it, then past (insert not overwrite) the block of code.

Our new overall PACH file size is 3A6 D150 (61,364,208).

Open up X-PacKer and inject the 3B file into the 003B slot we just created.

For files that end on an incomplete 4 byte block pad it out with 0 bytes'
eg; File 3C ends on "0D 0A 2A 0D   0A" so change it to "0D 0A 2A 0D   0A 00 00 00".

and remember to take into account these extra bytes when it comes to the next files offset start.

###########################################

The easier method is just to rename the extensions in the order you need them and when you are creating pach just order by type and it will arrange them in the correct order ready to insert the pach without any rearranging in hex editor.

then if you need to slip a file in a certain place (lets say I want to add a file between 15 and 16 you can just name it .15b and it will then be in the correct build order with the new file in the correct position or you can just renumber them in order one by one.

That obviously isn't a clear instruction but basically it means I can take a pach and add a new dummy file into it via hex editing by creating the spacing and altering the offsets then I can inject directly into that dummy file in the pach perfect to the byte.

Therefore I am hoping that there is some way to extend the vertice count in the headerinfo portion of the pac and be able to do something with the hex offsets to be able to inject a larger (vert count) obj file.

If Tekken is reading this could you possibly add a function for setting the number of verts manually and fooling X-PacKer into thinking the vert count is correct so we can inject a larger object? Maybe with a warning that if we inject an object with the incorrect number of verts (not in line with what we inputted) that it will corrupt the yobj file. So we need to use blender to see how many verts the object has, then input that into the input box, then inject that new obj file and it will save out with the new size with the required adjustments to the header, or even just make us have to adjust the vert count in the header, import that then inject the new object for it to rebuild the yobj file with the new offsets and info etc.

Edited August 5, 2016 by Perfectplex

[4 Life]

That last paragraph is a fantastic idea, if possible, but then changing the number of verts drastically may affect how the object looks. I've yet to try this to see if it works correctly in-game, but in Blender there are options to merge or slice verts to create less or more, respectively, though obviously if the object has a significantly larger number than the object you're trying to inject it into then you might have problems trying to get the vert count down without it looking shitty if it's a particularly 'curvy' object.

Edited July 28, 2016 by 4 Life

[Perfectplex]

Updated first post with new findings.

Any help to decipher all the files would really be appreciated.

I'd really like help trying to find out why file 0050 clashes with file 00CD (at least in Mr Perfects case).

This is while they are unmodified too.

To test, decompress all from pach, use the renumbering method to rebuild a new pach with the files, you will find it crashes when choosing Mr Perfect, then you can rebuild and you must leave out 0050 or 00CD for it to not crash, however not including 0050 means if Perfect has a title it wont wrap it around his waste but rather will be up his bottom with the strap hanging out 

It also means leaving out a file will result in the game crashing when loading Mr Perfect in creations.

 

Edit: Fixed the problem with Perfect, I must have somehow screwed up his base file, luckily I have a back up of all of them twice.

Still crashes in creation suite though but I'm not bothered really because creation suite isn't really needed for people who mod/use mods.

Edited August 6, 2016 by Perfectplex

[Perfectplex]

As I mentioned in @Bannockburns thread here

I wish to be challenged as a motivating factor to light a fire under my backside, can somebody create an obj file, it doesn't have to be textured, don't include a skeleton structure just a simple static object, make it a hat or something basic, doesn't have to be expertly created but make it so its clear what its meant to be.

The challenge for me is to take this custom vertice count object and be able to create a yobj file from it, the second part of my challenge is to then have this object placed on the head of a character and working as a static object (no physics or animations).

Make it a single mesh with a vertice count of maybe a couple of hundred verts, nothing too complex, a hat or a crazy thing that has a clear structure so I know its facing the right way and in the right rotation to place on the head properly (like a cap would have the peak at the front and the shape is obvious how its placed).

Use an exported characters head as a guide to create the object, don't go crazy with it as it may never see the light of day if I fail so just something thrown together in your spare time will do.

Anybody welcome to submit their obj files, the idea being that I want to be able to open the possibilities of custom vertice count objects for people to be able to mod with and hopefully help provide a method which can be further developed on to allow the programming bods to create tools to do the leg work.

Edited August 8, 2016 by Perfectplex

[tekken57]

I'm glad to see someone else taking the initiative to look into yobj file's. I've detailed everything I know about the format here: http://sf4mods.blogspot.co.za/2014/05/yobj-model-format.html

The first issue with the vert limitation is due to the hardcoded offsets in the yobj. Not only are there offsets in the header for each object but there are also offsets listed for the shader params, animations, faces, etc in the relevant sections of the file. If you had to inject an object with a greater number of verts, all the offsets will need to change. I've tried coding changes to the offsets but any change causes the game to crash, hence you are stuck with injecting objects with less or the same number of verts and faces.

The second issue is with the rigging of the objects. With games on the xbox 360 / ps3, the rigging data was contained in the header of the file which listed the bones for each object and then the weight data controlled the movement of the object in relation to the bones. The pc version uses a different method of rigging as there is a greater number of bones. This is the reason why faces corrupt when animations are applied to mods and why the inject smaller function doesn't work as well as with the xbox 360 version.

If you want to invest time looking into creating your own yobj files, I would spend time figuring out how the rigging works first. In this way even though you are limited to injecting objects with less verts than the original, you can use multiple yobj files for different objects. e.g yobj 1 object 0 (3936 verts) = head of the wrestler, yobj 2 object 0 (used to the head of wrestler 2, 3936 verts) = singlet of wrestler. 

I'm not a specialist in 3d modelling and my knowledge is limited to what I can find on google. If you can get people with expertise in 3d modelling to assist, I think it would benefit us greatly.

Keep me in the loop if you figure anything out.

 

 

 

[Cave Waverider]

Just wanted to add that 2716 is also a valid yobj slot, although I'm not sure what it's normally used for. Probably something with hair as well.

Edited August 10, 2016 by Cave Waverider

[malan1998]

Just wanted to add that 2716 is also a valid yobj slot, although I'm not sure what it's normally used for. Probably something with hair as well.

2717 too! We can also use Attire objects. 

[Perfectplex]

Thanks for the info Tekken. I see your points about the limitations as any verts/faces beyond the count won't be covered by the rigging, I have limited knowledge on skeletal animation rigging too, only what I've played around with in the past so have a basic understanding, my hope is to at least have an object that is attached to the root bone, this would be no good for clothing as it wouldn't contort properly but for static meshes which require no warping and morphing it would be fine but that is going to be very limited objects as most people will want skinned weighted objects which can stretch and warp so it is probably not very useful for most.

 

If possible can you create an obj to yobj converter that does the complete opposite of what the export does, it just creates the yobj object, im not saying it could be used out the box and it doesn't have to be included on xrey but just something that creates a file with the data converted to the method used by yobj'ects.

Programming is way out my forte and I am using theory but I am assuming when the extract method is used on yobj files that it takes the chunk of hex code that is the object info and creates a data lists of the vertices and faces and UV's to be read as an obj file, is it possible to reverse the process? Even if nothing can be done with it for most I would find it interesting just to look into (I'm a bit sad like that), much like you have an option in x-rey to load up .obj files but just to view them but with an option to export it as yobj style conversion.

Something else I've looked into a bit more are the masking properties, I have had mild success with this but it would certainly require a tool to do the job as it does take a bit of tedious hex editing to achieve, there is still a lot more investigating to be done but its getting there and if I can't crack how to do the intricate masking on objects there would at least be an easy method to create a basic mask file which you can hide objects rather than have to shrink them down under clothing.

Edited August 12, 2016 by Perfectplex

[tekken57]

Sorry, I accidentally edited your post above without realizing. I clicked the edit button on your post instead of the reply button and ended losing the images in your reply.

 

Thanks for the info Tekken. I see your points about the limitations as any verts/faces beyond the count won't be covered by the rigging, I have limited knowledge on skeletal animation rigging too, only what I've played around with in the past so have a basic understanding, my hope is to at least have an object that is attached to the root bone, this would be no good for clothing as it wouldn't contort properly but for static meshes which require no warping and morphing it would be fine but that is going to be very limited objects as most people will want skinned weighted objects which can stretch and warp so it is probably not very useful for most.

 

I haven't tried messing with the arena objects but the the inject smaller function should do what you are suggesting here. Some of the objects are meant to be static so importing of the objects with less verts should be possible without any rigging issues. I don't do arena modding so I can't confirm. Try it out and post your results.

 

If possible can you create an obj to yobj converter that does the complete opposite of what the export does, it just creates the yobj object, im not saying it could be used out the box and it doesn't have to be included on xrey but just something that creates a file with the data converted to the method used by yobj'ects.

Programming is way out my forte and I am using theory but I am assuming when the extract method is used on yobj files that it takes the chunk of hex code that is the object info and creates a data lists of the vertices and faces and UV's to be read as an obj file, is it possible to reverse the process? Even if nothing can be done with it for most I would find it interesting just to look into (I'm a bit sad like that), much like you have an option in x-rey to load up .obj files but just to view them but with an option to export it as yobj style conversion.

 

The data for each object within the yobj is in a proprietary format and there is a significant amount of code written into X-Rey to convert this data into a human readable format i.e. wavefront object files. The conversion to yobj functionality already exists every single time you do an import back into the X-Rey. The data from the object being imported is converted back into the proprietary format expected by the yobj. 

 

The yobj has the following for each object:

1. Shader params (not imported / exported by X-Rey)

2. Vertices (imported by X-Rey)

3. Vertex normals (imported by X-Rey)

4. UV's (imported by X-Rey)

6. Weights and bone assignments (imported by X-Rey using the data from the weight 3dsmax script I coded)

5. Faces (not imported - explained below).

 

So if you would like to view what the data looks like when importing the obj back into the yobj, you can click the extract button in X-Rey and view the object info file in a hex editor.

 

I've spent a few months looking into importing of faces back into the yobj but didn't have any success. Basically a reverse triangle strip algorithm is required i.e. Face Data in the obj files are in triangles. The face data in the yobj is in triangle strip format. I've written code to convert the triangles into triangle strips but because the data in the yobj is in a proprietary format, I have not had any success in converting the triangles back into the same format.

If we could inject faces back into the file, we could import any object back into the game, even other models from other games. The vert limitation could be overcome by using the optimise functions in 3dsmax.

 

Having spent an immense amount of time on this and after seeking help on many forums without response, I gave up on importing faces.

 

The main yobj files also contains the following:

1. headers for the entire yobj

2. Headers for each object

3. Material assisgnments (not imported)

4. Bones (partially imported via X-rey as I haven't deciphered all the data stored here)

5. Some pofo data stored at the end of the file which I haven't deciphered. 

 

So in order to generate a full working yobj, you would need to generate all of the above which would take thousands of lines of code and a full understanding of all the data within the yobj. Keep in mind that 2k probably has a plugin for 3dsmax / maya, etc which generates the yobj files for them, trying to replicate this is not an easy task.

 

Something else I've looked into a bit more are the masking properties, I have had mild success with this but it would certainly require a tool to do the job as it does take a bit of tedious hex editing to achieve, there is still a lot more investigating to be done but its getting there and if I can't crack how to do the intricate masking on objects there would at least be an easy method to create a basic mask file which you can hide objects rather than have to shrink them down under clothing.

 

Figuring out how the masking works would really make life easier. I know @bannockburn  looked into this some time ago and may be able to shed some light on this.

Edited August 11, 2016 by tekken57

[Perfectplex]

Thanks for the info again Tekken, I am likely never going to accomplish what I wished for the reasons you already explained but its a learning experience regardless given that I haven't looked at a hex code editor before I took a stab at it for 2k16 and it has at least resulted in a few tricks that have served useful.

For the arena objects I already managed to convert an elbow pad to a motor bike with much larger vertice count but it wouldn't load as a clothing model, I have yet to try loading a custom sized mesh as an arena object, I too don't do arena modding but do see the merit of if only static meshes can be loaded with custom vert counts then at least it may serve others who do like that sort of modding.

What I meant by the obj to yobj converter was for the general obj viewer option you have on x-rey, although they would likely be unusable yobj files as they would lack headers and such I was curious as to if you could make a basic object info exporter to see how any obj files info looks without having to inject it into something first.

Updated first post, new cleaner layout plus updated info.

Edited August 12, 2016 by Perfectplex

[tekken57]

That would be impossible to do. Without headers, there is no way to determine how many faces and verts there in the yobj and such the model cannot be rendered. I've also already indicated to you that the faces cannot be created which prevents custom objects from being injected.

 

Edited August 12, 2016 by tekken57

[Perfectplex]

Ah ok, I was under the impression when using the load .obj file it was converting it to a .yobj format in memory then displaying it as it would look when injected.

I guess its a straight forward .obj loader with no conversion applied, my mistake.

I don't have 3ds max I use blender so have to rely on .obj conversions back and forth while testing things.