Category:FastFiles: Difference between revisions

From COD Engine Research
← Older edit
Jakes625 (talk | contribs)
confirmed
63 edits
Jakes625 (talk | contribs)
confirmed
63 edits
mNo edit summary
 
(One intermediate revision by the same user not shown)
Line 35: Line 35:
Stream blocks exist within a stack, when a loader is using a block, it pops the requested block onto the stack, reads data into it, then when complete, pops that block off the stack, and returns to the previous block at the block stream cursor position.
Stream blocks exist within a stack, when a loader is using a block, it pops the requested block onto the stack, reads data into it, then when complete, pops that block off the stack, and returns to the previous block at the block stream cursor position.


=== flow ===
=== Block Stream Flow ===
<div class="mw-collapsible mw-collapsed" style="border:1px solid #a2a9b1; padding:0.5em; margin:0.5em 0;">
'''Show'''
<div class="mw-collapsible-content">
<pre>
<pre>
START
START
Line 167: Line 170:
END
END
</pre>
</pre>
</div>
</div>


Pop behavior:
Pop behavior:
Line 221: Line 226:


The offset is only correct after the zone file has been loaded into the respective g_streamBlocks/memory. This is because data gets loaded into different blocks depending on how each executable loader works. In addition, data is aligned based on data type.
The offset is only correct after the zone file has been loaded into the respective g_streamBlocks/memory. This is because data gets loaded into different blocks depending on how each executable loader works. In addition, data is aligned based on data type.
=== Padding Rules ===
{| class="wikitable"
|  ||'''XStruct'''||'''Pointers'''||'''Byte'''||'''Bool'''||'''int16/uint16'''||'''Default'''
|-
|Byte Alignment||4||4||1||1||2||4
|}


== Insert Pointers ==
== Insert Pointers ==
Line 242: Line 240:
the original pointer field that contained -2
the original pointer field that contained -2
the new inserted pointer cell in LARGE
the new inserted pointer cell in LARGE
== Padding Rules ==
[[BlockStream_Padding|block stream padding docs]]

Latest revision as of 21:35, 15 June 2026

FastFiles are the CoD engine's way of storing Zone files. FastFiles (FFs) almost always have some sort of RSA2048 signature, a zlib compression along the way, and then on later CoDs some encryption. The Zones are a library of assets. Every FF in every CoD game starts the same way...

Name Offset Type Description
Magic 0x0 char[8] "IWffu100" for unsigned fastfiles, "IWff0100" for signed fastfiles, "TAff0100" for treyarch's Black Ops 2 fastfile, and "S1ff0100" for Sledgehammer Advanced Warfare fastfiles.
Version 0x8 int See below.

So by reading the version of the input FF, you can tell what system and game the FF is from. Note that pre-alpha versions of Call of Duty 4 are missing the version integer, and must be handled specially. Also keep in mind that the Quantum of Solace FFs are missing the magic in the struct above.

XFILE_VERSION 007:QoS CoD4 WaW MW2 BO1 MW3 BO2 Ghosts CoD: Online Advanced Warfare BO3 Beta BO3
Xbox 360 0x1D6 0x1 0x183 0x10D 0x1D9 0x70 0x92 0x22E n/a 0x72E n/a 0x253
Playstation 3 0x1D6 0x1 0x183 0x10D 0x1D9 0x70 0x92 0x22E n/a 0x72E n/a 0x25E
Wii/Wii-U 0x1D2 0x1A2 0x19B n/a 0x1DD 0x6B 0x94 0x22E n/a n/a n/a n/a
PC 0x1D6 0x5 0x183 0x114 0x1D9 0x1 0x93 0x135 0x13C 0x72E 0x1FB n/a

Stream Blocks

After the Zone (XFile) is extracted from the fastfile, you read the block sizes of each g_streamBlock allocation. This determines how much memory to allocate to each stream to read data into. Each block stream has a specific role in the data that is read and it goes as follows:

Temp Physical Runtime Virtual Large Callback Vertex
Purpose Asset Pool Headers, Counts, Root Pointers Sound Data ?? PC uses this for loading assets Strings and Data Chunks (rawfiles) Used in some loader methods Material and Image data

One distinction is that PC uses virtual for a load of loading where PS3 mainly uses large.

Stream blocks exist within a stack, when a loader is using a block, it pops the requested block onto the stack, reads data into it, then when complete, pops that block off the stack, and returns to the previous block at the block stream cursor position.

Block Stream Flow

Show 

START
  |
  v
+--------------------------------------------------+
| Init XBlockStream                                |
| ActiveBlock = TEMP                               |
| Stack = empty                                    |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Read header                                      |
| Source cursor advances                           |
| Blocks are created from header block sizes       |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Read XAssetList root                             |
| ActiveBlock = TEMP                               |
| Reads: scriptStringCount, scriptStringsPtr,      |
|        assetCount, assetsPtr                     |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Materialize ScriptStringsPtr                     |
| Payload block = LARGE                            |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| PUSH LARGE                                       |
| Stack push: previous = TEMP                      |
| ActiveBlock = LARGE                              |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Read script string pointer array                 |
| ActiveBlock = LARGE                              |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| For each script string pointer                   |
|                                                  |
|  0        -> null, no payload                    |
| -1 inline -> read CString at current LARGE pos   |
| -2 insert -> reserve extra pointer cell in LARGE |
|              patch original + inserted cells     |
|              read CString in LARGE               |
| offset    -> decode block+offset, resolve/defer  |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| POP LARGE                                        |
| ActiveBlock returns to TEMP                      |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Materialize AssetsPtr                            |
| Payload block = LARGE                            |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| PUSH LARGE                                       |
| Stack push: previous = TEMP                      |
| ActiveBlock = LARGE                              |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Phase 1: read XAsset table                       |
| ActiveBlock = LARGE                              |
| Each row: asset type + asset pointer             |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Phase 2: for each asset row                      |
| Materialize asset root pointer                   |
| Root payload block = TEMP                        |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Asset root pointer kind                          |
|                                                  |
|  0        -> asset value = null                  |
| -1 inline -> PUSH TEMP, read asset payload       |
| -2 insert -> reserve pointer cell in LARGE       |
|              patch original + inserted cells     |
|              PUSH TEMP, read asset payload       |
| offset    -> decode block+offset                 |
|              PUSH decoded block, read payload    |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Read asset object fields                         |
| Nested pointers may push TEMP, LARGE, VIRTUAL,   |
| or another decoded/allocated block               |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| POP nested block(s)                              |
| POP asset payload block                          |
| Continue next asset                              |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| After all assets                                 |
| POP LARGE                                        |
| ActiveBlock returns to TEMP                      |
+--------------------------------------------------+
  |
  v
+--------------------------------------------------+
| Resolve deferred CString pointers                |
| Validate source cursor                           |
| Seal/pad emitted blocks                          |
+--------------------------------------------------+
END

Pop behavior: 

POP behavior:

If popping from TEMP:
  restore TEMP cursor to the position saved when TEMP was pushed

If popping from any other block:
  keep that block's advanced cursor

Always:
  ActiveBlock = previous block from the stack frame

Important:
  Source _position is not rewound by POP.
  The stack only controls emitted block cursors.

Pointers in Zone Files

Pointers can be classified as one of the following:

  • 0 : null, no data follows
  • -1 : inline, data follows after the struct, usually in order but NOT ALWAYS
  • -2 : insert, space is allocated in the stream block and inserted into later
  • else : everything else is an encoded offset, it contains which block and offset it's pointing too.

Pointers have different types as well, depending on the type the data is loaded differently.

  • Direct : It points directly to data, such as char*, int*, itemDef_t*
  • Alias : It pointers to shared assets, such as Material*

For encoded offsets the first 4 bits (or 3 bits in BO2) are extracted and used as an index for the g_streamBlocks array. The other bits are the offset and is added to the data pointer in g_streamBlocks.

Encoding

int EncodeOffset(int blockIndex, int offset)
{
    return (blockIndex << 28) | (offset + 1);
}

Decoding

XBlockAddress DecodeOffset(int value)
{
    int block = (XFILE_BLOCK)(value >> 28);
    int offset = (int)(value & 0x0FFFFFFF) - 1;

    return new XBlockAddress(block, offset);
}

For example a pointer with a value of 0x40001689 means that the data belongs in block stream 4 (large) at offset 0x1688 .

The offset is only correct after the zone file has been loaded into the respective g_streamBlocks/memory. This is because data gets loaded into different blocks depending on how each executable loader works. In addition, data is aligned based on data type.

Insert Pointers

For -2 pointers, the flow goes as follows: 1. ReadPointer reads raw -2, writes that 4-byte value into the active emitted block, and remembers that cell as PatchAddress.

2. Later, when the pointer is materialized it recognizes -2 (insert)

3. For insert, it allocates one extra 4-byte pointer cell in the LARGE block, aligned to 4, initialized to 0

4. It resolves the actual payload address according to the plan: current block position, allocated block position, or current stream position.

5. It patches two cells to the encoded payload address: the original pointer field that contained -2 the new inserted pointer cell in LARGE

Padding Rules

block stream padding docs

Retrieved from "https://codresearch.dev/index.php?title=Category:FastFiles&oldid=1055"