In fact OpenSSL’s integrity check has always been over the openssl portion of the binary as it is loaded into application address space. At runtime the module does not inspect the binary on disk, but inspects its own RAM. Thus no change was required to openSSL to support iOS (other than adding support for the Mach-O format). To compute the RAM image at compile time they have a utility that understands how ELF, Mach-O and other formats are loaded into RAM. To perform validation at runtime, their SK know how to find itself in RAM.
From the User Guide:
In general the C compiler is required to segregate constant data in a contiguous area (e.g. by placing it in a dedicated segment) to compile the FIPS module. Some compilers were found to fail to meet the const data segment requirement. In the cases where the errant behavior was observed, the compiler was instructed to generate position-independent code.
Position independent code is generated by default so the generated FIPS Object Module can be included in a shared library.
Embedding of the runtime digest can be accomplished by In-place Editing of the Object Code. In order to ease the task of cross-compiling the FIPS Object Module (compilaton on one architecture for execution on another architecture), instead of determining the runtime digest value by actual execution on the target system, a utility is used to analyze the compiled object code on the build system and calculate the digest. This utility is platform (or object code format) sensitive. For ELF binaries it is called incore, for Microsoft Windows msincore, for OS X and iOS incore_macho.
ios-incore-2.0.1.tar.gz contains OSX and iOS specific Incore utility to determine the object code digest.
The Incore utility is a native application used to embed the FIPS Object Module's fingerprint in the ARM library. Building Incore is a two step process – first, build a native version of libcrypto.a, and then build Incore using the previously built native libcrypto.a.
This is indicated while incore integrity testing of the module itself is being performed. This operation performs an HMAC over sections of incore data and checks the value against an expected value set when the application is compiled.
The FIPS Object Module is generated in binary file format, with an embedded pre-calculated HMAC-SHA-1 digest covering the module as it is loaded into application address space. The Module integrity check consists of recalculating that digest from the memory areas and comparing it to the embedded value which resides in an area not included in the calculated digest This “incore hashing” integrity test is designed to be both executable format independent and fail-safe. For this scenario the Module is the text and data segments as mapped into memory for the running application. The term Module is also used, less accurately, to designate the antecedent of that memory mapped code and data, the FIPS Object Module file residing on disk.