[Qemu-devel] Self-modifying code with REP-prefix on TCG

[Clemens Kolbitsch]

Hi guys,

a quick search on the archives did not reveal anything - if this is already a known issue, please don't kick me ;)

We encountered an issue using TCG, where self-modifying code is overwriting the executing instruction, causing different behavior in TCG and a real CPU.

Clearly, when code is overwriting itself, the QEMU logic will detect and invalidate the TB - and one would assume this is the correct behavior. However, when the current instruction has a REP-prefix, this should cause the ORIGINAL instruction to execute, and NOT generate an exception.

Since TCG uses a loop, however, a repeated invocation of the REP'ed instruction will cause an exception, TCG retranslates the TB, and QEMU will execute the new code.

For example:

seg000:0039083D                      loc_39083D:.........
seg000:0039083D E8 00 00 00 00                       call    $+5
seg000:00390842 59                                   pop     ecx
seg000:00390843 8D BB 00 10 40 00                    lea     edi, (loc_390000 - 0FFF8F000h)[ebx]
seg000:00390849 29 F9                                sub     ecx, edi
seg000:0039084B 83 C1 10                             add     ecx, 10h
seg000:0039084E 30 C0                                xor     al, al
seg000:00390850 FC                                   cld
; here ECX = 0x852 and EDI = 0x390000
; so at the step 0x851 the instruction rewrites itself
; but EIP stays the same = 0x00390851
; and instruction becomes invalid
; seg000:00390851 00 AA..
; when QEMU tries to regenerate TB
; the generated instruction throws an exception
seg000:00390851 F3 AA                                rep stosb
seg000:00390853 56                                   push    esi
seg000:00390854 C3                                   retn

On a real CPU, the instruction becomes invalid when overwritten, but the CPU continues execution without raising an exception (because it's already been loaded into the execution registers).

Getting around this is not trivial, so if it's a new issue, does anyone have an idea how to approach it most meaningfully? I was thinking about detecting this very specific case and marking the TB for re-translation only after the TB is exited, but this is not very clean and also somewhat prone to errors.

Thanks!

-Clemens

--

Clemens Kolbitsch

Security Researcher

address@hidden

Mobile +1 (206) 356-7745

Land +1 (805) 456-7076

Lastline, Inc.

6950 Hollister Avenue, Suite 101

Goleta, CA 93117

www.lastline.com