Skip to content

xrCore: rt_compressor.cpp: Use appropriate integer type#10

Merged
themrdemonized merged 1 commit intothemrdemonized:all-in-one-vs2022from
clayne:1692030340-rt-compressor-stack-smash
Aug 15, 2023
Merged

xrCore: rt_compressor.cpp: Use appropriate integer type#10
themrdemonized merged 1 commit intothemrdemonized:all-in-one-vs2022from
clayne:1692030340-rt-compressor-stack-smash

Conversation

@clayne
Copy link
Contributor

@clayne clayne commented Aug 14, 2023

  • rtc_compress, rtc_decompress: Both of these functions pass a pointer to an out_size variable which lzo1x_1_compress and lzo1x_1_decompress dereference to store the value for the caller to use. The problem is that with a u32 type it's passing a pointer to a 4-byte integer but cast as an 8-byte integer hence the dereference and store is unsafe.
    *out_len = 0;
00000001405064F6  mov         rax,qword ptr [out_len]
00000001405064FE  mov         qword ptr [rsp+58h],rax
0000000140506503  mov         rcx,qword ptr [rsp+58h]
0000000140506508  call        __asan_store8 (014355B111h)
000000014050650D  mov         rax,qword ptr [out_len]
0000000140506515  mov         qword ptr [rax],0 <----- *qword*

Found with address sanitizer.

* rtc_compress, rtc_decompress: Both of these functions pass a pointer
  to an out_size variable which lzo1x_1_compress and lzo1x_1_decompress
  dereference to store the value for the caller to use. The problem is
  that with a u32 type it's passing a pointer to a 4-byte integer but
  cast as an 8-byte integer hence the dereference and store is unsafe.

    *out_len = 0;
00000001405064F6  mov         rax,qword ptr [out_len]
00000001405064FE  mov         qword ptr [rsp+58h],rax
0000000140506503  mov         rcx,qword ptr [rsp+58h]
0000000140506508  call        __asan_store8 (014355B111h)
000000014050650D  mov         rax,qword ptr [out_len]
0000000140506515  mov         qword ptr [rax],0 <----- *qword*

  Found with address sanitizer.
@themrdemonized themrdemonized merged commit fa77274 into themrdemonized:all-in-one-vs2022 Aug 15, 2023
themrdemonized added a commit that referenced this pull request Aug 15, 2023
xrCore: rt_compressor.cpp: Use appropriate integer type
(cherry picked from commit fa77274)
themrdemonized added a commit that referenced this pull request Aug 15, 2023
xrCore: rt_compressor.cpp: Use appropriate integer type
(cherry picked from commit fa77274)
(cherry picked from commit fa5edb4)
@clayne clayne deleted the 1692030340-rt-compressor-stack-smash branch August 19, 2023 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants