Bad symbols in code

[xezon]

There is this code in a file:

void Debug_Date_And_Time_Stamp ( void )
{
	//-------------------------------------------------------------------------
	//	tm_sec	- Seconds after minute (0 � 59)
	//	tm_min	- Minutes after hour (0 � 59)
	//	tm_hour	- Hours after midnight (0 � 23)
	//	tm_mday	- Day of month (1 � 31)
	//	tm_mon	- Month (0 � 11; January = 0)
	//	tm_year	- Year (current year minus 1900)
	//	tm_wday	- Day of week (0 � 6; Sunday = 0)
	//	tm_yday	- Day of year (0 � 365; January 1 = 0)
	//-------------------------------------------------------------------------

When I changed and saved this file it messed up the file diff on these lines with the bad characters.

We may want to

• Strip non ascii characters from code
• Add a .gitattributes to force proper text file rules

[OmniBlade]

Yeah, its because whatever opened them treated the text as utf-8 and treats the values > 127 as unknown, when its really the windows code page 1252. VSCode is always doing this for me and its a pain to clean up if you don't notice before a commit. For comments like these its probably best to convert correctly to utf-8 these days, for print statements maybe just remove the none ascii as you suggest?

[xezon]

Let us strip such characters from all code please. I see no reason to use them. Everything can be Ascii.

For game data, I added the following .gitattributes file, which avoided troubles ever since, but never understood how this all works exactly:

https://github.com/TheSuperHackers/GeneralsGamePatch/blob/main/.gitattributes

Anyone wants to take care of massaging this?

[xezon]

Can someone take care of this?

[DevGeniusCode]

Is this good?
https://github.com/TheSuperHackers/GeneralsGameCode/compare/main...DevGeniusCode:GeneralsGameCode:utf-8?expand=1

[xezon]

Hmm. So you convert ascii files to utf8? Can you ask OmniBlade about that?

For Keyboard.cpp and Debug.cpp this is probably necessary, but all others look like they could remain ascii.

[DevGeniusCode]

@tomsons26

Any change but hex literal will alter what the char is in the binary
Potentially making it draw garbage when its drawn

[OmniBlade]

Most of these are comments and in general I'm okay with bumping the comments to UTF-8. I think we need to be careful with the literals in the code as that will change what is in the binary but I can only spot a handful of places where they were changed. Ideally as @tomsons26 said in the discord chat, those need replacing carefully, especially the keyboard related ones. The logging ones, well the risk is there that you get a garbage character in your output, not too serious.

[DevGeniusCode]

Using a Python script I converted all files to utf-8, with 66 files having special characters.
https://github.com/TheSuperHackers/GeneralsGameCode/compare/main...DevGeniusCode:GeneralsGameCode:Hex-literals?expand=1
Most of them were in comments, only in 7 cases were they part of the code, I binary compared the files before and after the change

Any idea?

VS2022

\diff_binary.py -l \before_vs22.exe \after_vs22.exe 
Result of comparison: content
File 1 length:  21053952
File 2 length:  21053952
Num. differences:  11
offset differs: 0x158
List of differences:
offset 0x00000158: 0x9c != 0x72 ('�' != 'r')
offset 0x00000159: 0x70 != 0x76 ('p' != 'v')
offset 0x00f1e5e9: 0x35 != 0x36 ('5' != '6')
offset 0x00f1e5eb: 0x35 != 0x32 ('5' != '2')
offset 0x00f1e5ec: 0x38 != 0x33 ('8' != '3')
offset 0x00f1e5ee: 0x34 != 0x33 ('4' != '3')
offset 0x0117fbfc: 0x9c != 0x72 ('�' != 'r')
offset 0x0117fbfd: 0x70 != 0x76 ('p' != 'v')
offset 0x0117fc18: 0x9c != 0x72 ('�' != 'r')
offset 0x0117fc19: 0x70 != 0x76 ('p' != 'v')
offset 0x011b3f4c: 0x06 != 0x07 (  6 !=   7)
Elapsed time:  00:00:01

VS6

\diff_binary.py -l \before_vs6.exe \after_vs6.exe 
Result of comparison: content
File 1 length:  15413360
File 2 length:  15413360
Num. differences:  9
offset differs: 0x158
List of differences:
offset 0x00000158: 0x56 != 0x49 ('V' != 'I')
offset 0x00000159: 0x72 != 0x79 ('r' != 'y')
offset 0x00b24004: 0x56 != 0x49 ('V' != 'I')
offset 0x00b24005: 0x72 != 0x79 ('r' != 'y')
offset 0x00bd756b: 0x30 != 0x33 ('0' != '3')
offset 0x00bd756c: 0x36 != 0x35 ('6' != '5')
offset 0x00bd756e: 0x31 != 0x34 ('1' != '4')
offset 0x00bd756f: 0x32 != 0x39 ('2' != '9')
offset 0x00eb300c: 0x02 != 0x03 (  2 !=   3)
Elapsed time:  00:00:01

[tomsons26]

Right so there aren't bad symbols, just win1252, in actual code they should be replaced with hex literals since the game is coded with 1252 in mind
comments - depending on situation

bytes from retail binaries
CPU string
4C 31 20 49 6E 73 74 72 75 63 74 69 6F 6E 20 54 72 61 63 65 20 43 61 63 68 65 3A 20 25 64 20 77 61 79 20 73 65 74 20 61 73 73 6F 63 69 61 74 69 76 65 2C 20 25 64 6B 20 B5 4F 50 73 0D 0A 00
messagebox string 45 73 20 69 73 74 20 65 69 6E 20 67 72 61 76 69 65 72 65 6E 64 65 72 20 46 65 68 6C 65 72 20 61 75 66 67 65 74 72 65 74 65 6E 2E 20 53 6F 6C 63 68 65 20 46 65 68 6C 65 72 20 6B F6 6E 6E 65 6E 20 64 75 72 63 68 20 76 69 65 6C 65 20 76 65 72 73 63 68 69 65 64 65 6E 65 20 44 69 6E 67 65 20 77 69 65 20 56 69 72 65 6E 2C 20 FC 62 65 72 68 69 74 7A 74 65 20 48 61 72 64 77 61 72 65 20 75 6E 64 20 48 61 72 64 77 61 72 65 2C 20 64 69 65 20 64 65 6E 20 4D 69 6E 64 65 73 74 61 6E 66 6F 72 64 65 72 75 6E 67 65 6E 20 64 65 73 20 53 70 69 65 6C 73 20 6E 69 63 68 74 20 65 6E 74 73 70 72 69 63 68 74 2C 20 61 75 73 67 65 6C F6 73 74 20 77 65 72 64 65 6E 2E 20 54 69 70 70 73 20 7A 75 72 20 56 6F 72 67 65 68 65 6E 73 77 65 69 73 65 20 66 69 6E 64 65 73 74 20 64 75 20 69 6E 20 64 65 6E 20 46 6F 72 65 6E 20 75 6E 74 65 72 20 77 77 77 2E 67 65 6E 65 72 61 6C 73 2E 65 61 2E 63 6F 6D 2C 20 49 6E 66 6F 72 6D 61 74 69 6F 6E 65 6E 20 7A 75 6D 20 54 65 63 68 6E 69 73 63 68 65 6E 20 4B 75 6E 64 65 6E 64 69 65 6E 73 74 20 69 6D 20 48 61 6E 64 62 75 63 68 20 7A 75 6D 20 53 70 69 65 6C 2E

L'$' line replacement is 0x20AC

all the strange - should go and be replaced with normal -

or the integerßin host byte order to or the integer in host byte order

imo the keyboard comments should describe the char instead of using it

copyright as (c) instead of the symbol