The database contains objects with non-Unicode characters

[Vladimir]

When Pre-Upgrade Verifier indicates that "The database contains objects with non-Unicode characters", the corresponding SK directs us to download and execute this utility on a PC running SmartConsole: Check_Point_R80_Encoding_Detection_sk109795.zip

The results are then supposed to be defined in db_encoding.txt

But since we are not prompted for the management server's IP or credentials, I suspect that this utility is detecting encoding on local machine, not the Management Server. So if you have multiple workstations with different encoding on each, indicated result may be one of many, but we are not aware of that at this point.

Can someone explain to me what the outcome of the situation described above would be?

Thank you,

Vladimir

[Norbert_Bohusch]

The Encoding of the names, comments, etc in R77 was dependent of the client. If you had different encoding on the clients they might have seen it different.

R80 switched it to UTF-8. so just use the encoding of your main administrator(s) and it should be fine.

[HeikoAnkenbrand]

The R7x Smart Dashboard uses the character set of the Windows client for example German character set WINDOWS-1252. If a company works internationally, it can happen that there are several character set in the R7x policy (German->WINDOWS-1252, Russian -> CP1251,...) . Then this solution does not work when you upgrading to R80.x.

In this case, in some files, for example in objekts_5_0.C (please backup them :-), you should replace the corresponding  character set manually.

If a wrong encoding was selected, then the upgrade would still succeed, but during the upgrade, contents of object fields that contain non-English characters
will be converted to an ASCII representation of bytes used for the original text in the field.
For example, the Russian word for 'hello' ('ïðèâåò') in CP1251 encoding will be converted to the following string: 'ef_f0_e8_e2_e5_f2'

[KennyManrique]

Hi Vladimir

You can verify policy and objects files (objects.C, objects_5_0.C, rules.C) located in $FWDIR/conf using the following command:

grep --color='auto' -P -n "[\x80-\xFF]" filename

This command will return all lines where non ASCII chars exists. Usually, the most ocurrences are at objects name or description. You can either manually edit the file or modify the object through SmartDashboard by doing a manual search.

[HeikoAnkenbrand]

Hi @KennyManrique 

there's a SK describing your procedure.
Unfortunately I can't find it. Do you still have the SK number?

[KennyManrique]

Hi @HeikoAnkenbrand, I found the grep search solution online way before the SK´s were published, since i got this error on my first migration from R77.XX when no SK was available yet.

The ones I saw on SK are the following:

• https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...
• https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Vladimir]

I have tried this command previously, but this gem from CPUG by jflemingeds actually covered all of them:

 Re: Eliminate non-UTF-8 encoded chars

Ok.. so 2nd stab.


works for object_5_0.C hits

grep --color='auto' -P '^\t\t:|[^\x00-\x7F]' objects_5_0.C | grep -B1 -P '^\t\t\t'

This works for rulebases_5_0.fws hits 

grep --color='auto' -P 'rule-base|[^\x00-\x7F]' rulebases_5_0.fws | grep -P -B 1 '^\t\t\t'

[JozkoMrkvicka]

https://community.checkpoint.com/t5/General-Management-Topics/Upgrade-with-non-Unicode-characters/m-...