topic Re: The database contains objects with non-Unicode characters in Firewall and Security Management

The database contains objects with non-Unicode characters

Vladimir — Fri, 23 Aug 2019 12:34:09 GMT

When Pre-Upgrade Verifier indicates that "The database contains objects with non-Unicode characters", the corresponding SK directs us to download and execute this utility on a PC running SmartConsole: Check_Point_R80_Encoding_Detection_sk109795.zip

The results are then supposed to be defined in db_encoding.txt

But since we are not prompted for the management server's IP or credentials, I suspect that this utility is detecting encoding on local machine, not the Management Server. So if you have multiple workstations with different encoding on each, indicated result may be one of many, but we are not aware of that at this point.

Can someone explain to me what the outcome of the situation described above would be?

Thank you,

Vladimir

Re: The database contains objects with non-Unicode characters

Norbert_Bohusch — Fri, 23 Aug 2019 14:35:26 GMT

The Encoding of the names, comments, etc in R77 was dependent of the client. If you had different encoding on the clients they might have seen it different.

R80 switched it to UTF-8. so just use the encoding of your main administrator(s) and it should be fine.

Re: The database contains objects with non-Unicode characters

HeikoAnkenbrand — Fri, 23 Aug 2019 16:16:14 GMT

The R7x Smart Dashboard uses the character set of the Windows client for example German character set WINDOWS-1252. If a company works internationally, it can happen that there are several character set in the R7x policy (German->WINDOWS-1252, Russian -> CP1251,...) . Then this solution does not work when you upgrading to R80.x.

In this case, in some files, for example in objekts_5_0.C (please backup them :-), you should replace the corresponding character set manually.

If a wrong encoding was selected, then the upgrade would still succeed, but during the upgrade, contents of object fields that contain non-English characters
will be converted to an ASCII representation of bytes used for the original text in the field.
For example, the Russian word for 'hello' ('ïðèâåò') in CP1251 encoding will be converted to the following string: 'ef_f0_e8_e2_e5_f2'

Re: The database contains objects with non-Unicode characters

KennyManrique — Fri, 23 Aug 2019 18:02:46 GMT

Hi Vladimir

You can verify policy and objects files (objects.C, objects_5_0.C, rules.C) located in $FWDIR/conf using the following command:

grep --color='auto' -P -n "[\x80-\xFF]" filename

This command will return all lines where non ASCII chars exists. Usually, the most ocurrences are at objects name or description. You can either manually edit the file or modify the object through SmartDashboard by doing a manual search.

Re: The database contains objects with non-Unicode characters

HeikoAnkenbrand — Fri, 23 Aug 2019 19:59:10 GMT

Hi @KennyManrique

there's a SK describing your procedure.
Unfortunately I can't find it. Do you still have the SK number?

Re: The database contains objects with non-Unicode characters

Vladimir — Fri, 23 Aug 2019 20:20:00 GMT

I have tried this command previously, but this gem from CPUG by jflemingeds actually covered all of them:

Re: Eliminate non-UTF-8 encoded chars

Ok.. so 2nd stab.

works for object_5_0.C hits

grep --color='auto' -P '^\t\t:|[^\x00-\x7F]' objects_5_0.C | grep -B1 -P '^\t\t\t'

This works for rulebases_5_0.fws hits

grep --color='auto' -P 'rule-base|[^\x00-\x7F]' rulebases_5_0.fws | grep -P -B 1 '^\t\t\t'

Re: The database contains objects with non-Unicode characters

JozkoMrkvicka — Fri, 23 Aug 2019 21:03:30 GMT

https://community.checkpoint.com/t5/General-Management-Topics/Upgrade-with-non-Unicode-characters/m-p/47246/highlight/true#M8117

Re: The database contains objects with non-Unicode characters

KennyManrique — Mon, 26 Aug 2019 15:10:40 GMT

Hi @HeikoAnkenbrand, I found the grep search solution online way before the SK´s were published, since i got this error on my first migration from R77.XX when no SK was available yet.

The ones I saw on SK are the following: