Flash Access error in Dr.Spark

[G_W_Albrecht]

Dr.Spark gives me a serious error with the flash card.

Flash Usage:

Test passed. Flash usage: 60-70% OK.

Flash Access:

Test failed. Access to flash rate error. Access to flash rate is too high. Flash expectancy: .15 years.

So in less than two month the flash will be unusable ? Can it be exchanged ?

[Chris_Atkinson]

What does "show diag" report about the flash health, how old is the appliance and have you opened an SR with TAC?

[G_W_Albrecht]

1.

Current system info

-----------------------------------

Current image name: R81_996003913_10_15

Current image version: 913

Previous image name: R81_996004071_10_15

Previous image version: 071

Default image name: R81_996003913_10_15_def_250122

Default image version: 913

Bootloader version: 992000052

HW version: 011

Serial number: WA19800093

HW MAC address: 00:1C:7F:95:E5:05

LAN MAC address: 00:1C:7F:95:E5:06

Unit version: <Undefined>

Unit model: V0

Marketing capabilities: 0

Management opaque: klEFIIdNAVg=:j9eHr/J916s=:4ZWmR

Hardware capabilities: 3 - SD card + Wireless

RTC status: OK

Voltage 3V3: 3.3528V - OK (valid: 3.1500V ~ 3.4500V)

Voltage 12V: 12.2853V - OK (valid: 11.4000V ~ 12.6000V)

Voltage 1V8: 1.8240V - OK (valid: 1.7000V ~ 1.9000V)

Voltage 0V9: 0.9720V - OK (valid: 0.8100V ~ 1.1000V)

Voltage 1V2: 1.2098V - OK (valid: 1.1600V ~ 1.2400V)

Voltage 1V2_SRM: 1.2026V - OK (valid: 1.1600V ~ 1.2400V)

CPU Temperature: 46.0000C - OK (valid: 0.0000C ~ 82.0000C)

CPU Temperature(internal): 51.7890C - OK (valid: 0.0000C ~ 100.0000C)

DDR Temperature: 45.0000C - OK (valid: 0.0000C ~ 80.0000C)

EMMC % of device lifetime used: 60-70% - OK (valid: 0 ~ 90%)

EMMC EOL status: Normal - OK

Branding file: N/A

Preset File installed: no

 2. I have received this appliance from CP for beta testing the 15xx series, so it is one of the very early units

3. This is in CP UC Account, so i can open no SR#with TAC...

[Amir_Ayalon]

Hi All

The eMMC is a flash memory based device, and it life span is determined by the number of times your write/erase on it.

Very early SW versions had a bug that causes multiple read/write cycles that impacted the eMMC endurance, this issue has been resolved in R80.20.40 GA, but GW's that run older SW for long duration of time, can exhibit increased eMMC usage percentage, and as a result a shorted life span.

GW's running updated SW, should not exhibit the problem.

In any case, if your GW reached 90% eMMC usage, you can request an RMA (and it will be approved)

Thanks

[G_W_Albrecht]

It is the Dr.Spark error that troubles me 😉 Or is Access to Flash rate not so important?

[Thomas_Eichelbu]

Hello team!

iam facing the same issue ...
Check Pro Support ist starting to send alarms each day for Quantum Sparks reaching end of life for eMMC Cards ...

i see a difference.

1535 use Type:
EMMC % of device lifetime used:         90-100% - NOK (valid: 0 ~ 90%)

1575 uses  Type:
EMMC(Type B) % of device lifetime used:         10-20% - OK (valid: 0 ~ 90%

1900 uses Type:
EMMC(Type B) % of device lifetime used: 0-10% - OK (valid: 0 ~ 90%)

the Quantum Sparks are used for tiny "point of sales" installations. with little daily traffic, all central managed.
while the 1575 are used for smaller branches ... 
so i would suspect the larger sites wear  out faster, but no.
the 1535 Series seems to utilize a more low cost type of eMMC.

with the "mmc" command you can read the statistics off the device ...
alot of data there ...
"mmc extcsd read /dev/mmcblk1 | more"

TAC recommends not to use IoC feeds, or at least raise the feeds interval to a very high timeframe ...
but the 1575 are running the same IoC Feeds ... they dont wear out ...
also they found alot of load sigs operation or policy load operation ... IPS signature updates
So i tear down my appliance when using the latest and greates security.

a shame .. i fear i have to run many many RMA´s for all the 1535 on this customer ...
but investigation is still ongoing ... 

check this: https://developer.toradex.com/software/linux-resources/linux-features/emmc-linux/

then:
1535
[Expert@1535]# mmc extcsd read /dev/mmcblk1 | grep LIFE
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x0a
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x00
[Expert@1535]# mmc extcsd read /dev/mmcblk1 | grep EOL
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01



1575
[Expert@1575]# mmc extcsd read /dev/mmcblk1 | grep LIFE
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x01
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x02
[Expert@1575]# mmc extcsd read /dev/mmcblk1 | grep EOL
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01

it is written:

The life time values are provided in 10% steps:

• 0x01 = 0-10% device life time used

• 0x02 = 10-20% device life time used

• …

whereas life time estimation type A is for SLC eraseblocks and type B is for MLC eraseblocks.

The EOL information is an overall status for reserved blocks, given as follows:

• 0x00 = Not defined

• 0x01 = Normal: consumed less than 80% of the reserved blocks

• 0x02 = Warning: consumed 80% of the reserved blocks

• 0x03 = Urgent: consumed 90% of the reserved blocks

so a hex of 0x0a is 100% life time reach and "dead"?
but EOL is NORMAL?

confusing to me ... maybe the flash is not dead ... or is it dead?

[G_W_Albrecht]

I assume it means: no more reserved blocks available. My experience: 5 years of work will a SMB do...

[PhoneBoy]

As far as I know, the only hardware difference between 15x5 models is the license.
As noted above, the "premature wearing out of the MMC" may be caused by a software bug on older releases.

Whether that's the case with your customer or not, I can't say.
Unfortunately, an RMA is the only way to fix it.

[Amir_Ayalon]

1530/1550 and 1570/1590 has different eMMC size.

As for the eMMC status, there are two indicators, current status of flash usage, and rate.

High rate can cause problem in the future.

current eMMC status  - if above 90% needs a replacement.

older firmware's has some bug which cause high rate of writes, which will shorter the Flash lifespan, newer firmware's should not have this problem. if however, you are running a newer FW and still see high rate alarm, please contact us (directly amiray@checkpoint.com, or via a task, and we will investigate)

thanks

[J_admin12]

We have also experienced similar issues on the 1530/1570 devices where the flash usage was at 90% usage within a year of installation. These devices had never run older software versions of <R81.10.x in production (In some cases these may have had the factory version of R80.20.x but they would have been upgraded within 10 minutes of booting, others were straight on R81.10.x).

It was observed that these devices were constantly using SWAP which we suspect was the cause of the high flash usage.

Following a number of recommendations from TAC. Such as reducing IPS signatures to 5 years, disabling IOC feeds, disabling monitoring solutions among a number of other things, but none of these seemed to have an impact.

An underlying issue was found, where the SFWD process was crashing between 10 - 30 times a day.

Most of these devices were replaced with 15x5 variants, the swap usage went away and the high flash usage has not re-occurred. I assume this is because these newer models have 4GB RAM compared to the 1530 2GB, therefore the SWAP is not used.

The SFWD crashing on the other hand is exactly the same. Currently working with TAC/R&D.

[Amir_Ayalon]

Hi

Can you please share SR / Task number ?

(amiray@checkpoint.com)

[Thomas_Eichelbu]

Hello, 

since many of my customers 1535 appliances reach 90% iam forced to run many RMA´s now ... 
We also always used the latest firmware, but we ran all blades and used IoC quite heavily ...
Again we were told to ran the lastest firmware and raise the IoC Feed interval to the maximum possible.


Since this interval is a global property defined in SmartConsole, i always run the same interval on all full GAiA gateways too and thats not ideal for my taste ... would be great to set a IoC feed interval per Threat Prevention profile ... 

Check Point should update its documentation regarding this behavior. And should warn people.
Also Check Point should move away from this low cost / trash eMMC flash.

So iam starting with my RMA´s now 

[ohadp]

Hi

Can you share your SR number?

[Werner133633]

Hi

How long did they take to replace and what did you have to do when the new (RMA) Unit arrived - I suppose a backup from old and restore to new one? - what version of firmware does the new one come in at?

I received an approved RMA letter today - so now I do not know the process and when it will arrive - will this Flash work till it arrive?

tx

[G_W_Albrecht]

- Usually, after RMA is approved, the replacement is due NBD - but all depends on the location this models replacement is on stock. So it can also take 3-4 days sometimes.

- If the replacement has the same firmware as the defective installed, backup from old and restore to new one is possible - you will only have to activate the replacement in WebGUI as the next step because the old license from the backup is applied (and will not work).

- If the firmware version is different, i would suggest to use USB media to install the version from defective following sk98549: How to re-image SMB / Quantum Spark appliances firmware version using USB Flash Drive

- Flash should work until the replacement arrives, but just to be save do the backup now 😉