Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
796570686578
Collaborator

Is it possible to analyze kernel crash files in Gaia?

Hello everyone!

 

I know this is a somewhat odd question, but I would like to learn to analyze kernel crash files in Gaia Kernel 3.10 (R80.40) myself out of interest and to get a better understanding for it. I have never done this before so I am not sure if it is possible at all, but I am stuck now.. If you have any other advice, I appreciate it

Since our beloved Gaia can't install packages out of the box, my first hurdle was to install crash utility which I was able to do by installing a Debian chroot.(sk164752 - Installing DOOM on Gaia

After a few hours of troubleshooting and ~50 open Browser tabs I at least got some output with the crash utility.

 

the /boot/vmlinux file is the decompressed version of the vmlinuz-3.10.0-957.21.3cpx86_64 minus the gzip header.

dd if=vmlinuz bs=1 skip=24584 | zcat > vmlinux

 

root@LABCPR8040DEBUG:/# crash -d 15 /boot/System.map-3.10.0-957.21.3cpx86_64 /boot/vmlinux /home/admin/2021-xx-xx-1356/vmcore_zero64 

crash 7.0.8
Copyright (C) 2002-2014  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
compressed kdump: header->utsname.machine: x86_64
compressed kdump: memory bitmap offset: 5000
diskdump_data: 
          filename: /home/admin/2021-xx-xx-1356/vmcore_zero64
             flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) 
               dfd: 3
               ofp: 0
      machine_type: 62 (EM_X86_64)

            header: 1e98fe0
           signature: "KDUMP   "
      header_version: 6
             utsname:
               sysname: Linux
              nodename: firewallname
               release: 3.10.0-957.21.3cpx86_64
               version: #1 SMP Mon Jun 28 22:58:55 IDT 2021
               machine: x86_64
            domainname: domainname
           timestamp:
                tv_sec: xxxxxxx
               tv_usec: 0
              status: 1 (DUMP_DH_COMPRESSED_ZLIB)
          block_size: 4096
        sub_hdr_size: 4
       bitmap_blocks: 1056
           max_mapnr: 17301504
    total_ram_blocks: 0
       device_blocks: 0
      written_blocks: 0
         current_cpu: 0
             nr_cpus: 32
      tasks[nr_cpus]: 0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0

        sub_header: 0 (n/a)

  sub_header_kdump: 1e99ff0 
           phys_base: 0
          dump_level: 31 (0x1f) (|DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
               split: 0
           start_pfn: (unused)
             end_pfn: (unused)
   offset_vmcoreinfo: 15616 (0x3d00)
     size_vmcoreinfo: 1719 (0x6b7)
                      OSRELEASE=3.10.0-957.21.3cpx86_64
                      PAGESIZE=4096
                      SYMBOL(init_uts_ns)=ffffffff81c1a300
                      SYMBOL(node_online_map)=ffffffff81d5c588
                      SYMBOL(swapper_pg_dir)=ffffffff81c12000
                      SYMBOL(_stext)=ffffffff81000000
                      SYMBOL(vmap_area_list)=ffffffff81c7c690
                      SYMBOL(mem_section)=ffffffff81fdf340
                      LENGTH(mem_section)=4096
                      SIZE(mem_section)=32
                      OFFSET(mem_section.section_mem_map)=0
                      SIZE(page)=56
                      SIZE(pglist_data)=16192
                      SIZE(zone)=1664
                      SIZE(free_area)=72
                      SIZE(list_head)=16
                      SIZE(nodemask_t)=8
                      OFFSET(page.flags)=0
                      OFFSET(page._count)=28
                      OFFSET(page.mapping)=8
                      OFFSET(page.lru)=32
                      OFFSET(page._mapcount)=24
                      OFFSET(page.private)=48
                      OFFSET(pglist_data.node_zones)=0
                      OFFSET(pglist_data.nr_zones)=16000
                      OFFSET(pglist_data.node_start_pfn)=16008
                      OFFSET(pglist_data.node_spanned_pages)=16024
                      OFFSET(pglist_data.node_id)=16032
                      OFFSET(zone.free_area)=136
                      OFFSET(zone.vm_stat)=1112
                      OFFSET(zone.spanned_pages)=1512
                      OFFSET(free_area.free_list)=0
                      OFFSET(list_head.next)=0
                      OFFSET(list_head.prev)=8
                      OFFSET(vmap_area.va_start)=0
                      OFFSET(vmap_area.list)=48
                      LENGTH(zone.free_area)=11
                      SYMBOL(log_buf)=ffffffff81c4a440
                      SYMBOL(log_buf_len)=ffffffff81c4a438
                      SYMBOL(log_first_idx)=ffffffff81f59410
                      SYMBOL(log_next_idx)=ffffffff81f59400
                      SIZE(log)=16
                      OFFSET(log.ts_nsec)=0
                      OFFSET(log.len)=8
                      OFFSET(log.text_len)=10
                      OFFSET(log.dict_len)=12
                      LENGTH(free_area.free_list)=4
                      NUMBER(NR_FREE_PAGES)=0
                      NUMBER(PG_lru)=5
                      NUMBER(PG_private)=11
                      NUMBER(PG_swapcache)=16
                      NUMBER(PG_slab)=7
                      NUMBER(PG_head_mask)=16384
                      NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
                      SYMBOL(free_huge_page)=ffffffff811f2f50
                      NUMBER(phys_base)=0
                      SYMBOL(init_level4_pgt)=ffffffff81c12000
                      SYMBOL(node_data)=ffffffff81d59ac0
                      LENGTH(node_data)=64
                      KERNELOFFSET=0
                      NUMBER(KERNEL_IMAGE_SIZE)=268435456
                      CRASHTIME=1629281181
                               offset_note: 4200 (0x1068)
           size_note: 13136 (0x3350)
  num_prstatus_notes: 32
           notes_buf: 1e9b000
            notes[0]: 1e9b000
            notes[1]: 1e9b164
            notes[2]: 1e9b2c8
            notes[3]: 1e9b42c
            notes[4]: 1e9b590
            notes[5]: 1e9b6f4
            notes[6]: 1e9b858
            notes[7]: 1e9b9bc
            notes[8]: 1e9bb20
            notes[9]: 1e9bc84
            notes[10]: 1e9bde8
            notes[11]: 1e9bf4c
            notes[12]: 1e9c0b0
            notes[13]: 1e9c214
            notes[14]: 1e9c378
            notes[15]: 1e9c4dc
            notes[16]: 1e9c640
            notes[17]: 1e9c7a4
            notes[18]: 1e9c908
            notes[19]: 1e9ca6c
            notes[20]: 1e9cbd0
            notes[21]: 1e9cd34
            notes[22]: 1e9ce98
            notes[23]: 1e9cffc
            notes[24]: 1e9d160
            notes[25]: 1e9d2c4
            notes[26]: 1e9d428
            notes[27]: 1e9d58c
            notes[28]: 1e9d6f0
            notes[29]: 1e9d854
            notes[30]: 1e9d9b8
            notes[31]: 1e9db1c
  NT_PRSTATUS_offset: 1068
                      11cc
                      1330
                      1494
                      15f8
                      175c
                      18c0
                      1a24
                      1b88
                      1cec
                      1e50
                      1fb4
                      2118
                      227c
                      23e0
                      2544
                      26a8
                      280c
                      2970
                      2ad4
                      2c38
                      2d9c
                      2f00
                      3064
                      31c8
                      332c
                      3490
                      35f4
                      3758
                      38bc
                      3a20
                      3b84
    offset_eraseinfo: 0 (0x0)
      size_eraseinfo: 0 (0x0)
        start_pfn_64: (unused)
          end_pfn_64: (unused)
        max_mapnr_64: 17301504 (0x1080000)

       data_offset: 425000
        block_size: 4096
       block_shift: 12
            bitmap: 7f390bae9010
        bitmap_len: 4325376
         max_mapnr: 17301504 (0x1080000)
   dumpable_bitmap: 7f390b6c8010
              byte: 0
               bit: 0
   compressed_page: 1ec6790
         curbufptr: 0

 page_cache_hdr[0]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1eb6780
        pg_hit_count: 0
 page_cache_hdr[1]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1eb7780
        pg_hit_count: 0
 page_cache_hdr[2]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1eb8780
        pg_hit_count: 0
 page_cache_hdr[3]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1eb9780
        pg_hit_count: 0
 page_cache_hdr[4]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1eba780
        pg_hit_count: 0
 page_cache_hdr[5]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ebb780
        pg_hit_count: 0
 page_cache_hdr[6]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ebc780
        pg_hit_count: 0
 page_cache_hdr[7]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ebd780
        pg_hit_count: 0
 page_cache_hdr[8]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ebe780
        pg_hit_count: 0
 page_cache_hdr[9]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ebf780
        pg_hit_count: 0
page_cache_hdr[10]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ec0780
        pg_hit_count: 0
page_cache_hdr[11]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ec1780
        pg_hit_count: 0
page_cache_hdr[12]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ec2780
        pg_hit_count: 0
page_cache_hdr[13]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ec3780
        pg_hit_count: 0
page_cache_hdr[14]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ec4780
        pg_hit_count: 0
page_cache_hdr[15]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 1ec5780
        pg_hit_count: 0

    page_cache_buf: 1eb6780
       evict_index: 0
         evictions: 0
          accesses: 0
      cached_reads: 0 
       valid_pages: 1eae370
readmem: read_diskdump() 
crash: pv_init_ops exists: ARCH_PVOPS
compressed kdump: phys_base: 0
gdb /boot/vmlinux 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

crash: /boot/vmlinux: no debugging data available

 

 

At least I have some output now but there are still two things bugging me.

This GDB was configured as "x86_64-unknown-linux-gnu"...

crash: /boot/vmlinux: no debugging data available

 

From my research I think that the kernel-debuginfo, which is needed for analysis, is missing. please correct me if I'm wrong though.

Now  that I have already invested a lot of time and learned a lot I want to ask if there is a debuginfo available for the kernel or if I am completely wasting my time here 😉

Any help, advice or methods are appreciated!

 

Thanks for reading.

 

0 Kudos
4 Replies
_Val_
Admin
Admin

Not on your own, but you can ask TAC to assist

0 Kudos
796570686578
Collaborator

Yeah I know that TAC can analyze the files but I was wondering to if I am able to do it myself. But thanks for the help

0 Kudos
PhoneBoy
Admin
Admin

Pretty sure our kernel and modules are shipped stripped, which would be necessary for GDB to return a useful stack trace.
As a TAC engineer at Nokia back in the day, the utility of a stack trace was limited to determining (not always conclusively) whether it was an IPSO OS issue or a Firewall issue.
And yes, we had access to unstripped binaries 🙂

While I can appreciate the curiosity here, I can say from experience that this is best left to the experts.

0 Kudos
the_rock
Legend
Legend

There is a tool I used before for this, but I would prefer not to disclose what it was, because its not supported by Check Point.

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events