Device crashed after applying ZBF configuration via Netconf (also observed on CLI)

[swagner-de]

TAC Case: 691241896

During designing the ZBF configuration we have observed an ASR1002-HX crashing twice when a ZBF configuration was applied.

I attached 2 core dumps from the occurrence to the case. Regarding the earlier core dump, the device crashed after applying ZBF related configuration by CLI. I unfortunately cannot remember the configuration specifically.
The later core dump and the crash happened when applying ZBF configuration using netconf-yang. See the below snippet which was sent to the box using edit-config:

<native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
    <ip>
        <access-list>
            <extended xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-acl" operation="merge">
                <name>ACL-FWAAS-7e2a161a85fd454f8838bc4de9563333-OUT2IN</name>
                <access-list-seq-rule>
                    <sequence>10</sequence>
                    <ace-rule>
                        <action>permit</action>
                        <protocol>tcp</protocol>
                        <any/>
                        <dst-any/>
                        <dst-eq>www</dst-eq>
                    </ace-rule>
                </access-list-seq-rule>
                <access-list-seq-rule>
                <sequence>20</sequence>
                    <ace-rule>
                        <action>permit</action>
                        <protocol>udp</protocol>
                        <any/>
                        <dst-any/>
                        <dst-eq>domain</dst-eq>
                    </ace-rule>
                </access-list-seq-rule>
                <access-list-seq-rule>
                    <sequence>30</sequence>
                    <ace-rule>
                        <action>permit</action>
                        <protocol>icmp</protocol>
                        <any/>
                        <dst-any/>
                        <dst-eq-port2>echo</dst-eq-port2>
                    </ace-rule>
                </access-list-seq-rule>
            </extended>
            <extended xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-acl" operation="merge">
                <name>ACL-FWAAS-7e2a161a85fd454f8838bc4de9563333-IN2OUT</name>
                <access-list-seq-rule>
                    <sequence>10</sequence>
                    <ace-rule>
                        <action>permit</action>
                        <protocol>ip</protocol>
                        <any/>
                        <dst-any/>
                    </ace-rule>
                </access-list-seq-rule>
            </extended>
        </access-list>
    </ip>
    <policy>
        <class-map xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-policy" operation="merge">
            <name>CM-FWAAS-7e2a161a85fd454f8838bc4de9563333-IN2OUT</name>
            <type>inspect</type>
            <prematch>match-all</prematch>
            <match>
                <access-group>
                    <name>ACL-FWAAS-7e2a161a85fd454f8838bc4de9563333-IN2OUT</name>
                </access-group>
            </match>
        </class-map>
        <class-map xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-policy" operation="merge">
            <name>CM-FWAAS-7e2a161a85fd454f8838bc4de9563333-OUT2IN</name>
            <type>inspect</type>
            <prematch>match-all</prematch>
            <match>
                <access-group>
                    <name>ACL-FWAAS-7e2a161a85fd454f8838bc4de9563333-OUT2IN</name>
                </access-group>
            </match>
        </class-map>
        <policy-map xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-policy" operation="merge">
            <name>PM-FWAAS-7e2a161a85fd454f8838bc4de9563333-IN2OUT</name>
            <type>inspect</type>
            <class>
                <name>CM-FWAAS-7e2a161a85fd454f8838bc4de9563333-IN2OUT</name>
                <type>inspect</type>
                <policy>
                    <action>inspect</action>
                </policy>
            </class>
            <class>
                <name>class-default</name>
                <policy>
                    <action>drop</action>
                    <log/>
                </policy>
            </class>
        </policy-map>
        <policy-map xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-policy" operation="merge">
            <name>PM-FWAAS-7e2a161a85fd454f8838bc4de9563333-OUT2IN</name>
            <type>inspect</type>
            <class>
                <name>CM-FWAAS-7e2a161a85fd454f8838bc4de9563333-OUT2IN</name>
                <type>inspect</type>
                <policy>
                    <action>inspect</action>
                </policy>
            </class>
            <class>
                <name>class-default</name>
                <policy>
                    <action>drop</action>
                    <log/>
                </policy>
                </class>
        </policy-map>
    </policy>
	<parameter-map>
		<type>
			<inspect-global xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-policy">
			    <inspect>
			        <vrf operation="merge">
			            <id>7e2a161a85fd454f8838bc4de9563333</id>
			            <name>PAM-FWAAS-FWAAS-POLICE</name>
			        </vrf>
			    </inspect>
			</inspect-global>
		</type>
	</parameter-map>
	<zone>
	    <security xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-zone" operation="merge">
        	<id>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-out</id>
	    </security>
	</zone>
	<zone>
	    <security xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-zone" operation="merge">
        	<id>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-in</id>
	    </security>
	</zone>
    <zone-pair>
        <security xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-zone" operation="merge">
            <id>ZP-FWAAS-7e2a161a85fd454f8838bc4de9563333-IN2OUT</id>
            <source>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-in</source>
            <destination>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-out</destination>
            <service-policy>
                <type>
                    <inspect>PM-FWAAS-7e2a161a85fd454f8838bc4de9563333-IN2OUT</inspect>
                </type>
            </service-policy>
        </security>
        <security xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-zone" operation="merge">
            <id>ZP-FWAAS-7e2a161a85fd454f8838bc4de9563333-OUT2IN</id>
            <source>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-out</source>
            <destination>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-in</destination>
            <service-policy>
                <type>
                    <inspect>PM-FWAAS-7e2a161a85fd454f8838bc4de9563333-OUT2IN</inspect>
                </type>
            </service-policy>
        </security>
    </zone-pair>
    <interface xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
        <BD-VIF>
            <name>6856</name>
            <zone-member xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-zone" operation="merge">
                <security>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-in</security>
            </zone-member>
            <redundancy>
                <rii><id>6856</id></rii>
            </redundancy>
        </BD-VIF>
    </interface>
    <interface xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
        <BD-VIF>
            <name>7154</name>
            <zone-member xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-zone" operation="merge">
                <security>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-in</security>
            </zone-member>
            <redundancy>
                <rii><id>7154</id></rii>
            </redundancy>
        </BD-VIF>
    </interface>
    <interface xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
        <BD-VIF>
            <name>7005</name>
            <zone-member xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-zone" operation="merge">
                <security>ZN-FWAAS-7e2a161a85fd454f8838bc4de9563333-out</security>
            </zone-member>
            <redundancy>
                <rii><id>7005</id></rii>
            </redundancy>
        </BD-VIF>
    </interface>
</native>

[swagner-de]

A Cisco Bug has been raised for this. Description provided:

It turned that the Firewall registers with NAT to cache firewall sessions. When NAT runs, it puts this cached session in the firewall packet sub-block. The crash occurs because the firewall session is not valid. (the memory it points to is all 0's). What appears to have happened is that one (or more) sessions could not be torn down (e.g. reference count did not reach 0) When firewall is unconfiguring it can eventually give up. Hence, and based on this analysis, the Development team are working on fixing the code to avoid this behavior again.