out of memory on opening a signed container

[mbakhoff]

Hi!

Creating this issue as recommended by @rsarendus in #112

Here's my problem with digidoc4j 5.0.0:

1. create an asice container with a large file (250mb) and a small file (1mb). sign it with with any SK demo account. i've been using LT profile.
2. run the following snippet with -Xmx128M -Ddigidoc4j.mode=TEST

public static void main(String[] args) { // args[0] = path to asice
  Configuration.getInstance().setMaxFileSizeCachedInMemoryInMB(1);
  ContainerBuilder.aContainer().fromExistingFile(args[0]).build();
}

It will immediately blow up with OOM:

java.lang.OutOfMemoryError: Java heap space
	at org.apache.commons.io.output.AbstractByteArrayOutputStream.needNewBuffer(AbstractByteArrayOutputStream.java:106) ~[commons-io-2.8.0.jar:2.8.0]
	at org.apache.commons.io.output.AbstractByteArrayOutputStream.writeImpl(AbstractByteArrayOutputStream.java:135) ~[commons-io-2.8.0.jar:2.8.0]
	at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:66) ~[commons-io-2.8.0.jar:2.8.0]
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1159) ~[commons-io-2.8.0.jar:2.8.0]
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:878) ~[commons-io-2.8.0.jar:2.8.0]
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1135) ~[commons-io-2.8.0.jar:2.8.0]
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:854) ~[commons-io-2.8.0.jar:2.8.0]
	at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2240) ~[commons-io-2.8.0.jar:2.8.0]
	at eu.europa.esig.dss.utils.apache.impl.ApacheCommonsUtils.toByteArray(ApacheCommonsUtils.java:221) ~[dss-utils-apache-commons-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.utils.Utils.toByteArray(Utils.java:418) ~[dss-utils-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.spi.DSSUtils.toByteArray(DSSUtils.java:610) ~[dss-spi-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.spi.DSSUtils.toByteArray(DSSUtils.java:593) ~[dss-spi-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.xades.validation.DetachedSignatureResolver.createFromCommonDocument(DetachedSignatureResolver.java:74) ~[dss-xades-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.xades.validation.DetachedSignatureResolver.engineResolveURI(DetachedSignatureResolver.java:68) ~[dss-xades-5.9.d4j.1.jar:na]
	at org.apache.xml.security.utils.resolver.ResourceResolver.resolve(ResourceResolver.java:201) ~[xmlsec-2.2.4.jar:2.2.4]
	at org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Reference.java:436) ~[xmlsec-2.2.4.jar:2.2.4]
	at org.apache.xml.security.signature.Reference.calculateDigest(Reference.java:694) ~[xmlsec-2.2.4.jar:2.2.4]
	at org.apache.xml.security.signature.Reference.verify(Reference.java:788) ~[xmlsec-2.2.4.jar:2.2.4]
	at org.apache.xml.security.signature.Manifest.verifyReferences(Manifest.java:337) ~[xmlsec-2.2.4.jar:2.2.4]
	at org.apache.xml.security.signature.SignedInfo.verify(SignedInfo.java:286) ~[xmlsec-2.2.4.jar:2.2.4]
	at org.apache.xml.security.signature.XMLSignature.checkSignatureValue(XMLSignature.java:887) ~[xmlsec-2.2.4.jar:2.2.4]
	at eu.europa.esig.dss.xades.validation.XAdESSignatureIntegrityValidator.verify(XAdESSignatureIntegrityValidator.java:50) ~[dss-xades-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.spi.x509.SignatureIntegrityValidator.isSignatureIntact(SignatureIntegrityValidator.java:118) ~[dss-spi-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.spi.x509.SignatureIntegrityValidator.validate(SignatureIntegrityValidator.java:63) ~[dss-spi-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.xades.validation.XAdESSignature.checkSignatureIntegrity(XAdESSignature.java:894) ~[dss-xades-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.validation.DefaultAdvancedSignature.getSignatureCryptographicVerification(DefaultAdvancedSignature.java:358) ~[dss-document-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.validation.DefaultAdvancedSignature.getSigningCertificateToken(DefaultAdvancedSignature.java:388) ~[dss-document-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.validation.BaselineRequirementsChecker.containsSigningCertificate(BaselineRequirementsChecker.java:210) ~[dss-document-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.xades.validation.XAdESBaselineRequirementsChecker.hasBaselineBProfile(XAdESBaselineRequirementsChecker.java:178) ~[dss-xades-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.validation.DefaultAdvancedSignature.hasBProfile(DefaultAdvancedSignature.java:513) ~[dss-document-5.9.d4j.1.jar:na]
	at eu.europa.esig.dss.xades.validation.XAdESSignature.getDataFoundUpToLevel(XAdESSignature.java:1359) ~[dss-xades-5.9.d4j.1.jar:na]
	at org.digidoc4j.impl.asic.xades.XadesSignatureParser.parse(XadesSignatureParser.java:40) ~[digidoc4j-5.0.0.jar:na]
    at org.digidoc4j.impl.asic.AsicSignatureParser.createXadesSignature(AsicSignatureParser.java:43)
    at org.digidoc4j.impl.asic.AsicSignatureParser.parse(AsicSignatureParser.java:38)
    at org.digidoc4j.impl.asic.AsicContainerParser.parseSignatures(AsicContainerParser.java:276)
    at org.digidoc4j.impl.asic.AsicContainerParser.populateParseResult(AsicContainerParser.java:264)
    at org.digidoc4j.impl.asic.AsicContainerParser.read(AsicContainerParser.java:97)
    at org.digidoc4j.ContainerOpener.openAsicContainer(ContainerOpener.java:126)
    at org.digidoc4j.ContainerOpener.open(ContainerOpener.java:60)
    at org.digidoc4j.ContainerOpener.open(ContainerOpener.java:80)
    at org.digidoc4j.ContainerBuilder.openContainerFromFile(ContainerBuilder.java:266)
    at org.digidoc4j.impl.asic.asice.AsicEContainerBuilder.openContainerFromFile(AsicEContainerBuilder.java:31)
    at org.digidoc4j.ContainerBuilder.build(ContainerBuilder.java:129)
    at main()

The issue is in DetachedSignatureResolver::engineResolveURI which tries to load the full file in memory. My current workaround is to tweak AsicContainerParser::parseSignatures and change what is passed to the parser. I wrap each detached content document into a custom DigestDocument subclass that simpliy delegates getDigest to the original StreamDocument. Seems to work great for LT signed documents.

    private static class LazyDigestDocument extends DigestDocument {

        private final DSSDocument wrappedData;

        private LazyDigestDocument(DSSDocument data) {
            this.name = data.getName();
            this.wrappedData = data;
        }

        @Override
        public String getDigest(eu.europa.esig.dss.enumerations.DigestAlgorithm digestAlgorithm) {
            // there's no obvious reason why DetachedSignatureResolver::engineResolveURI couldn't use getDigest on
            // every DSSDocument but it doesn't so we have to create this wrapper hack
            String result = base64EncodeDigestMap.get(digestAlgorithm);
            if (result == null) {
                result = wrappedData.getDigest(digestAlgorithm);
                base64EncodeDigestMap.put(digestAlgorithm, result);
            }
            return result;
        }

        @Override
        public Digest getExistingDigest() {
            if (base64EncodeDigestMap.isEmpty()) // make sure we have at least one
                getDigest(eu.europa.esig.dss.enumerations.DigestAlgorithm.SHA256);
            return super.getExistingDigest();
        }
    }

Currently I have to use reflection and hacks to get this working. I would very happy if the library made it easier to implement custom signature parsing or have better large file support out of the box.

[smartman]

Just a wild guess. Underlying DSS has very similar bug fixed in new 5.11.RC1 version https://ec.europa.eu/digital-building-blocks/tracker/browse/DSS-2472