Resolving VESID based on document type id and/or process id

[kukel]

I'm looking into resolving the VESID based on the document type id and/or process id, so outgoing messages that are already containing an SBDH can be validated without the need to pass in an explicit VESID. Incoming messages could be validated as well and an MLR could be created based on this.

Would this be feasible and what would be the best approach?

• Create a mapping file somewhere
• Add explicit info to the different validations
• How to deal with different versions/deprecation

[phax]

Well, this is a quite "common" issue, and I think from a Peppol perspective alone it would work very well (Map "document type ID" plus "process ID" to one "VES ID").
It just doesn't work for "non-standard" Peppol document types that are not UBL based, hence you don't have a "Customization ID" and a "Profile ID" in the document to be send. If you however already have the SBDH with a set "Document Type ID" and "Process ID" (from another source), than this should be doable quite easy.

Would you be willing to come up with an initial PR on this?

[kukel]

Ok, two different cases. I'll take on the former first (personal interest) ;-)

So an explicit mapping has your preference. Not (trying to) split fields and construct the VESID from it. Ok... But then there are still several approaches of which I'm not sure which one fits your vision best :

• There are already 2 'mapping' documents in https://github.com/phax/peppol-practical/tree/master/docs. Using https://github.com/phax/peppol-practical/blob/master/docs/doctypeid-mapping.xml and adding the VESID there (in group:artifact id format or with all versions
• Add Documen Type Identifier and Process Identifier information to the Validation classes (make it 'spi' like?) and use that.

And how to validate against the latest if that fails validate against the previous one (assuming it is not deprecated?) and...
And if all fails, what to report back? Validations against latest? Oldest?

[phax]

• The existing mapping files are for "known document types", so all Peppol Code List entries are in there
• The Document Type/Process to VESID mapping is however not Peppol specific, hence it should be decoupled
• Having an SPI interface sounds like a nice idea - the interface in "phive-api" and each "phive-rules-*" library offers the SPI implementations? Or do you think this is too much?

[kukel]

The SPI allows for a 'deploy' of a rule and becoming active. No need to add them anywhere in code or whatever so I'm in favour.
What would be good in this regard is to add the Documen Type/ Process IN the VESID.

I also found the peppol-common, with id's in there as objects, are those useful? Would make peppol validation rules dependent on this, but prevents typo's maybe

[phax]

Yes, I also find the SPI kind of sexy - and it could easily be aggregated and collected in an XML as necessary (similar to the existing document type and process ID listing).

What about the following data structure: we map from XMLDocumentDescriptor to VESID, where XMLDocumentDescriptor consists of the following fields:

String xmlRootElementNamespaceURI;
String xmlRootElementLocalName;
String customizationID;
String syntaxVersion;
String processID;

The first 4 values can be assembled to a Peppol Document type Identifer ({rootElementNamespaceUri}::{rootElementLocal}##{customizationId}::{syntaxVersion}) but would work for SBDH and for plain documents.

What do you think?

[kukel]

I was by chance doing already (almost) that but with the explicit use of EPredefinedDocumentTypeIdentifier

	IPeppolDocumentTypeIdentifierParts parts=EPredefinedDocumentTypeIdentifier.INVOICE_EN16931_PEPPOL_V30;
	
	System.out.println(EPredefinedDocumentTypeIdentifier.INVOICE_EN16931_PEPPOL_V30.getAllProcessIDs());
	System.out.println(parts.getRootNS());
	System.out.println(parts.getLocalName());
	System.out.println(parts.getCustomizationID());
	System.out.println(parts.getVersion());

It can have multiple process id's (weird btw that it is a IPeppolDocumentTypeIdentifierParts at the same time too, cannot explain that)

This class is generated as it seems according to the source of it

/**
 * This file was automatically generated.
 * Do NOT edit!
 */
@CodingStyleguideUnaware
public enum EPredefinedDocumentTypeIdentifier
    implements IPeppolPredefinedDocumentTypeIdentifier
{

And since it is strongly typed, I 'like' that.

I myself was thinking to pass this information on to the registry in the init of the validations so you can resolve the validation based on either the VESID and/or the Document/Process id. No need for a separate mapping table then either and the option to have the 'mapping' tested in unit tests in the individual validation rules components.

[kukel]

After getting to grips with the code (at least some more), I created, as a test,

• A class that extends the ValidationExecutorSet and takes a PredefinedDocumentTypeIdentifier and EPredefinedProcessIdentifier as additional strongly typed parameters
• A class that servers as a registry lookup and populates the mapping dynamically
• Call this initial class in PeppolValidation3_12_0 (as a 'test')

Custom Executor Set

public class PeppolValidationExecutorSet<SOURCETYPE extends IValidationSource>
	extends ValidationExecutorSet<SOURCETYPE> {

private EPredefinedProcessIdentifier m_aPI;
private EPredefinedDocumentTypeIdentifier m_aDTI;

public PeppolValidationExecutorSet(VESID aID, String sDisplayName, boolean bDeprecated,
		EPredefinedDocumentTypeIdentifier aDTI, EPredefinedProcessIdentifier aPI) {
	super(aID, sDisplayName, bDeprecated);
	m_aDTI = aDTI;
	m_aPI = aPI;
}

@Nonnull
@SafeVarargs
public static <ST extends IValidationSource> PeppolValidationExecutorSet<ST> create(@Nonnull final VESID aID,
		@Nonnull @Nonempty final String sDisplayName, final boolean bIsDeprecated,
		EPredefinedDocumentTypeIdentifier aDTI, EPredefinedProcessIdentifier aPI,
		@Nonnull final IValidationExecutor<ST>... aValidationExecutors) {

	// Value enforcers here

	final PeppolValidationExecutorSet<ST> ret = new PeppolValidationExecutorSet<>(aID, sDisplayName, bIsDeprecated,
			aDTI, aPI);
	for (final IValidationExecutor<ST> aItem : aValidationExecutors)
		ret.addExecutor(aItem);

	return ret;
}

public EPredefinedDocumentTypeIdentifier getDTI() {
	return m_aDTI;
}

public EPredefinedProcessIdentifier getPI() {
	return m_aPI;
}

}

Temporarily adapted PeppolValidation3_12_0.java

aRegistry.registerValidationExecutorSet (PeppolValidationExecutorSet.create (VID_OPENPEPPOL_INVOICE_V3,
                  "OpenPEPPOL Invoice" + sVersion + sAkaVersionBilling,
                  bNotDeprecated,
                   EPredefinedDocumentTypeIdentifier.urn_oasis_names_specification_ubl_schema_xsd_Invoice_2__Invoice__urn_cen_eu_en16931_2017_compliant_urn_fdc_peppol_eu_2017_poacc_billing_3_0__2_1,
                  EPredefinedProcessIdentifier.urn_fdc_peppol_eu_2017_poacc_billing_01_1_0,
                  ValidationExecutorXSD.create (EUBL21DocumentType.INVOICE),
                  _createXSLT (INVOICE_CEN),
                  _createXSLT (INVOICE_PEPPOL)));

Custom registry

@Immutable
public final class PeppolValidationKeyRegistry {

public static final ValidationExecutorSetRegistry<IValidationSourceXML> VES_REGISTRY = new ValidationExecutorSetRegistry<>();

public static final ICommonsOrderedMap<Pair<EPredefinedDocumentTypeIdentifier, EPredefinedProcessIdentifier>, VESID> dtiPiToVESIDMap = new CommonsLinkedHashMap<>();

static {
	PeppolValidation3_12_0.init(VES_REGISTRY);
	VES_REGISTRY.getAll()
			.forEach(x -> dtiPiToVESIDMap.put(
					new Pair(((PeppolValidationExecutorSet) x).getDTI(), ((PeppolValidationExecutorSet) x).getPI()),
					x.getID()));
}

private PeppolValidationKeyRegistry() {
}

@Nullable
public static IValidationExecutorSet<IValidationSourceXML> getFromDtiPiOrNull(
		@Nullable final EPredefinedDocumentTypeIdentifier aDTI, EPredefinedProcessIdentifier aPI) {
	Pair<EPredefinedDocumentTypeIdentifier, EPredefinedProcessIdentifier> dtiPi = new Pair<>(aDTI, aPI);
	return VES_REGISTRY.getOfID(dtiPiToVESIDMap.get(dtiPi));
}

public static void cleanupOnShutdown() {
	VES_REGISTRY.removeAll();
}
}

Unit test:

@Test
public void testResolveValidationRulesByDtiPi() throws Exception {

	//Both parameters can be retrieved from the SBDH and an additional method can be created that takes individual values like discussed
	IValidationExecutorSet<IValidationSourceXML> val = PeppolValidationKeyRegistry.getFromDtiPiOrNull(
			EPredefinedDocumentTypeIdentifier.urn_oasis_names_specification_ubl_schema_xsd_Invoice_2__Invoice__urn_cen_eu_en16931_2017_compliant_urn_fdc_peppol_eu_2017_poacc_billing_3_0__2_1,
            EPredefinedProcessIdentifier.urn_fdc_peppol_eu_2017_poacc_billing_01_1_0);
	
	assertEquals(PeppolValidation3_12_0.VID_OPENPEPPOL_INVOICE_V3, val.getID());
	
}

I prefer the strong typing, but I understand that this is maybe not possible for all (non-peppol) validations. Please comment.

The main work for me now is to create a full list of the VESID <-> DocumentType ID/Process ID. This is required anyway, regardless of the further approach.

[phax]

Sorry - had no time yesterday.
I think I understand - you want to integrate this directly in the registration of validation artefacts.
To be honest I am not quite convinced, if this is the best way forward, because:

• Not all validation artefacts have a mapping to Peppol
• Not for all Peppol documents types, predefined VES IDs are available

Let me try to come with a quick example

[kukel]

• Not all validation artefacts have a mapping to Peppol

It does not need to be always based on the Peppol identifiers (like in my example) and I'm not against using an XMLDocumentDescriptor but where possible, populate that based on strongly typed objects.

-Not for all Peppol documents types, predefined VES IDs are available
Isn't this a problem regardless of the strongly typed usage?

So it would still be possible to have it done from the registration of the artifact?

[kukel]

any feedback? I'd like to finish work on this ;-)

[kukel]

Validations like PeppolVAlidation.java need to have their private constructors removed when we want then to be used via SPI. Do you see a problem with this and/or an alternative besides making it public?

[kukel]

any feeddback on this?

[kukel]

I've done this:

    aRegistry.registerValidationExecutorSet (PeppolValidationExecutorSet.create (VID_OPENPEPPOL_INVOICE_V3,
				"OpenPEPPOL Invoice" + sVersion + sAkaVersionBilling,
				bNotDeprecated,
				new PeppolXMLDocumentDescriptor(
				        EPredefinedDocumentTypeIdentifier.urn_oasis_names_specification_ubl_schema_xsd_Invoice_2__Invoice__urn_cen_eu_en16931_2017_compliant_urn_fdc_peppol_eu_2017_poacc_billing_3_0__2_1,
				        EPredefinedProcessIdentifier.urn_fdc_peppol_eu_2017_poacc_billing_01_1_0),
				ValidationExecutorXSD.create (EUBL21DocumentType.INVOICE),
					 _createXSLT (INVOICE_CEN),
					 _createXSLT (INVOICE_PEPPOL)));

Where the PeppolXMLDocumentDescriptor with the DocumentTypeIdentifier and ProcessIdentifier is shorthand for

XMLDocumentDescriptor(xmlRootElementNamespaceURI, xmlRootElementLocalName, customizationID, syntaxVersion,processID)

And resolving based on the latter works.

Questions remaining now are:

• What to do with deprecated validations? Multiple VESID's can have the same DocumentTypeIdentifier/ProcessIdentifier? A fallback mechanisme (validate latest and if it fails validate latest -1 (or the only deprecated one?)
• In what source/module/.... to put the convenience PeppolXMLDocumentDescriptor and the PeppolValidationExecutorSet (or do you want to have a separate 'mapping' lookup registry instead of having it as part of the registry (based on the plain XMLDocumentDescriptor of course

Will commit to my git repos today so you can have a look without actual pull requests.

[phax]

In the meantime I created DDD - the Document Details Determinator - that can achieve exactly this: https://github.com/phax/ddd
It is outside of phase4 to be usable as versatile as possible. Additionally, the pseudo version latest was introduced in phive 8.x, so that DDD can always refer to group:artifact:latest if necessary. Combined with the strict semantic versioning requirement in phive 9.x I think we have a decsent, sustainable and extensible solution at hand - eventually just 2 years too late ;-)