Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resourceListCreator: Different order of input resources leads to different results #274

Closed
1 of 6 tasks
RandomByte opened this issue Nov 16, 2020 · 0 comments · Fixed by SAP/ui5-builder#551 · May be fixed by SAP/ui5-builder#553
Closed
1 of 6 tasks

resourceListCreator: Different order of input resources leads to different results #274

RandomByte opened this issue Nov 16, 2020 · 0 comments · Fixed by SAP/ui5-builder#551 · May be fixed by SAP/ui5-builder#553
Assignees
@RandomByte
Labels
bug Something isn't working

Comments

@RandomByte
Copy link
Member

RandomByte commented Nov 16, 2020 •
edited
Loading

Expected Behavior

The order of resources as provided by workspace.byGlob should have no effect on the produced resources.json.

Current Behavior

Due to the minor randomization applied in byGlob (https://github.com/SAP/ui5-fs/blob/70d9cf42abc34faa73a86e2ccb75279a806c82d2/lib/AbstractReader.js#L45-L50), we spotted rare occurrences of differences in the generated resources.json in the current pomahawk voter.

This indicates that the correct generation of a resources.json currently depends on a certain order of resources. I'm still not sure what that order is.

Note that on my system, in >99% of all executions an equal resources.json has been produced. However, we do not know yet whether the content of that specific version is actually correct.

Example 1

--- local-pomahawk-build/maven-tooling (= majority of UI5 Tooling executions)
+++ local-pomahawk-build/ui5-tooling
@@ -3,8 +3,5 @@
             "module": "sap/ui/thirdparty/jqueryui/jquery-effects-shake.js",
             "size": 559,
             "format": "raw",
-            "isDebug": true,
-            "required": [
-                "sap/ui/thirdparty/jqueryui/jquery-effects-core.js"
-            ]
+            "isDebug": true
         },

Example 2

--- openui5/src/sap.ui.core/resources-0.json (= majority of executions)
+++ openui5/src/sap.ui.core/resources-44.json
@@ -1157,12 +1157,7 @@
            "module": "ui5loader.js",
            "size": 115623,
            "format": "raw",
-           "isDebug": true,
-           "required": [
-               "sap/ui/thirdparty/baseuri.js",
-               "sap/ui/thirdparty/es6-promise.js",
-               "sap/ui/thirdparty/es6-shim-nopromise.js"
-           ]
+           "isDebug": true
        },

Example 3

--- openui5/src/sap.ui.core/resources-0.json (= majority of executions)
+++ openui5/src/sap.ui.core/resources-68.json
@@ -7734,7 +7734,8 @@
            ],
            "condRequired": [
                "sap/ui/thirdparty/blanket.js"
-           ]
+           ],
+           "dynRequired": true
        },

Steps to reproduce the issue

Even though we already identified the mentioned pseudo-randomization as the cause (while not root cause), for future reference here's how we reproduced this locally. Basically we generated a lot of resources.json files and compared their sizes.

Note that it's probably easier to reproduce this now by reversing the order of resources in https://github.com/SAP/ui5-fs/blob/70d9cf42abc34faa73a86e2ccb75279a806c82d2/lib/AbstractReader.js#L45-L50 or by searching for -dbg resources and swapping them with their non--dbg counterparts.

1. Helper Scripts

start.sh

#!/bin/bash
set -e

mkdir -p dist/
mkdir -p logs/

trap "exit" INT TERM ERR
trap "kill 0" EXIT

./do-it.sh 0 &
./do-it.sh 20 &
./do-it.sh 40 &
./do-it.sh 60 &
./do-it.sh 80 &

wait

do-it.sh

#!/bin/bash
set -e

COUNTER=$1

while [ $COUNTER -ne $(($1 + 20)) ]
do
    echo " $COUNTER "
    UI5_CLI_NO_LOCAL=true ui5 build --dest "dist/dist-$COUNTER" --verbose --include-task generateResourcesJson > "logs/$COUNTER.log" 2>&1
    cp "dist/dist-$COUNTER/resources/sap/ui/core/resources.json" ./resources-$COUNTER.json
    COUNTER=$[$COUNTER +1]
done

2. Patch ui5-builder (optional)

If possible, patch the UI5 Builder at https://github.com/SAP/ui5-builder/blob/bc4a197090bfd9c2062d4dad92dc9b5b2eff147a/lib/builder/builder.js#L375 to only write out the resources.json

 								resource.setPath(resource.getPath().replace(
 									new RegExp(`^/resources/${project.metadata.namespace}`), ""));
 							}
+							if (!resource.getPath().endsWith("resources.json")) {
+								return;
+							}
 							return fsTarget.write(resource);
 						}));
 					});

3. Execute builds

Execute start.sh in a library of your choice. For example sap.ui.core.

4. Compare sizes of resulting JSON files

ls -lS resources-*

5. Cleanup once done analyzing

Cleanup

rm resources-*
rm -rf logs dist

Context

  • UI5 Module Version (output of ui5 --version when using the CLI): v2.6.4
  • Node.js Version: v14.15.0
  • npm Version: 6.14.8
  • OS/Platform: macOS

Affected components (if known)
@RandomByte RandomByte added the bug Something isn't working label Nov 16, 2020
@RandomByte RandomByte self-assigned this Nov 16, 2020
RandomByte added a commit to SAP/ui5-builder that referenced this issue Nov 24, 2020
@RandomByte
[FIX] generateResourcesJson: Make resources.json generation determini…
44665bb 
…stic

Stop analyzing debug-resources if they correspond to a non-debug
resource.

However, try to analyze the "source" version of a resource's content if
possible. In some cases the dependency analysis has problems with
minified code.

Finally, copy any dependency information of a non-debug resource to
the corresponding debug-resource.

This ensures that we generate the same resources.json, independently
from the order of debug and non-debug resources supplied to the
resourceListCreator processor.

Resolves SAP/ui5-tooling#274
@RandomByte RandomByte mentioned this issue Nov 24, 2020
Merged
RandomByte added a commit to SAP/ui5-builder that referenced this issue Nov 24, 2020
@RandomByte
[FIX] generateResourcesJson: Make resources.json generation determini…
90cd9d0 
…stic

Stop analyzing debug-resources if they correspond to a non-debug
resource.

However, try to analyze the "source" version of a resource's content if
possible. In some cases the dependency analysis has problems with
minified code.

Finally, copy any dependency information of a non-debug resource to
the corresponding debug-resource.

This ensures that we generate the same resources.json, independently
from the order of debug and non-debug resources supplied to the
resourceListCreator processor.

Resolves SAP/ui5-tooling#274
RandomByte added a commit to SAP/ui5-builder that referenced this issue Nov 26, 2020
@RandomByte
[FIX] generateResourcesJson: Make resources.json generation determini…
6144d90 
…stic

Stop analyzing debug-resources if they correspond to a non-debug
resource.

Only analze the non-debug resource and  copy any dependency information
to the corresponding debug-resource.

While this might not be the best approach given that analyzing the
source ("debug") version of a resource might result in finding more
dependencies (i.e. those mangled during minification), this change
ensures that we generate the same resources.json, independently
from the order of debug and non-debug resources supplied to the
resourceListCreator processor.

Resolves SAP/ui5-tooling#274
RandomByte added a commit to SAP/ui5-builder that referenced this issue Nov 26, 2020
@RandomByte
[FIX] generateResourcesJson: Make resources.json generation determini…
b9d397e 
…stic

Stop analyzing debug-resources if they correspond to a non-debug
resource.

Only analyze the non-debug resource and copy any dependency information
to the corresponding debug-resource.

While this might not be the best approach given that analyzing the
source ("debug") version of a resource might result in finding more
dependencies (i.e. those mangled during minification), this change
ensures that we generate the same resources.json, independently
from the order of debug and non-debug resources supplied to the
resourceListCreator processor.

Resolves SAP/ui5-tooling#274
@RandomByte RandomByte mentioned this issue Nov 26, 2020
Draft
RandomByte added a commit to SAP/ui5-builder that referenced this issue Nov 30, 2020
@RandomByte
[FIX] generateResourcesJson: Make resources.json generation determini…
41d3335 
…stic

Stop analyzing debug-resources if they correspond to a non-debug
resource.

Only analyze the non-debug resource and copy any dependency information
to the corresponding debug-resource.

While this might not be the best approach given that analyzing the
source ("debug") version of a resource might result in finding more
dependencies (i.e. those mangled during minification), this change
ensures that we generate the same resources.json, independently
from the order of debug and non-debug resources supplied to the
resourceListCreator processor.

Resolves SAP/ui5-tooling#274
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RandomByte RandomByte

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@RandomByte