feat(transforms): add SES censorship evasion function #1812

boneskull · 2023-10-06T21:42:16Z

Description

Extracts the rewriteComment function from @endo/bundle-source/src/transform.js and exposes it as transformComment in the evade-censor export. This should be considered an experimental API.

Motivation

We could use this functionality in LavaMoat itself, so extracting it into a module lighter than @endo/bundle-source seems appropriate.

The function is not exported from the main entry point, nor documented publicly, as to avoid committing to a API we may not be ready to commit to. I can change this, if desired!

Notes

@babel/types is added as a production dependency, because the "public" API surface depends on types from this module.
Did not export the new function from the main entry point.
Added some unit tests.
Used homoglyphs.
Removed the dummy test index.
Added barebones README.md

Questions

Why do we make everything a CommentBlock?
Why do we replace the end-of-comment marker?
Is it preferable to use character escapes or no?

kriskowal

What do you think of exporting evadeCensor(source, sourceMap) => {source, sourceMap} for less public coupling?

kriskowal · 2023-10-06T21:45:08Z

packages/transforms/src/evade-censor.js

+    // (featuring homoglyphs for @kriskowal)
+    .replace(IMPORT_RE, '\u{406}\u{16D6}\u{420}\u{39F}\u{13D2}\u{422}$2')


summoning @mhofman, resident homoglyph idea haver. (I do not need credit)

Is there a reason to use homoglyphs in addition to, uh, lulz?

They have the benefit of preserving the column numbers for subsequent code on the same line. But mostly lulz.

kriskowal · 2023-10-06T21:49:58Z

packages/transforms/src/evade-censor.js

+    .replace(HTML_COMMENT_START_RE, '<!X-')
+    .replace(HTML_COMMENT_END_RE, '-X>')


Also potential homoglyph, e.g., \u2010.

kriskowal

Also would like:

unit test
integration test in compartment-mapper
refactor bundle-source in place to take on this dependency

Let’s be sure to separate the commit that changes the behavior to use homoglyphs so it can be easily reverted if necessary.

kriskowal · 2023-10-06T22:00:49Z

I defer to @michaelfig or @erights regarding the comment end marker escape and why we replace // with /* style comments. We should capture the reasoning in the code!

I have no strong feelings about how to escape homoglphys.

Also worth noting that there are other evasive transforms we hope to add one day to provide better compatibility with the ecosystem, particularly transforms that break strings that have censored terms.

boneskull · 2023-10-06T23:04:28Z

@kriskowal Which unit test do you mean?

erights · 2023-10-10T23:30:18Z

why we replace // with /* style comments. We should capture the reasoning in the code!

I do not remember being aware that we changed comments this way, so I also do not remember why. If I was aware at the time, my apologies for not capturing or at least leaving more clues.

kriskowal · 2023-10-12T20:27:45Z

@kriskowal Which unit test do you mean?

There is none as such and we are relying on integration tests inside bundle-source for the effect. I’m willing to continue in that vein.

boneskull · 2023-10-12T20:37:10Z

There is none as such and we are relying on integration tests inside bundle-source for the effect. I’m willing to continue in that vein.

For posterity: this PR does include some unit tests.

boneskull · 2023-10-12T20:51:09Z

What do you think of exporting evadeCensor(source, sourceMap) => {source, sourceMap} for less public coupling?

@kriskowal

This sounds like I'd need to copy makeLocationUnmapper() and other functions from bundle-source/src/transform.js here as well.

Was the intention to rehome the entirety of bundle-source/src/transform.js into the transform pkg? Apologies if that wasn't clear. And if so, you want it all in this PR (with multiple, logical commits)?

kriskowal · 2023-10-12T20:53:48Z

What do you think of exporting evadeCensor(source, sourceMap) => {source, sourceMap} for less public coupling?

@kriskowal

This sounds like I'd need to copy makeLocationUnmapper() and other functions from bundle-source/src/transform.js here as well.

Was the intention to rehome the entirety of bundle-source/src/transform.js into the transform pkg? Apologies if that wasn't clear. And if so, you want it all in this PR (with multiple, logical commits)?

I had not thought that many steps ahead, but that would seem to be necessary in order to decouple the public interface from the encapsulated dependencies, so I’m in favor.

boneskull · 2023-10-16T23:34:21Z

this is getting closer, but I need to add some more tests. I'm not sure how unit-y I can make them, though...

boneskull · 2023-10-17T21:03:53Z

@kriskowal Please let me know if my test fixtures in @endo/transforms are overkill

boneskull · 2023-10-17T21:04:26Z

I suppose I should update README.md with API docs too

kriskowal

The substance of this change is perfect. I challenge you to consider my comments about anticipating future changes in the names and paths of functions and modules.

packages/transforms/README.md

packages/bundle-source/src/nested-evaluate-and-get-exports.js

boneskull · 2023-10-20T22:28:06Z

@kriskowal OK, I've

Renamed the package (I did not change the version number, however)
Updated the README
Narrowed the API so that only a single function is exported. Renamed it
Created a couple additional adapter functions so that, ideally, swapping out the underlying AST implementation won't be so painful
Accidentally discovered how to implement function overloading via JSDoc

boneskull · 2023-10-21T05:16:07Z

linefeeds. lol

kriskowal

Left some homoglyph suggestions that should make comments look the same to a developer in a debugger without tripping the censor.

packages/evasive-transform/src/transform-comment.js

boneskull · 2023-10-23T22:18:57Z

OK. Don't touch anything. @kriskowal can I be done please

boneskull · 2023-10-23T22:20:28Z

8d9d2b7 was committed from windows where I am evidently not setup properly with gpg.

kriskowal · 2023-10-24T06:52:46Z

Thanks, @boneskull. You’re green. Would you like to squash some or all of the commits or shall I command Github to do so on your behalf?

… package This extracts everything from `@endo/bundle-source/src/transform.js` into a new package (formerly `@endo/transforms`). LavaMoat has need to consume the SES-censorship-evasion functionality, and it would be cumbersome to try to use `@endo/bundle-source` for this.

This jettisons `src/transform.js` and consumes `@endo/evasive-transform` instead.

This normalizes linebreaks to `lf`. Sorry, Windows contributors. Additionally, forces `.snap` and `.agar` to be binary, if not already recognized as such.

boneskull · 2023-10-25T22:28:59Z

@kriskowal Squashed

kriskowal reviewed Oct 6, 2023

View reviewed changes

boneskull force-pushed the boneskull/transform branch from fc26c2f to de80b75 Compare October 16, 2023 22:42

boneskull force-pushed the boneskull/transform branch 2 times, most recently from 7f6e715 to 059e724 Compare October 17, 2023 21:00

boneskull force-pushed the boneskull/transform branch 3 times, most recently from 564fe4b to 07c843b Compare October 19, 2023 00:14

kriskowal reviewed Oct 19, 2023

View reviewed changes

packages/transforms/README.md Outdated Show resolved Hide resolved

packages/transforms/README.md Outdated Show resolved Hide resolved

packages/transforms/README.md Outdated Show resolved Hide resolved

packages/bundle-source/src/nested-evaluate-and-get-exports.js Show resolved Hide resolved

boneskull force-pushed the boneskull/transform branch 2 times, most recently from 4e90108 to 83d0fb2 Compare October 20, 2023 22:24

boneskull force-pushed the boneskull/transform branch 2 times, most recently from c44e721 to 93d3a29 Compare October 21, 2023 04:15

kriskowal approved these changes Oct 21, 2023

View reviewed changes

packages/evasive-transform/src/transform-comment.js Outdated Show resolved Hide resolved

packages/evasive-transform/src/transform-comment.js Outdated Show resolved Hide resolved

packages/evasive-transform/src/transform-comment.js Outdated Show resolved Hide resolved

boneskull force-pushed the boneskull/transform branch 4 times, most recently from 943d3b9 to b81d6d2 Compare October 23, 2023 22:07

boneskull mentioned this pull request Oct 23, 2023

feat(core): ship some types LavaMoat/LavaMoat#732

Merged

boneskull added 3 commits October 25, 2023 15:28

chore(bundle-source): consume @endo/evasive-transform

00b2c39

This jettisons `src/transform.js` and consumes `@endo/evasive-transform` instead.

chore: add .gitattributes file

ce8f3f0

This normalizes linebreaks to `lf`. Sorry, Windows contributors. Additionally, forces `.snap` and `.agar` to be binary, if not already recognized as such.

boneskull force-pushed the boneskull/transform branch from b81d6d2 to ce8f3f0 Compare October 25, 2023 22:28

kriskowal merged commit 126d1b0 into endojs:master Oct 25, 2023

kriskowal mentioned this pull request Jan 25, 2024

fix(bundle-source): apply eval evasion for cjs modules too #2003

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(transforms): add SES censorship evasion function #1812

feat(transforms): add SES censorship evasion function #1812

boneskull commented Oct 6, 2023

kriskowal left a comment

kriskowal Oct 6, 2023

boneskull Oct 6, 2023

kriskowal Oct 12, 2023

kriskowal Oct 6, 2023

kriskowal left a comment •

edited

Loading

kriskowal commented Oct 6, 2023

boneskull commented Oct 6, 2023

erights commented Oct 10, 2023

kriskowal commented Oct 12, 2023

boneskull commented Oct 12, 2023

boneskull commented Oct 12, 2023

kriskowal commented Oct 12, 2023

boneskull commented Oct 16, 2023

boneskull commented Oct 17, 2023

boneskull commented Oct 17, 2023

kriskowal left a comment

boneskull commented Oct 20, 2023

boneskull commented Oct 21, 2023

kriskowal left a comment

boneskull commented Oct 23, 2023

boneskull commented Oct 23, 2023

kriskowal commented Oct 24, 2023

boneskull commented Oct 25, 2023

		// (featuring homoglyphs for @kriskowal)
		.replace(IMPORT_RE, '\u{406}\u{16D6}\u{420}\u{39F}\u{13D2}\u{422}$2')

		.replace(HTML_COMMENT_START_RE, '<!X-')
		.replace(HTML_COMMENT_END_RE, '-X>')

feat(transforms): add SES censorship evasion function #1812

feat(transforms): add SES censorship evasion function #1812

Conversation

boneskull commented Oct 6, 2023

Description

Motivation

Notes

Questions

kriskowal left a comment

Choose a reason for hiding this comment

kriskowal Oct 6, 2023

Choose a reason for hiding this comment

boneskull Oct 6, 2023

Choose a reason for hiding this comment

kriskowal Oct 12, 2023

Choose a reason for hiding this comment

kriskowal Oct 6, 2023

Choose a reason for hiding this comment

kriskowal left a comment • edited Loading

Choose a reason for hiding this comment

kriskowal commented Oct 6, 2023

boneskull commented Oct 6, 2023

erights commented Oct 10, 2023

kriskowal commented Oct 12, 2023

boneskull commented Oct 12, 2023

boneskull commented Oct 12, 2023

kriskowal commented Oct 12, 2023

boneskull commented Oct 16, 2023

boneskull commented Oct 17, 2023

boneskull commented Oct 17, 2023

kriskowal left a comment

Choose a reason for hiding this comment

boneskull commented Oct 20, 2023

boneskull commented Oct 21, 2023

kriskowal left a comment

Choose a reason for hiding this comment

boneskull commented Oct 23, 2023

boneskull commented Oct 23, 2023

kriskowal commented Oct 24, 2023

boneskull commented Oct 25, 2023

kriskowal left a comment •

edited

Loading