feat: add fingerprints for JDK classes #63

MartinWitt · 2023-08-25T11:22:06Z

Fixes #62

watchdog-agent/src/main/java/io/github/algomaster99/Options.java

watchdog-agent/src/test/java/OptionsTest.java

algomaster99 · 2023-08-25T14:12:15Z

Oh, so some class is not recognized if we index the jdk classes.

MartinWitt · 2023-08-25T14:15:23Z

The error is:

Hash: a3c9aea8fdc0c03ddf28ce1134f7b0436cf518254b2522ed59b2b2d262f295d7
Candidate hash: eb05de28c45d2d312ef2b25b9bc74c7fed2058004a71e2346643ea8ed5ec8959
[MODIFIED]: jdk/internal/jrtfs/JrtFileSystemProvider

Super interesting that we have different hashes

algomaster99 · 2023-08-25T14:16:59Z

Here is the list of all classes:

[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor15
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor16
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor17
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor18
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor19
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor20
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor21
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor22
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor23
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedMethodAccessor1
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedMethodAccessor2
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedConstructorAccessor24
[NOT WHITELISTED]: jdk/internal/reflect/GeneratedMethodAccessor3
[MODIFIED]: jdk/internal/jrtfs/JrtFileSystemProvider
[MODIFIED]: jdk/internal/jrtfs/JrtFileSystem
[MODIFIED]: jdk/internal/jrtfs/JrtPath
[MODIFIED]: jdk/internal/jrtfs/JrtFileStore
[MODIFIED]: jdk/internal/jrtfs/SystemImage
[MODIFIED]: jdk/internal/jrtfs/SystemImage$1
[MODIFIED]: jdk/internal/jrtfs/ExplodedImage
[MODIFIED]: jdk/internal/jrtfs/SystemImage$2
[MODIFIED]: jdk/internal/jimage/ImageReader
[MODIFIED]: jdk/internal/jimage/ImageReader$SharedImageReader
[MODIFIED]: jdk/internal/jimage/BasicImageReader
[MODIFIED]: jdk/internal/jimage/ImageStrings
[MODIFIED]: jdk/internal/jimage/ImageReader$Node
[MODIFIED]: jdk/internal/jimage/ImageReader$Directory
[MODIFIED]: jdk/internal/jimage/ImageReader$LinkNode
[MODIFIED]: jdk/internal/jimage/ImageReader$Resource
[MODIFIED]: jdk/internal/jimage/BasicImageReader$1
[MODIFIED]: jdk/internal/jimage/BasicImageReader$2
[MODIFIED]: jdk/internal/jimage/ImageHeader
[MODIFIED]: jdk/internal/jimage/ImageStringsReader
[MODIFIED]: jdk/internal/jimage/decompressor/Decompressor
[MODIFIED]: jdk/internal/jrtfs/JrtDirectoryStream
[MODIFIED]: jdk/internal/jrtfs/JrtDirectoryStream$1
[MODIFIED]: jdk/internal/jimage/ImageLocation
[MODIFIED]: jdk/internal/jimage/ImageReader$SharedImageReader$LocationVisitor
[MODIFIED]: jdk/internal/jimage/ImageBufferCache
[MODIFIED]: jdk/internal/jimage/ImageBufferCache$1
[MODIFIED]: jdk/internal/jimage/ImageBufferCache$2
[MODIFIED]: jdk/internal/jrtfs/JrtFileAttributes
[MODIFIED]: jdk/internal/jrtfs/JrtFileSystem$1

I am also getting some un-whitelisted classes.

MartinWitt · 2023-08-25T14:19:27Z

The GeneratedClasses are a Result of reflection and should be detectable hopefully, see https://support.oracle.com/knowledge/Middleware/1669883_1.html.

algomaster99 · 2023-08-25T14:23:42Z

Okay. Should I look into it, or are you doing it?

MartinWitt · 2023-08-25T14:24:28Z

I am doing it

algomaster99 · 2023-08-25T16:10:18Z

@MartinWitt added method to skip the ASM generated classes. We can merge this once we identify the difference between the JDK classes.

algomaster99 · 2023-08-25T16:18:24Z

--- foo.txt/who-2023-08-25 18:13:56.580294255 +0200iff -u foo.txt moduleFinder.txt                                                                                                                                                                                                             ✔  at 18:14:15 
+++ moduleFinder.txt    2023-08-25 18:14:15.760310815 +0200
@@ -1,15 +1,15 @@
 17.0.5
-Classfile /home/aman/personal/who-are-you/foo.class
-  Last modified Aug 25, 2023; size 13513 bytes
-  SHA-256 checksum a3c9aea8fdc0c03ddf28ce1134f7b0436cf518254b2522ed59b2b2d262f295d7
+Classfile /home/aman/personal/who-are-you/moduleFinder.class
+  Last modified Aug 25, 2023; size 13539 bytes
+  SHA-256 checksum 91909cea1f05909dccbd419b6fd21f605a199685b16599d7304e742ea21bda26
   Compiled from "JrtFileSystemProvider.java"
 public final class jdk.internal.jrtfs.JrtFileSystemProvider extends java.nio.file.spi.FileSystemProvider
   minor version: 0
-  major version: 52
+  major version: 61
   flags: (0x0031) ACC_PUBLIC, ACC_FINAL, ACC_SUPER
   this_class: #35                         // jdk/internal/jrtfs/JrtFileSystemProvider
   super_class: #2                         // java/nio/file/spi/FileSystemProvider
-  interfaces: 0, fields: 2, methods: 30, attributes: 2
+  interfaces: 0, fields: 2, methods: 30, attributes: 3
 Constant pool:
     #1 = Methodref          #2.#3         // java/nio/file/spi/FileSystemProvider."<init>":()V
     #2 = Class              #4            // java/nio/file/spi/FileSystemProvider
@@ -447,15 +447,16 @@
   #434 = Utf8               value
   #435 = Utf8               SourceFile
   #436 = Utf8               JrtFileSystemProvider.java
-  #437 = Utf8               InnerClasses
-  #438 = Class              #439          // java/nio/file/DirectoryStream$Filter
-  #439 = Utf8               java/nio/file/DirectoryStream$Filter
-  #440 = Class              #441          // java/nio/file/DirectoryStream
-  #441 = Utf8               java/nio/file/DirectoryStream
-  #442 = Utf8               Filter
-  #443 = Class              #444          // jdk/internal/jrtfs/JrtFileSystemProvider$JrtFsLoader
-  #444 = Utf8               jdk/internal/jrtfs/JrtFileSystemProvider$JrtFsLoader
-  #445 = Utf8               JrtFsLoader
+  #437 = Utf8               NestMembers
+  #438 = Class              #439          // jdk/internal/jrtfs/JrtFileSystemProvider$JrtFsLoader
+  #439 = Utf8               jdk/internal/jrtfs/JrtFileSystemProvider$JrtFsLoader
+  #440 = Utf8               InnerClasses
+  #441 = Class              #442          // java/nio/file/DirectoryStream$Filter
+  #442 = Utf8               java/nio/file/DirectoryStream$Filter
+  #443 = Class              #444          // java/nio/file/DirectoryStream
+  #444 = Utf8               java/nio/file/DirectoryStream
+  #445 = Utf8               Filter
+  #446 = Utf8               JrtFsLoader
 {
   private volatile java.nio.file.FileSystem theFileSystem;
     descriptor: Ljava/nio/file/FileSystem;
@@ -622,10 +623,10 @@
          1: invokestatic  #76                 // Method java/util/Objects.requireNonNull:(Ljava/lang/Object;)Ljava/lang/Object;
          4: pop
          5: aload_0
-         6: invokespecial #82                 // Method checkPermission:()V
+         6: invokevirtual #82                 // Method checkPermission:()V
          9: aload_0
         10: aload_1
-        11: invokespecial #84                 // Method checkUri:(Ljava/net/URI;)V
+        11: invokevirtual #84                 // Method checkUri:(Ljava/net/URI;)V
         14: aload_2
         15: ldc           #88                 // String java.home
         17: invokeinterface #90,  2           // InterfaceMethod java/util/Map.containsKey:(Ljava/lang/Object;)Z
@@ -637,7 +638,7 @@
         34: checkcast     #38                 // class java/lang/String
         37: aload_1
         38: aload_2
-        39: invokespecial #98                 // Method newFileSystem:(Ljava/lang/String;Ljava/net/URI;Ljava/util/Map;)Ljava/nio/file/FileSystem;
+        39: invokevirtual #98                 // Method newFileSystem:(Ljava/lang/String;Ljava/net/URI;Ljava/util/Map;)Ljava/nio/file/FileSystem;
         42: areturn
         43: new           #102                // class jdk/internal/jrtfs/JrtFileSystem
         46: dup
@@ -846,7 +847,7 @@
     Code:
       stack=3, locals=3, args_size=2
          0: aload_0
-         1: invokespecial #82                 // Method checkPermission:()V
+         1: invokevirtual #82                 // Method checkPermission:()V
          4: aload_1
          5: invokevirtual #28                 // Method java/net/URI.getScheme:()Ljava/lang/String;
          8: aload_0
@@ -902,7 +903,7 @@
        113: invokespecial #47                 // Method java/lang/IllegalArgumentException."<init>":(Ljava/lang/String;)V
        116: athrow
        117: aload_0
-       118: invokespecial #223                // Method getTheFileSystem:()Ljava/nio/file/FileSystem;
+       118: invokevirtual #223                // Method getTheFileSystem:()Ljava/nio/file/FileSystem;
        121: new           #132                // class java/lang/StringBuilder
        124: dup
        125: invokespecial #134                // Method java/lang/StringBuilder."<init>":()V
@@ -950,7 +951,7 @@
     Code:
       stack=5, locals=5, args_size=1
          0: aload_0
-         1: invokespecial #82                 // Method checkPermission:()V
+         1: invokevirtual #82                 // Method checkPermission:()V
          4: aload_0
          5: getfield      #228                // Field theFileSystem:Ljava/nio/file/FileSystem;
          8: astore_1
@@ -1031,12 +1032,12 @@
     Code:
       stack=2, locals=2, args_size=2
          0: aload_0
-         1: invokespecial #82                 // Method checkPermission:()V
+         1: invokevirtual #82                 // Method checkPermission:()V
          4: aload_0
          5: aload_1
-         6: invokespecial #84                 // Method checkUri:(Ljava/net/URI;)V
+         6: invokevirtual #84                 // Method checkUri:(Ljava/net/URI;)V
          9: aload_0
-        10: invokespecial #223                // Method getTheFileSystem:()Ljava/nio/file/FileSystem;
+        10: invokevirtual #223                // Method getTheFileSystem:()Ljava/nio/file/FileSystem;
         13: areturn
       LineNumberTable:
         line 222: 0
@@ -1517,7 +1518,10 @@
       throws java.io.IOException
 }
 SourceFile: "JrtFileSystemProvider.java"
+NestMembers:
+  jdk/internal/jrtfs/JrtFileSystemProvider$JrtFsLoader
+  jdk/internal/jrtfs/JrtFileSystemProvider$1
 InnerClasses:
   #198;                                   // class jdk/internal/jrtfs/JrtFileSystemProvider$1
-  public static #442= #438 of #440;       // Filter=class java/nio/file/DirectoryStream$Filter of class java/nio/file/DirectoryStream
-  private static #445= #443 of #35;       // JrtFsLoader=class jdk/internal/jrtfs/JrtFileSystemProvider$JrtFsLoader of class jdk/internal/jrtfs/JrtFileSystemProvider
+  public static #445= #441 of #443;       // Filter=class java/nio/file/DirectoryStream$Filter of class java/nio/file/DirectoryStream
+  private static #446= #438 of #35;       // JrtFsLoader=class jdk/internal/jrtfs/JrtFileSystemProvider$JrtFsLoader of class jdk/internal/jrtfs/JrtFileSystemProvider

There is at least a difference in class file versions. This is so weird. Why is it even getting a class from Java8 because I use 17? 😞

I used javap -version -l -v -p -c -s -sysinfo moduleFinder to get the bytecode instructions.

MartinWitt · 2023-08-30T10:43:04Z

Okay, the ModuleFinder uses JavaHome to search for all modules. This can be an absolutely incorrect assumption.

MartinWitt · 2023-08-30T12:19:11Z

Okay I found three different versions of the same class file with the modulefinder. Will investigate.

MartinWitt · 2023-09-01T12:22:00Z

I found the problem in a way too long debugging section. We will switch from modulefinder to classgraph.

This reverts commit c6eddf3.

algomaster99

Thank you! Let's merge this tomorrow :D

...-commons/src/main/java/io/github/algomaster99/terminator/commons/fingerprint/JdkIndexer.java

...mons/src/main/java/io/github/algomaster99/terminator/commons/fingerprint/provenance/Jdk.java

watchdog-agent/src/main/java/io/github/algomaster99/Terminator.java

algomaster99 · 2023-09-04T18:50:23Z

watchdog-agent/src/main/java/io/github/algomaster99/Terminator.java

+            // FIXME: we need to check inner classes without loading them. Maybe add the hashes for inner classes in the
+            // fingerprints?


I think it is added by default.

public class A { static class B { } }

will be compiled to A.class and A$B.class. Or do you mean something else?

It will be compiled to a single class file. The files are currently missing in the index for jars.

watchdog-agent/src/test/java/AgentTest.java

Co-authored-by: Aman Sharma <[email protected]>

This reverts commit 6ff64f3.

feat: add fingerprints for JDK classes

16775bc

MartinWitt mentioned this pull request Aug 25, 2023

Explore hashing JDK classes #62

Closed

use faster gen with JDK methods

aabd918

algomaster99 requested changes Aug 25, 2023

View reviewed changes

watchdog-agent/src/main/java/io/github/algomaster99/Options.java Outdated Show resolved Hide resolved

watchdog-agent/src/test/java/OptionsTest.java Outdated Show resolved Hide resolved

MartinWitt added 5 commits August 25, 2023 15:23

up

76bfaaa

up

aab672e

up

6553033

Merge branch 'main' into molecular-puma

891d80a

spotless

441743b

MartinWitt and others added 4 commits August 25, 2023 16:36

add more prints and 2 class files

61e87cd

Merge branch 'main' into molecular-puma

a175ef1

Please spotless

cac3eac

fix: skip classes that are generated by ASM

96237d0

MartinWitt added 6 commits September 1, 2023 16:17

up

fab4705

up

c787df7

up

c6245a2

up

2c88cb7

Merge branch 'main' into molecular-puma

8101634

up

3dc7f3c

MartinWitt added 16 commits September 4, 2023 14:43

up

92d6e4d

up

8c3859b

/ ->.

e5f4bfe

also allow JDK package

8db8322

enbale scanning of private classes

251e9c4

remove logging

e89a183

up

93a5620

reduce log

fea9374

up

bcadc30

up

29bf50d

spotless

c5770f8

cleanup

dc7d9df

up

54591f3

remove old code

f79c388

try removing flag

c6eddf3

Revert "try removing flag"

3c2b52b

This reverts commit c6eddf3.

MartinWitt requested a review from algomaster99 September 4, 2023 16:03

algomaster99 reviewed Sep 4, 2023

View reviewed changes

add contract

a4cc082

algomaster99 reviewed Sep 4, 2023

View reviewed changes

watchdog-agent/src/test/java/AgentTest.java Outdated Show resolved Hide resolved

algomaster99 and others added 7 commits September 4, 2023 21:16

Refactor test out of nested class

b80d078

Update watchdog-agent/src/test/java/AgentTest.java

ac2ffaa

Co-authored-by: Aman Sharma <[email protected]>

add org.xml and netscape

6b59668

remove old class

6ff64f3

remove sysout from testcase

ab259df

Revert "remove old class"

cc9fd5a

This reverts commit 6ff64f3.

fix testcase

e37fbf9

algomaster99 approved these changes Sep 5, 2023

View reviewed changes

algomaster99 merged commit 19191f4 into main Sep 5, 2023
6 checks passed

algomaster99 deleted the molecular-puma branch September 5, 2023 12:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add fingerprints for JDK classes #63

feat: add fingerprints for JDK classes #63

MartinWitt commented Aug 25, 2023 •

edited by algomaster99

Loading

algomaster99 commented Aug 25, 2023

MartinWitt commented Aug 25, 2023 •

edited

Loading

algomaster99 commented Aug 25, 2023 •

edited

Loading

MartinWitt commented Aug 25, 2023

algomaster99 commented Aug 25, 2023

MartinWitt commented Aug 25, 2023

algomaster99 commented Aug 25, 2023

algomaster99 commented Aug 25, 2023 •

edited

Loading

MartinWitt commented Aug 30, 2023

MartinWitt commented Aug 30, 2023 •

edited

Loading

MartinWitt commented Sep 1, 2023

algomaster99 left a comment

algomaster99 Sep 4, 2023

MartinWitt Sep 5, 2023

		// FIXME: we need to check inner classes without loading them. Maybe add the hashes for inner classes in the
		// fingerprints?

feat: add fingerprints for JDK classes #63

feat: add fingerprints for JDK classes #63

Conversation

MartinWitt commented Aug 25, 2023 • edited by algomaster99 Loading

algomaster99 commented Aug 25, 2023

MartinWitt commented Aug 25, 2023 • edited Loading

algomaster99 commented Aug 25, 2023 • edited Loading

MartinWitt commented Aug 25, 2023

algomaster99 commented Aug 25, 2023

MartinWitt commented Aug 25, 2023

algomaster99 commented Aug 25, 2023

algomaster99 commented Aug 25, 2023 • edited Loading

MartinWitt commented Aug 30, 2023

MartinWitt commented Aug 30, 2023 • edited Loading

MartinWitt commented Sep 1, 2023

algomaster99 left a comment

Choose a reason for hiding this comment

algomaster99 Sep 4, 2023

Choose a reason for hiding this comment

MartinWitt Sep 5, 2023

Choose a reason for hiding this comment

MartinWitt commented Aug 25, 2023 •

edited by algomaster99

Loading

MartinWitt commented Aug 25, 2023 •

edited

Loading

algomaster99 commented Aug 25, 2023 •

edited

Loading

algomaster99 commented Aug 25, 2023 •

edited

Loading

MartinWitt commented Aug 30, 2023 •

edited

Loading