Nuclei grinds to a halt on HTTP templates with many paths as the internal valuesMap starts to grow beyond expectation #4705

Nirusu · 2024-01-30T17:10:11Z

Nirusu
Jan 30, 2024

Nuclei version:

3.1.7

Current Behavior:

On a large template with many HTTP request across different request methods, Nuclei starts becoming very slow.
I mainly discovered this after auto-generating some templates for convenience in a pipeline.

From looking with a debugger what's happening, the internal valuesMap is growing way larger than anticipated. Given that plenty of places in the code iterate over this map, a large valueMap is completely killing the performance of Nuclei and making it basically unusable with the given template.

I assume that using a fuzzer attack type might be better in this case and it would likely also be my go-to workaround (haven't tested it, though).
But I wonder if there is a way this can also be done with "simple requests".

Expected Behavior:

Nuclei does not become this slow.

Steps To Reproduce:

Use this template:

id: make-nuclei-slow

info:
  name: make-nuclei-slow
  author: Nirusu
  severity: high

http:
  - method: GET
    path:
      - "{{BaseURL}}/get/1"
      - "{{BaseURL}}/get/2"
      - "{{BaseURL}}/get/3"
      - "{{BaseURL}}/get/4"
      - "{{BaseURL}}/get/5"
      - "{{BaseURL}}/get/6"
      - "{{BaseURL}}/get/7"
      - "{{BaseURL}}/get/8"
      - "{{BaseURL}}/get/9"
      - "{{BaseURL}}/get/10"
      - "{{BaseURL}}/get/11"
      - "{{BaseURL}}/get/12"
      - "{{BaseURL}}/get/13"
      - "{{BaseURL}}/get/14"
      - "{{BaseURL}}/get/15"
      - "{{BaseURL}}/get/16"
      - "{{BaseURL}}/get/17"
      - "{{BaseURL}}/get/18"
      - "{{BaseURL}}/get/19"
      - "{{BaseURL}}/get/20"
      - "{{BaseURL}}/get/21"
      - "{{BaseURL}}/get/22"
      - "{{BaseURL}}/get/23"
      - "{{BaseURL}}/get/24"
      - "{{BaseURL}}/get/25"
      - "{{BaseURL}}/get/26"
      - "{{BaseURL}}/get/27"
      - "{{BaseURL}}/get/28"
      - "{{BaseURL}}/get/29"
      - "{{BaseURL}}/get/30"
      - "{{BaseURL}}/get/31"
      - "{{BaseURL}}/get/32"
      - "{{BaseURL}}/get/33"
      - "{{BaseURL}}/get/34"
      - "{{BaseURL}}/get/35"
      - "{{BaseURL}}/get/36"
      - "{{BaseURL}}/get/37"
      - "{{BaseURL}}/get/38"
      - "{{BaseURL}}/get/39"
      - "{{BaseURL}}/get/40"
      - "{{BaseURL}}/get/41"
      - "{{BaseURL}}/get/42"
      - "{{BaseURL}}/get/43"
      - "{{BaseURL}}/get/44"
      - "{{BaseURL}}/get/45"
      - "{{BaseURL}}/get/46"
  - method: POST
    path:
      - "{{BaseURL}}/post/1"
      - "{{BaseURL}}/post/2"
      - "{{BaseURL}}/post/3"
      - "{{BaseURL}}/post/4"
      - "{{BaseURL}}/post/5"
      - "{{BaseURL}}/post/6"
      - "{{BaseURL}}/post/7"
      - "{{BaseURL}}/post/8"
      - "{{BaseURL}}/post/9"
      - "{{BaseURL}}/post/10"
      - "{{BaseURL}}/post/11"
      - "{{BaseURL}}/post/12"
      - "{{BaseURL}}/post/13"
      - "{{BaseURL}}/post/14"
      - "{{BaseURL}}/post/15"
      - "{{BaseURL}}/post/16"
      - "{{BaseURL}}/post/17"
      - "{{BaseURL}}/post/18"
      - "{{BaseURL}}/post/19"
      - "{{BaseURL}}/post/20"
      - "{{BaseURL}}/post/21"
      - "{{BaseURL}}/post/22"
      - "{{BaseURL}}/post/23"
  - method: PUT
    path:
      - "{{BaseURL}}/put/1"
      - "{{BaseURL}}/put/2"
      - "{{BaseURL}}/put/3"
      - "{{BaseURL}}/put/4"
      - "{{BaseURL}}/put/5"
      - "{{BaseURL}}/put/6"
      - "{{BaseURL}}/put/7"
      - "{{BaseURL}}/put/8"
      - "{{BaseURL}}/put/9"
      - "{{BaseURL}}/put/10"
      - "{{BaseURL}}/put/11"
      - "{{BaseURL}}/put/12"
      - "{{BaseURL}}/put/13"
      - "{{BaseURL}}/put/14"
      - "{{BaseURL}}/put/15"
      - "{{BaseURL}}/put/16"
      - "{{BaseURL}}/put/17"
      - "{{BaseURL}}/put/18"
      - "{{BaseURL}}/put/19"
      - "{{BaseURL}}/put/20"
  - method: DELETE
    path:
      - "{{BaseURL}}/delete/1"
      - "{{BaseURL}}/delete/2"
      - "{{BaseURL}}/delete/3"
      - "{{BaseURL}}/delete/4"
      - "{{BaseURL}}/delete/5"
      - "{{BaseURL}}/delete/6"
      - "{{BaseURL}}/delete/7"
      - "{{BaseURL}}/delete/8"
      - "{{BaseURL}}/delete/9"
      - "{{BaseURL}}/delete/10"
      - "{{BaseURL}}/delete/11"
      - "{{BaseURL}}/delete/12"
      - "{{BaseURL}}/delete/13"
      - "{{BaseURL}}/delete/14"
      - "{{BaseURL}}/delete/15"
      - "{{BaseURL}}/delete/16"
      - "{{BaseURL}}/delete/17"
      - "{{BaseURL}}/delete/18"
      - "{{BaseURL}}/delete/19"
      - "{{BaseURL}}/delete/20"
      - "{{BaseURL}}/delete/21"
      - "{{BaseURL}}/delete/22"
      - "{{BaseURL}}/delete/23"
      - "{{BaseURL}}/delete/24"
      - "{{BaseURL}}/delete/25"
      - "{{BaseURL}}/delete/26"
      - "{{BaseURL}}/delete/27"
      - "{{BaseURL}}/delete/28"
      - "{{BaseURL}}/delete/29"
      - "{{BaseURL}}/delete/30"
      - "{{BaseURL}}/delete/31"
      - "{{BaseURL}}/delete/32"
      - "{{BaseURL}}/delete/33"
      - "{{BaseURL}}/delete/34"
      - "{{BaseURL}}/delete/35"
      - "{{BaseURL}}/delete/36"
      - "{{BaseURL}}/delete/37"
      - "{{BaseURL}}/delete/38"
      - "{{BaseURL}}/delete/39"
      - "{{BaseURL}}/delete/40"
      - "{{BaseURL}}/delete/41"
  - method: PATCH
    path:
      - "{{BaseURL}}/patch/1"
      - "{{BaseURL}}/patch/2"
      - "{{BaseURL}}/patch/3"
      - "{{BaseURL}}/patch/4"
      - "{{BaseURL}}/patch/5"
      - "{{BaseURL}}/patch/6"
      - "{{BaseURL}}/patch/7"
      - "{{BaseURL}}/patch/8"
      - "{{BaseURL}}/patch/9"
      - "{{BaseURL}}/patch/10"
      - "{{BaseURL}}/patch/11"
      - "{{BaseURL}}/patch/12"
      - "{{BaseURL}}/patch/13"
      - "{{BaseURL}}/patch/14"
      - "{{BaseURL}}/patch/15"
      - "{{BaseURL}}/patch/16"
      - "{{BaseURL}}/patch/17"
      - "{{BaseURL}}/patch/18"

Run nuclei -target https://google.com -t large_template.yaml -v --stats and see Nuclei starting to slow down massively after > 80-90 requests.

Anything else:

This is how it looks in the debugger, if it helps.
Here, we are iterating over previousValues which at this point (after a few minutes of running on above template) has 6 million entries over which we are iterating.

Answered by tarunKoyalwar

Jan 30, 2024

@Nirusu , you guessed it right . manually adding each request in a array is not expected way to write templates , because such style of writing is meant to be used when there is some sort or condition or data sharing between requests . a good way to write this is using payloads for example this would be written as

id: example-fuzz
info:
  name: placeholder
  author: pdteam
  severity: info

http:
  - raw:
      - |
        {{method}} /{{to_lower(method)}}/{{numbers}} HTTP/1.1
        Host: {{Hostname}}
        Origin: {{BaseURL}}
        Connection: close
        User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko)
        Accept: text/html,ap…

View full answer

tarunKoyalwar · 2024-01-30T18:29:49Z

tarunKoyalwar
Jan 30, 2024
Collaborator

@Nirusu , you guessed it right . manually adding each request in a array is not expected way to write templates , because such style of writing is meant to be used when there is some sort or condition or data sharing between requests . a good way to write this is using payloads for example this would be written as

id: example-fuzz
info:
  name: placeholder
  author: pdteam
  severity: info

http:
  - raw:
      - |
        {{method}} /{{to_lower(method)}}/{{numbers}} HTTP/1.1
        Host: {{Hostname}}
        Origin: {{BaseURL}}
        Connection: close
        User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko)
        Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        Accept-Language: en-US,en;q=0.9

    payloads:
      method: 
        - "GET"
        - "POST"
        - "PUT"
        - "DELETE"
        - "PATCH"
      numbers:
        - 1
        - 2
        - 3
        - 4
        - 5
        - 6
        - 7
        - 8
        - 9
        - 10
    attack: clusterbomb
    
    # threads: 50 # concurrency of these request while fuzzing

    matchers:
        - type: status
          status: 
            - 502

also i would recommend to refer existing nuclei templates and use autocomplete or AI template writer feature at https://cloud.projectdiscovery.io/templates

2 replies

Nirusu Jan 30, 2024
Author

Okay, I see. Thanks for the quick reply!

I was naively going for the path list as I am generating templates through a template engine anyway, so initially I did not think it actually makes a difference before trying and seeing it suddenly becomes veeeery slow for some reason.

I will give the proper way through payloads a shot tomorrow.

Nirusu Jan 31, 2024
Author

Actually, I keep running into the same issues if I use payload but define a raw block for each HTTP method.

In the example template I gave the method is part of the API Routes (which makes it easy to define the method as payload), but in reality each method has different API paths I want to test (and the method is not included in the name of the HTTP path) - that's why I define multiple HTTP requests per type in the template above.

So if I define one HTTP raw request per method, and just put all the different API methods for this HTTP method as a payload and use the attack batterringram, I run into the same exact issue as above.

Sorry, it's hard to me to provide an example since the actual template is an API I don't quite want to publish here. But generally is there any way I can to this in one template instead of splitting into multiple templates?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ProjectDiscovery

Nuclei grinds to a halt on HTTP templates with many paths as the internal valuesMap starts to grow beyond expectation #4705

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

ProjectDiscovery

Nuclei grinds to a halt on HTTP templates with many paths as the internal valuesMap starts to grow beyond expectation #4705

Nirusu Jan 30, 2024

Nuclei version:

Current Behavior:

Expected Behavior:

Steps To Reproduce:

Anything else:

Replies: 1 comment · 2 replies

tarunKoyalwar Jan 30, 2024 Collaborator

Nirusu Jan 30, 2024 Author

Nirusu Jan 31, 2024 Author

Nirusu
Jan 30, 2024

Replies: 1 comment 2 replies

tarunKoyalwar
Jan 30, 2024
Collaborator

Nirusu Jan 30, 2024
Author

Nirusu Jan 31, 2024
Author