Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem: Crash and Reboot after 20-30 minutes #1322

Open
alexkip1 opened this issue Sep 13, 2024 · 10 comments
Open

Problem: Crash and Reboot after 20-30 minutes #1322

alexkip1 opened this issue Sep 13, 2024 · 10 comments

Comments

@alexkip1
Copy link

Wiki Search Terms

N/A

Controller Board

Custom ESP32-wroom

Machine Description

3-axis milling machine

Input Circuits

No response

Configuration file

name: "ESP32 Dev Controller V4"
board: "ESP32 Dev Controller V4"

stepping:
  engine: RMT
  idle_ms: 250
  dir_delay_us: 10
  pulse_us: 20
  disable_delay_us: 0

axes:
  shared_stepper_disable_pin: gpio.13:low
  
  x:
    steps_per_mm: 320
    max_rate_mm_per_min: 1500
    acceleration_mm_per_sec2: 25
    max_travel_mm: 1000
    homing:
      cycle: 2
      mpos_mm: 1
      positive_direction: false
    
    motor0:
#      limit_neg_pin: gpio.25:low:pu
      stepstick:
        direction_pin: gpio.14
        step_pin: gpio.12
    motor1:
      null_motor:

  y:
    steps_per_mm: 320
    max_rate_mm_per_min: 1500
    acceleration_mm_per_sec2: 25
    max_travel_mm: 1000
    homing:
      cycle: 2
      mpos_mm: 1
      positive_direction: false

    motor0:
#      limit_neg_pin: gpio.35:low
      stepstick:
        direction_pin: gpio.16
        step_pin: gpio.26
    motor1:
      null_motor:

  z:
    steps_per_mm: 320
    max_rate_mm_per_min: 1500
    acceleration_mm_per_sec2: 25
    max_travel_mm: 1000
    homing:
      cycle: 2
      mpos_mm: 1
      positive_direction: true

    motor0:
#      limit_pos_pin: gpio.32:low:pu
      stepstick:
        direction_pin: gpio.33
        step_pin: gpio.27
    motor1:
      null_motor:

#spi:
#  miso_pin: gpio.19
#  mosi_pin: gpio.23
#  sck_pin: gpio.18

#sdcard:
#  cs_pin: gpio.5
#  card_detect_pin: NO_PIN
#  frequency_hz: 1000000

#coolant:
#  flood_pin: gpio.25
#  mist_pin:  gpio.21

        
#probe:
#  pin: gpio.32:low:pu

PWM:
  pwm_hz: 2000
  output_pin: gpio.2
  enable_pin: gpio.22
  direction_pin: NO_PIN
  disable_with_s0: false
  s0_with_disable: true
  spinup_ms: 0
  spindown_ms: 0
  tool_num: 0
  speed_map: 0=0% 20000=100%

Startup Messages

FluidTerm v1.2.1 (3773931-dirty) using COM3
Exit: Ctrl-C, Ctrl-Q or Ctrl-], Clear screen: CTRL-W
Upload: Ctrl-U, Reset ESP32: Ctrl-R, Send Override: Ctrl-O
No file selected
Resetting MCU
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1184
load:0x40078000,len:13260
load:0x40080400,len:3028
entry 0x400805e4
[MSG:INFO: uart_channel0 created]
[MSG:RST]
[MSG:INFO: FluidNC v3.8.2 https://github.com/bdring/FluidNC]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.7-dirty]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:INFO: Configuration file:config.yaml]
[MSG:INFO: Machine ESP32 Dev Controller V4]
[MSG:INFO: Board ESP32 Dev Controller V4]
[MSG:INFO: Stepping:RMT Pulse:20us Dsbl Delay:0us Dir Delay:10us Idle Delay:250ms]
[MSG:INFO: Axis count 3]
[MSG:INFO: Shared stepper disable gpio.13:low]
[MSG:INFO: Axis X (1.000,1001.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:gpio.12 Dir:gpio.14 Disable:NO_PIN]
[MSG:INFO:   Motor1]
[MSG:INFO: Axis Y (1.000,1001.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:gpio.26 Dir:gpio.16 Disable:NO_PIN]
[MSG:INFO:   Motor1]
[MSG:INFO: Axis Z (-999.000,1.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:gpio.27 Dir:gpio.33 Disable:NO_PIN]
[MSG:INFO:   Motor1]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Connecting to STA SSID:4G-CPE_8612]
[MSG:INFO: Connecting.]
[MSG:INFO: Connecting..]
[MSG:INFO: Connected - IP is 192.168.199.250]
[MSG:INFO: WiFi on]
[MSG:INFO: Start mDNS with hostname:http://fluidnc.local/]
[MSG:INFO: HTTP started on port 80]
[MSG:INFO: Telnet started on port 23]
[MSG:INFO: PWM Spindle Ena:gpio.22 Out:gpio.2 Dir:NO_PIN Freq:2000Hz Period:32767]

Grbl 3.8 [FluidNC v3.8.2 (noradio) '$' for help]
[MSG:INFO: ALARM: Unhomed]
ALARM:14
[MSG:INFO: Caution: Unlocked]

User Interface Software

UGS

What happened?

After 20-30 minutes machine stop working

GCode File

All files longer then 20 minutes of working

Other Information

Guru Meditation Error: Core 1 panic'ed (Cache disabled but cached memory region accessed).

Core 1 register dump:
PC : 0x40081bf7 PS : 0x00060035 A0 : 0x800827fa A1 : 0x3ffbf85c
A2 : 0x00000000 A3 : 0x3ffb3990 A4 : 0x3ffc4878 A5 : 0x00000003
A6 : 0x00000003 A7 : 0x00000002 A8 : 0xbad00bad A9 : 0x3ffbf83c
A10 : 0x3ffb3a94 A11 : 0x003fffff A12 : 0x00060023 A13 : 0x80000000
A14 : 0x00000000 A15 : 0x003fffff SAR : 0x00000001 EXCCAUSE: 0x00000007
EXCVADDR: 0x00000000 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000

Backtrace: 0x40081bf4:0x3ffbf85c |<-CORRUPTED

ELF file SHA256: 0828a1989017003c

Rebooting...
ets Jul 29 2019 12:21:46

rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1184
load:0x40078000,len:13260
load:0x40080400,len:3028
entry 0x400805e4
[MSG:INFO: uart_channel0 created]
[MSG:RST]
[MSG:INFO: FluidNC v3.8.2 https://github.com/bdring/FluidNC]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.7-dirty]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:ERR: Skipping configuration file due to panic]
[MSG:INFO: Using default configuration]
[MSG:INFO: Axes: using defaults]
[MSG:INFO: Machine Default (Test Drive)]
[MSG:INFO: Board None]
[MSG:INFO: Stepping:RMT Pulse:4us Dsbl Delay:0us Dir Delay:0us Idle Delay:255ms]
[MSG:INFO: Axis count 3]
[MSG:INFO: Axis X (-1000.000,0.000)]
[MSG:INFO: Motor0]
[MSG:INFO: Axis Y (-1000.000,0.000)]
[MSG:INFO: Motor0]
[MSG:INFO: Axis Z (-1000.000,0.000)]
[MSG:INFO: Motor0]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Connecting to STA SSID:4G-CPE_8612]
[MSG:INFO: Connecting.]
[MSG:INFO: Connecting..]
[MSG:INFO: Connected - IP is 192.168.199.250]
[MSG:INFO: WiFi on]
[MSG:INFO: Start mDNS with hostname:http://fluidnc.local/]
[MSG:INFO: HTTP started on port 80]
[MSG:INFO: Telnet started on port 23]

Grbl 3.8 [FluidNC v3.8.2 (noradio) '$' for help]
[MSG:ERR: Configuration is invalid. Check boot messages for ERR's.]

Backtrace show always same address. I tried older version of firmware but FluidNC crashed at same address
@bdring
Copy link
Owner

bdring commented Sep 13, 2024

Try turning the wifi off as a test. Run the file from a serial terminal.

@MitchBradley
Copy link
Collaborator

See also #1308. The crash dumps are very similar.

@MitchBradley MitchBradley mentioned this issue Sep 13, 2024
Open
@MitchBradley
Copy link
Collaborator

All files longer then 20 minutes of working

I think what you really mean is "all files longer than 20 minutes of working that I have tried". It would be nice to have an example of one such file so I don't have to guess.

@MitchBradley
Copy link
Collaborator

As an example of what I mean by not knowing, I though of trying a single line like "G1 X250 F10" which would take 25 minutes. That might trigger the problem, but it seems unlikely. Nevertheless I am trying it.

@MitchBradley
Copy link
Collaborator

The one-line command that takes 25 minutes did not crash. Please supply an example GCode program that does crash.

@Belott
Copy link

Belott commented Sep 16, 2024

I have similar issues here with a Makerbase MKS DLC32 board. It crashes randomly. I would say, it started to happen since 3.8.0, maybe 3.8.1. Never had this issue with 3.7.x.

@masterxq
Copy link

Can confirm, sadly no logs as I did downgrade and not saved them.

Playing gcode files -> random crash after 0-4 Hours (This could be caused by network environment, scans or unintended access)

Run gcode files and reload webui -> crash immediate (Lost SD-Card or full crash)

Have a simple move without running gcode and reload webui -> crash immediate (Lost SD-Card or full crash)

Can confirm this for 3.8.0 and 3.8.2

Not happens with 3.7.x as long as I can say, random crashes are hard to reproduce ^^

@alexkip1
Copy link
Author

I made a lot of experiments and now can crash ESP with same "Backtrace: 0x40081bf4:0x3ffbf85c |<-CORRUPTED".
For this I open FluidNC web ui then go to "FluidNC Settings" tab, then start any g code file from UGS and press "Manage local files".
ESP always crash at 0x40081bf4:0x3ffbf85c.

Why its randomly? Possible FireFox try update some cashed pages even if tab closed.

@MitchBradley
Copy link
Collaborator

With "Manage local files", the browser asks FluidNC for a list of FLASH files. That causes accesses to the FLASH filesystem, which interfere with the interrupt service routines (ISRs) that occur during GCode execution. We have tried and tried to ensure that all the code that can run from ISRs is "FLASH safe", but it is difficult to maintain that because the system, compiler, and the various third party libraries are so complex. We often think that we have found all the problem areas, but then a seemingly-unrelated change will cause a bit of unsafe code to creep back in to an ISR routine. I need to do another deep search. I have been putting it off because it is very tedious, boring work and there is other fun work that I have been doing instead.

@MitchBradley MitchBradley reopened this Sep 25, 2024
@MitchBradley
Copy link
Collaborator

This problem is still outstanding

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bdring @masterxq @MitchBradley @Belott @alexkip1