Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamp terminal output (progress bar, callback and stdout/stderr) #1132

Merged
merged 11 commits into from
Oct 23, 2022

Conversation

stefanrueger
Copy link
Collaborator

Supposed to fix Issue #1130

@mcuee mcuee added the enhancement New feature or request label Oct 18, 2022
@stefanrueger
Copy link
Collaborator Author

Now parms prints to stdout and is unimpressed by -qq... command line options

$ echo parms | avrdude -p m328p -qqqqqqc arduino -t -P /dev/ttyUSB0 
avrdude> parms
Vtarget         : 0.7 V
Varef           : 0.7 V
Oscillator      : 450.000 Hz
SCK period      : 7.6 us

$ echo parms | avrdude -p m328p -qqqqqqc arduino -t -P /dev/ttyUSB0  | wc
      6      20     127

@mcuee
Copy link
Collaborator

mcuee commented Oct 18, 2022

Indeed this fixed the issue mentioned.

@mcuee
Copy link
Collaborator

mcuee commented Oct 18, 2022

@stefanrueger

Is it true that only "parms" command suffers this issue? I was expecting changes across programmers based on your comments here.

@stefanrueger
Copy link
Collaborator Author

stefanrueger commented Oct 18, 2022

Is it true that only "parms" command suffers this issue?

Yes, it is parms and erase, which confirms erasing the chip. parms is implemented on a per-programmer basis, so these all had to be changed, and the API for pgm->print_parms() had to take a FILE * parameter.

@stefanrueger
Copy link
Collaborator Author

Using this PR to put code in to periodically call a programmer function so bootloader WDTs can be reset.

Note that the bootloader programmers still need to provide their "WDT reset" code, but the terminal side is finished.

Made use of @mariusgreuel's Windows port for readline callback functions (see Issue #1028 and his repo https://github.com/mariusgreuel/termreadline). This works under Linux and should work under Windows and MacOS, too, as long as you HAVE_LIBREADLINE. Would be great if you could test in all other OS whether terminal still works with this PR, both interactively, and with input redirection from file and pipes.

Right now there is a readline incompatibility for MacOS:

/Users/runner/work/avrdude/avrdude/src/term.c:1306:58: error: expected expression
  rl_callback_handler_install("avrdude> ", (rl_vcpfunc_t*) &term_gotline);

Wrong library?

@dl8dtl
Copy link
Contributor

dl8dtl commented Oct 18, 2022

As long as it's only related to terminal mode operation, I think it's fine to use stdout.
The remainder uses stderr simply since stdout is a possible stream target for data read from the device.

@stefanrueger
Copy link
Collaborator Author

As long as it's only related to terminal mode operation, I think it's fine to use stdout. The remainder uses stderr simply since stdout is a possible stream target for data read from the device.

That's exactly how this PR behaves. The terminal operations are printed on stdout (and can be used in a pipeline); errors, warnings etc are on stderr as always.

@stefanrueger
Copy link
Collaborator Author

stefanrueger commented Oct 18, 2022

OK, now that the readline() calbacks compile for MacOS, Windows and Un*xes, this PR needs just checking whether the terminal still works in either OS. Please try with your favourite programmer and part:

$ avrdude -c ...  -p ... -t
$ echo quit | avrdude -c ...  -p ... -t

You can recognise it's the readline library if arrow up/down gets your back and forth in the command line history. Known bug (reported by @mariusgreuel) that under Windows redirected input needs to end with quit as EOF is not recognised there.

If all other things work, I'd merge at this point (and get windows EOF reading from pipe/file fixed another time).

The big benefit of this PR is that the programmer can do things concurrently whilst the user types (requires the readline() callbacks). This allows us to write code to keep bootloaders from time out.

@mcuee
Copy link
Collaborator

mcuee commented Oct 19, 2022

Strange, even though github action has no issues building under MSYS2 mingw64, I have issues with missing reference to tput. But the relevant packages have been installed (readline and ncurses). Anyway, this may be due to my outdated MSYS2 installation in this machine. I will try another Windows machine.

Edit to add: even after I fully updated the MSYS2 installation, the issue remains. Strange.

$ ./build.sh
-- Building for: Ninja
-- The C compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/mingw64/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Git: C:/msys64/usr/bin/git.exe (found version "2.35.1")
-- Found FLEX: C:/msys64/usr/bin/flex.exe (found version "2.6.4")
-- Found BISON: C:/msys64/usr/bin/bison.exe (found version "3.8.2")
-- Looking for libelf.h
-- Looking for libelf.h - found
-- Looking for libelf/libelf.h
-- Looking for libelf/libelf.h - found
-- Looking for usb.h
-- Looking for usb.h - not found
-- Looking for lusb0_usb.h
-- Looking for lusb0_usb.h - found
-- Looking for libusb.h
-- Looking for libusb.h - not found
-- Looking for libusb-1.0/libusb.h
-- Looking for libusb-1.0/libusb.h - found
-- Looking for hidapi/hidapi.h
-- Looking for hidapi/hidapi.h - found
-- Looking for ftdi_tcioflush
-- Looking for ftdi_tcioflush - found
-- Configuration summary:
-- ----------------------
-- DO HAVE    libelf
-- DO HAVE    libusb
-- DO HAVE    libusb_1_0
-- DO HAVE    libhidapi
-- DON'T HAVE libftdi
-- DO HAVE    libftdi1
-- DISABLED   doc
-- DISABLED   parport
-- DISABLED   linuxgpio
-- DISABLED   linuxspi
-- ----------------------
-- Configuring done
-- Generating done
-- Build files have been written to: C:/work/avr/avrdude_test/avrdude_sr/build_mingw64_nt-10.0-19044
[66/66] Linking C executable src\avrdude.exe
FAILED: src/avrdude.exe
cmd.exe /C "cd . && C:\msys64\mingw64\bin\cc.exe -O2 -g -DNDEBUG  src/CMakeFiles/avrdude.dir/main.c.obj src/CMakeFiles/avrdude.dir/term.c.obj src/CMakeFiles/avrdude.dir/avrintel.c.obj src/CMakeFiles/avrdude.dir/developer_opts.c.obj src/CMakeFiles/avrdude.dir/whereami.c.obj src/CMakeFiles/avrdude.dir/windows.rc.obj -o src\avrdude.exe -Wl,--out-implib,src\libavrdude.dll.a -Wl,--major-image-version,0,--minor-image-version,0  src/libavrdude.a  -lm  C:/msys64/mingw64/lib/libelf.a  C:/msys64/mingw64/lib/libusb.a  C:/msys64/mingw64/lib/libusb-1.0.a  C:/msys64/mingw64/lib/libhid.a  C:/msys64/mingw64/lib/libhidapi.a  C:/msys64/mingw64/lib/libftdi1.a  C:/msys64/mingw64/lib/libreadline.a  -lsetupapi  -lws2_32  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):display.c:(.text+0xe80): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):display.c:(.text+0xee8): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):display.c:(.text+0x3633): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):display.c:(.text+0x37d2): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):display.c:(.text+0x384e): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):display.c:(.text+0x398f): more undefined references to `tputs' follow
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0x2f4): undefined reference to `tgetnum'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0x330): undefined reference to `tgetnum'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0x7fe): undefined reference to `tgetent'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0x847): undefined reference to `tgetstr'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0x8a6): undefined reference to `tgetflag'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0x8f8): undefined reference to `tgetflag'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0xa87): undefined reference to `tgetflag'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0xcdb): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0xdb9): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0xec7): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0xf1d): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0xd54): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):terminal.c:(.text+0xe43): more undefined references to `tputs' follow
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Build failed.

@mcuee
Copy link
Collaborator

mcuee commented Oct 19, 2022

You can recognise it's the readline library if arrow up/down gets your back and forth in the command line history. Known bug (reported by @mariusgreuel) that under Windows redirected input needs to end with quit as EOF is not recognised there.

If all other things work, I'd merge at this point (and get windows EOF reading from pipe/file fixed another time).

I am using Windows terminal and actually I do not see much differences between this one and git_main. I can already use arrow up/down with git main.

As for EOF, indeed CTRL-D does not work under git_main or this PR without libreadline. CTRL-D works with libreadline under Windows (tested with both Windows Terminal and Command Prompt).

But under Windows, even without libreadline, we can use CTRL-D + RETURN as EOF.
Ref: https://www.devdungeon.com/content/windows-eof-ctrl-d-equivalent

And there is a anomaly under Windows with F7. That is not fixed by this PR yet.

Reference: git main.

PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude_git -c wiring -P COM5 -p m2560 -qq -t
avrdude> dump eeprom 0 0x20
>>> dump eeprom 0 0x20
0000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
0010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

avrdude> write eeprom 0 0xaa 0x55 0xaa 0x55
>>> write eeprom 0 0xaa 0x55 0xaa 0x55
avrdude> ^Z
PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude_git -c wiring -P COM5 -p m2560 -qq -t
avrdude> dump eeprom 0 0x20
>>> dump eeprom 0 0x20
0000  aa 55 aa 55 ff ff ff ff  ff ff ff ff ff ff ff ff  |.U.U............|
0010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

avrdude>^Z (pressed CTRL-Z + Return, it displays ^Z)
PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude_git -c wiring -P COM5 -p m2560 -qq -t
avrdude> write eeprom 0 0x55 0xaa 0x55 0xaa
>>> write eeprom 0 0x55 0xaa 0x55 0xaa
avrdude> ^Z
PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude_git -c wiring -P COM5 -p m2560 -qq -t
avrdude> dump eeprom 0 0x20
>>> dump eeprom 0 0x20
0000  55 aa 55 aa ff ff ff ff  ff ff ff ff ff ff ff ff  |U.U.............|
0010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

avrdude> ^Z (pressed CTRL-Z + Return, it displays ^Z)
PS C:\work\avr\avrdude_test\avrdude_bin>

PR 1132 with libreadline (dynamic linking so that it will work).

PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude_pr1132v1 -c wiring -P COM5 -p m2560 -qq -t
avrdude> dump eeprom 0 0x20
>>> dump eeprom 0 0x20
0000  55 aa 55 aa ff ff ff ff  ff ff ff ff ff ff ff ff  |U.U.............|
0010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

avrdude> write eeprom 0 0xaa 0x55 0xaa 0x55
>>> write eeprom 0 0xaa 0x55 0xaa 0x55
avrdude> (hit CTRL-D, no display)
avrdude>
PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude_pr1132v1 -c wiring -P COM5 -p m2560 -qq -t
avrdude> dump eeprom 0 0x20
>>> dump eeprom 0 0x20
0000  aa 55 aa 55 ff ff ff ff  ff ff ff ff ff ff ff ff  |.U.U............|
0010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

avrdude>(hit CTRL-D, no display)
avrdude>

@mcuee
Copy link
Collaborator

mcuee commented Oct 19, 2022

Strange, even though github action has no issues building under MSYS2 mingw64, I have issues with missing reference to tput. But the relevant packages have been installed (readline and ncurses). Anyway, this may be due to my outdated MSYS2 installation in this machine. I will try another Windows machine.

Edit to add: even after I fully updated the MSYS2 installation, the issue remains. Strange.

I am thinking that libncurses.a is missing from the linking command, but it is still failing after I add that.

Edit: this is not correct.

MINGW64 /c/work/avr/avrdude_test/avrdude_sr/build_mingw64_nt-10.0-19044
$ cmd.exe /C "cd . && C:\msys64\mingw64\bin\cc.exe -O2 -g -DNDEBUG  src/CMakeFiles/avrdude.dir/main.c.obj src/CMakeFiles/avrdude.dir/term.c.obj src/CMakeFiles/avrdude.dir/avrintel.c.obj src/CMakeFiles/avrdude.dir/developer_opts.c.obj src/CMakeFiles/avrdude.dir/whereami.c.obj src/CMakeFiles/avrdude.dir/windows.rc.obj -o src\avrdude.exe -Wl,--out-implib,src\libavrdude.dll.a -Wl,--major-image-version,0,--minor-image-version,0  src/libavrdude.a  -lm  C:/msys64/mingw64/lib/libelf.a  C:/msys64/mingw64/lib/libusb.a  C:/msys64/mingw64/lib/libusb-1.0.a  C:/msys64/mingw64/lib/libhid.a  C:/msys64/mingw64/lib/libhidapi.a  C:/msys64/mingw64/lib/libftdi1.a  C:/msys64/mingw64/lib/libreadline.a C:/msys64/mingw64/lib/libncurses.a -lsetupapi  -lws2_32  -lkernel32 -luser32 -lgdi
32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libncurses.a(lib_termcap.o):(.bss+0x8): multiple definition of `UP'; C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.bss+0x98): first defined here
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libncurses.a(lib_termcap.o):(.bss+0x0): multiple definition of `BC'; C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.bss+0xa0): first defined here
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libncurses.a(lib_tputs.o):(.bss+0x6): multiple definition of `PC'; C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.bss+0xa8): first defined here
collect2.exe: error: ld returned 1 exit status

@mcuee
Copy link
Collaborator

mcuee commented Oct 19, 2022

I figured out what was the problem. It is not libncurses.a, rather I need to add libtermcap.a. However I can only use -ltermcap dynamic linking. I have not been able to figured out a way to statically link to termcap. Dynamic linking to readline is also working.

It is always very tricky to use static linking for mingw gcc but that is the default for avrdude Windows.

Ref:

$ pkg-config --libs readline
-LC:/msys64/mingw64/lib -lreadline

$ pkg-config --static --libs readline
-LC:/msys64/mingw64/lib -lreadline -LC:/msys64/mingw64/lib -ltermcap

$ pkg-config --static --libs termcap
-LC:/msys64/mingw64/lib -ltermcap

$ cmd.exe /C "cd . && C:\msys64\mingw64\bin\cc.exe -O2 -g -DNDEBUG  src/CMakeFiles/avrdude.dir/main.c.obj src/CMakeFiles/avrdude.dir/term.c.obj src/CMakeFiles/avrdude.dir/avrintel.c.obj src/CMakeFiles/avrdude.dir/developer_opts.c.obj src/CMakeFiles/avrdude.dir/whereami.c.obj src/CMakeFiles/avrdude.dir/windows.rc.obj -o src\avrdude.exe -Wl,--out-implib,src\libavrdude.dll.a -Wl,--major-image-version,0,--minor-image-version,0  src/libavrdude.a  -lm  C:/msys64/mingw64/lib/libelf.a  C:/msys64/mingw64/lib/libusb.a  C:/msys64/mingw64/lib/libusb-1.0.a  C:/msys64/mingw64/lib/libhid.a  C:/msys64/mingw64/lib/libhidapi.a  C:/msys64/mingw64/lib/libftdi1.a C:/msys64/mingw64/lib/libreadline.a -lsetupapi  -lws2_32  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -lol
eaut32 -luuid -lcomdlg32 -ladvapi32 -ltermcap && cd ."

$ ls ./src/avrdude.*
./src/avrdude.conf  ./src/avrdude.conf.in  ./src/avrdude.exe  ./src/avrdude.spec

Static linking will fail.

$ cmd.exe /C "cd . && C:\msys64\mingw64\bin\cc.exe -O2 -g -DNDEBUG  src/CMakeFiles/avrdude.dir/main.c.obj src/CMakeFiles/avrdude.dir/term.c.obj src/CMakeFiles/avrdude.dir/avrintel.c.obj src/CMakeFiles/avrdude.dir/developer_opts.c.obj src/CMakeFiles/avrdude.dir/whereami.c.obj src/CMakeFiles/avrdude.dir/windows.rc.obj -o src\avrdude.exe -Wl,--out-implib,src\libavrdude.dll.a -Wl,--major-image-version,0,--minor-image-version,0  src/libavrdude.a  -lm  C:/msys64/mingw64/lib/libelf.a  C:/msys64/mingw64/lib/libusb.a  C:/msys64/mingw64/lib/libusb-1.0.a  C:/msys64/mingw64/lib/libhid.a  C:/msys64/mingw64/lib/libhidapi.a  C:/msys64/mingw64/lib/libftdi1.a C:/msys64/mingw64/lib/libreadline.a C:/msys64/mingw64/lib/libtermcap.a -lsetupapi  -lws2_32  -lkernel32 -luser32 -lgdi3
2 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libtermcap.a(termcap.o):(.bss+0x8): multiple definition of `PC'; C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.bss+0xa8): first defined here
collect2.exe: error: ld returned 1 exit status

@mcuee
Copy link
Collaborator

mcuee commented Oct 19, 2022

@stefanrueger

Looking at the CMake artifacts, github action does not use libreadline at all. My MSYS2 mingw64 system has libreadline installed and that is the reason for failing.

I have no issues to build with VS2019 as it does not use libreadline either.

From the CMake artifacts for mingw64.
https://github.com/avrdudes/avrdude/suites/8843305690/artifacts/403172452

src/avrdude.exe: src/CMakeFiles/avrdude.dir/main.c.obj
src/avrdude.exe: src/CMakeFiles/avrdude.dir/term.c.obj
src/avrdude.exe: src/CMakeFiles/avrdude.dir/avrintel.c.obj
src/avrdude.exe: src/CMakeFiles/avrdude.dir/developer_opts.c.obj
src/avrdude.exe: src/CMakeFiles/avrdude.dir/whereami.c.obj
src/avrdude.exe: src/CMakeFiles/avrdude.dir/windows.rc.obj
src/avrdude.exe: src/CMakeFiles/avrdude.dir/build.make
src/avrdude.exe: src/libavrdude.a
src/avrdude.exe: D:/a/_temp/msys64/mingw64/lib/libelf.a
src/avrdude.exe: D:/a/_temp/msys64/mingw64/lib/libusb.a
src/avrdude.exe: D:/a/_temp/msys64/mingw64/lib/libusb-1.0.a
src/avrdude.exe: D:/a/_temp/msys64/mingw64/lib/libhid.a
src/avrdude.exe: D:/a/_temp/msys64/mingw64/lib/libhidapi.a
src/avrdude.exe: D:/a/_temp/msys64/mingw64/lib/libftdi1.a
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --green --bold --progress-dir=/D/a/avrdude/avrdude/build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_7) "Linking C executable avrdude.exe"
	cd /D/a/avrdude/avrdude/build/src && /D/a/_temp/msys64/mingw64/bin/cmake.exe -E rm -f CMakeFiles/avrdude.dir/objects.a
	cd /D/a/avrdude/avrdude/build/src && /D/a/_temp/msys64/mingw64/bin/ar.exe qc CMakeFiles/avrdude.dir/objects.a $(avrdude_OBJECTS) $(avrdude_EXTERNAL_OBJECTS)
	cd /D/a/avrdude/avrdude/build/src && /D/a/_temp/msys64/mingw64/bin/cc.exe -O2 -g -DNDEBUG -Wl,--whole-archive CMakeFiles/avrdude.dir/objects.a -Wl,--no-whole-archive -o avrdude.exe -Wl,--out-implib,libavrdude.dll.a -Wl,--major-image-version,0,--minor-image-version,0  libavrdude.a -lm /D/a/_temp/msys64/mingw64/lib/libelf.a /D/a/_temp/msys64/mingw64/lib/libusb.a /D/a/_temp/msys64/mingw64/lib/libusb-1.0.a /D/a/_temp/msys64/mingw64/lib/libhid.a /D/a/_temp/msys64/mingw64/lib/libhidapi.a /D/a/_temp/msys64/mingw64/lib/libftdi1.a -lsetupapi -lws2_32 -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 

# Rule to build all files generated by this target.
src/CMakeFiles/avrdude.dir/build: src/avrdude.exe
.PHONY : src/CMakeFiles/avrdude.dir/build


@mcuee
Copy link
Collaborator

mcuee commented Oct 19, 2022

@stefanrueger

I tried on another Windows machine with libreadline installed for MSYS2 mingw64, and it has the same issue. I can not remove it right now as it is needed by many other packages. That is the issue with pacman (similar thing for Arch Linux) where the library and development package are within the same package, unlike Debian/Ubuntu or Fedora.

$ pacman -R mingw-w64-x86_64-readline
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-arm-none-eabi-gdb
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-avr-gdb
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-gdb
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-sqlite3

$ pacman -R mingw-w64-x86_64-sqlite3
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: removing mingw-w64-x86_64-sqlite3 breaks dependency 'mingw-w64-x86_64-sqlite3' required by mingw-w64-x86_64-python

$ pacman -R mingw-w64-x86_64-readline
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-arm-none-eabi-gdb
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-avr-gdb
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-gdb
:: removing mingw-w64-x86_64-readline breaks dependency 'mingw-w64-x86_64-readline' required by mingw-w64-x86_64-sqlite3

$ pacman -R mingw-w64-x86_64-sqlite3
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: removing mingw-w64-x86_64-sqlite3 breaks dependency 'mingw-w64-x86_64-sqlite3' required by mingw-w64-x86_64-python

$ pacman -R mingw-w64-x86_64-python
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-arm-none-eabi-gdb
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-asciidoc
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-gdb
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-glib2
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-meson
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-python-distlib
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-python-numpy
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-python-pip
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-python-setuptools
:: removing mingw-w64-x86_64-python breaks dependency 'mingw-w64-x86_64-python' required by mingw-w64-x86_64-python-wheel

@mcuee
Copy link
Collaborator

mcuee commented Oct 19, 2022

@stefanrueger

All in all it is not a real issue for this PR. It is just that readline is still a problem for Windows (and MSYS2 mingw64 libreadline or libtermcap dynamic linking is broken). We can probably fix that with the discussion here.

I can still easily work around this issue by using dynamic linking for libreadline.

@stefanrueger
Copy link
Collaborator Author

@mcuee Yes, the gitlab runners work for MacOS; looking at the build report evidence they found libreadline, use it and it compiles.

Would be good if standard MacOS installation would work as well. Looking at your MacOS installation problems, it seems to be the same as msys2/MINGW-packages#8105 Maybe there is a solution hidden in this discussion?

@mariusgreuel has made readline work in Windows, and I used his code from https://github.com/mariusgreuel/termreadline
Using the code may not be sufficient. This may require some changes in the make/cmake environment. I am hoping that @mariusgreuel can help.

Windows terminal [...] can already use arrow up/down with git main

This may be down to fgets() in Windows having a sort of readline() equivalent built in, albeit one that does not allow callbacks (ie, concurrent working of caller and readline()) needed here. So, the presence of a cl history does not indicate whether the readline() library is used.

@mcuee
Copy link
Collaborator

mcuee commented Oct 20, 2022

@stefanrueger
However under macOS, default libedit version does not work well. I need to hit RETURN everywhere.

CTRL-D does not work either with libedit.

github action binary (x86_64) version has the same issue as it also uses libedit.

BTW, no issues with git main, history using arrow keys works fine. CTRL-D works fine as well.

mcuee@mcuees-Mac-mini avrdude_bin % ./avrdude_pr1132_libedit -c wiring 
-P /dev/tty.usbserial-221330 -p m2560 -qq -t
avrdude> help
>>> help 
Valid commands:
  dump    : dump <memory> [<addr> <len> | <addr> ... | <addr> | ...]
  read    : alias for dump
  write   : write <memory> <addr> <data>[,] {<data>[,]}
          : write <memory> <addr> <len> <data>[,] {<data>[,]} ...
  flush   : synchronise flash & EEPROM writes with the device
  abort   : abort flash & EEPROM writes (reset the r/w cache)
  erase   : perform a chip erase
  pgerase : pgerase <memory> <addr>
  sig     : display device signature bytes
  part    : display the current part information
  send    : send a raw command: send <b1> <b2> <b3> <b4>
  parms   : display adjustable parameters
  vtarg   : set <V[target]>
  varef   : set <V[aref]>
  fosc    : set <oscillator frequency>
  sck     : set <SCK period>
  verbose : change verbosity
  quell   : set quell level for progress bars
  help    : show help message
  ?       : same as help
  quit    : quit after writing out cache for flash & EEPROM

Note that not all programmer derivatives support all commands. Flash and
EEPROM type memories are normally read and written using a cache via paged
read and write access; the cache is synchronised on quit or flush commands.
The part command displays valid memory types for use with dump and write.
(stop here, hit to manually hit RETURN).

avrdude> 
avrdude> 
dump eeprom 0 0x10
avrdude> dump eeprom 0 0x10
>>> dump eeprom 0 0x10 
0000  aa 55 aa 55 ff ff ff ff  ff ff ff ff ff ff ff ff  |.U.U............|
(stop here, I need to hit return).

avrdude> ^D
avrdude> 

avrdude> 
quit
avrdude> quit
>>> quit

@mcuee
Copy link
Collaborator

mcuee commented Oct 20, 2022

@mariusgreuel
Just wondering if you can update CMake file to require libreadline to test this PR under macOS.

If I force link readline under macOS, it can be found using pkg-config.

mcuee@mcuees-Mac-mini avrdude_sr % brew unlink readline
Unlinking /opt/homebrew/Cellar/readline/8.2.1... 18 symlinks removed.

mcuee@mcuees-Mac-mini avrdude_sr % pkg-config --libs readline
Package readline was not found in the pkg-config search path.
Perhaps you should add the directory containing `readline.pc'
to the PKG_CONFIG_PATH environment variable
No package 'readline' found

mcuee@mcuees-Mac-mini avrdude_sr % brew link readline --force
Linking /opt/homebrew/Cellar/readline/8.2.1... 18 symlinks created.

mcuee@mcuees-Mac-mini avrdude_sr % pkg-config --cflags readline
-I/opt/homebrew/Cellar/readline/8.2.1/include

mcuee@mcuees-Mac-mini avrdude_sr % pkg-config --libs readline
-L/opt/homebrew/Cellar/readline/8.2.1/lib -lreadline

@mcuee
Copy link
Collaborator

mcuee commented Oct 20, 2022

@stefanrueger
Yes the MSYS2 build problem is exactly the same as msys2/MINGW-packages#8105. I will try to rebuild libtermcap to fix the issue.

Edit to add:
Indeed the suggestion to rename PC to TERMCAP_PC works fine.
msys2/MINGW-packages#8105 (comment)

We still need to fix avrdude side to add the termcap library. Right now I need to manually add it after the failure. That may require the help from @mariusgreuel again on the CMake side (find out readline and maybe termcap).

Not so sure if this helps or not.
https://medium.com/codex/msys2-build-processes-driving-you-crazy-83d4a23bf448

$ ./build.sh
-- Configuration summary:
-- ----------------------
-- DO HAVE    libelf
-- DO HAVE    libusb
-- DO HAVE    libusb_1_0
-- DO HAVE    libhidapi
-- DON'T HAVE libftdi
-- DO HAVE    libftdi1
-- DISABLED   doc
-- DISABLED   parport
-- DISABLED   linuxgpio
-- DISABLED   linuxspi
-- ----------------------
-- Configuring done
-- Generating done
-- Build files have been written to: C:/work/avr/avrdude_test/avrdude_sr/build_mingw64_nt-10.0-22000
[56/57] Building C object src/CMakeFiles/avrdude.dir/term.c.obj
In function 'tokenize',
    inlined from 'process_line' at C:/work/avr/avrdude_test/avrdude_sr/src/term.c:1212:10,
    inlined from 'term_gotline' at C:/work/avr/avrdude_test/avrdude_sr/src/term.c:1289:10:
C:/work/avr/avrdude_test/avrdude_sr/src/term.c:1118:17: warning: pointer 'buf' may be used after 'realloc' [-Wuse-after-free]
 1118 |       ptrdiff_t k = buf - bufp;
      |                 ^
C:/work/avr/avrdude_test/avrdude_sr/src/term.c:1102:18: note: call to 'realloc' here
 1102 |       buf_tmp  = realloc(buf, bufsize);
      |                  ^~~~~~~~~~~~~~~~~~~~~
[57/57] Linking C executable src\avrdude.exe
FAILED: src/avrdude.exe
cmd.exe /C "cd . && C:\msys64\mingw64\bin\cc.exe -O2 -g -DNDEBUG  src/CMakeFiles/avrdude.dir/main.c.obj src/CMakeFiles/avrdude.dir/term.c.obj src/CMakeFiles/avrdude.dir/avrintel.c.obj src/CMakeFiles/avrdude.dir/developer_opts.c.obj src/CMakeFiles/avrdude.dir/whereami.c.obj src/CMakeFiles/avrdude.dir/windows.rc.obj -o src\avrdude.exe -Wl,--out-implib,src\libavrdude.dll.a -Wl,--major-image-version,0,--minor-image-version,0  src/libavrdude.a  -lm  C:/msys64/mingw64/lib/libelf.a  C:/msys64/mingw64/lib/libusb.a  C:/msys64/mingw64/lib/libusb-1.0.a  C:/msys64/mingw64/lib/libhid.a  C:/msys64/mingw64/lib/libhidapi.a  C:/msys64/mingw64/lib/libftdi1.a  C:/msys64/mingw64/lib/libreadline.a  -lsetupapi  -lws2_32  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):(.text+0xea0): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):(.text+0xf08): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):(.text+0x3683): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):(.text+0x3812): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):(.text+0x388e): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(display.o):(.text+0x39cf): more undefined references to `tputs' follow
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0x29c): undefined reference to `tgetnum'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0x2d8): undefined reference to `tgetnum'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0x7f9): undefined reference to `tgetent'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0x837): undefined reference to `tgetstr'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0x896): undefined reference to `tgetflag'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0x8e6): undefined reference to `tgetflag'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0xa80): undefined reference to `tgetflag'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0xceb): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0xdb9): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0xea7): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0xefd): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0xd64): undefined reference to `tputs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/lib/libreadline.a(terminal.o):(.text+0xe23): more undefined references to `tputs' follow
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Build failed.

$ cd build_mingw64_nt-10.0-22000/

$ cmd.exe /C "cd . && C:\msys64\mingw64\bin\cc.exe -O2 -g -DNDEBUG  src/CMakeFiles/avrdude.dir/main.c.obj src/CMakeFiles/avrdude.dir/term.c.obj src/CMakeFiles/avrdude.dir/avrintel.c.obj src/CMakeFiles/avrdude.dir/developer_opts.c.obj src/CMakeFiles/avrdude.dir/whereami.c.obj src/CMakeFiles/avrdude.dir/windows.rc.obj -o src\avrdude.exe -Wl,--out-implib,src\libavrdude.dll.a -Wl,--major-image-version,0,--minor-image-version,0  src/libavrdude.a  -lm  C:/msys64/mingw64/lib/libelf.a  C:/msys64/mingw64/lib/libusb.a  C:/msys64/mingw64/lib/libusb-1.0.a  C:/msys64/mingw64/lib/libhid.a  C:/msys64/mingw64/lib/libhidapi.a  C:/msys64/mingw64/lib/libftdi1.a  C:/msys64/mingw64/lib/libreadline.a  C:/msys64/mingw64/lib/libtermcap.a -lsetupapi  -lws2_32  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."

This will produce the static linking version of avrude.exe. I can also use -lreadline to build dynamic link version of avrdude.

$ objdump.exe -t avrdude_pr1132v3static.exe | grep readline
[2791](sec  1)(fl 0x00)(ty   20)(scl   3) (nx 0) 0x000000000004b510 buspirate_readline_noexit.constprop.0
[5626](sec  3)(fl 0x00)(ty    0)(scl   3) (nx 1) 0x0000000000043cd0 .rdata$.refptr.rl_readline_state
[5942](sec  1)(fl 0x00)(ty   20)(scl   2) (nx 0) 0x00000000000a5800 readline_internal_setup
[5945](sec  1)(fl 0x00)(ty   20)(scl   2) (nx 0) 0x00000000000a5990 readline_internal_teardown
[5959](sec  1)(fl 0x00)(ty   20)(scl   2) (nx 0) 0x00000000000a6410 readline_internal_char
[5987](sec  1)(fl 0x00)(ty   20)(scl   2) (nx 0) 0x00000000000a6d60 readline
[6377](sec  6)(fl 0x00)(ty    0)(scl   3) (nx 0) 0x000000000000c740 current_readline_init_lineno
[6378](sec  6)(fl 0x00)(ty    0)(scl   3) (nx 0) 0x000000000000c748 current_readline_init_file
[6458](sec  6)(fl 0x00)(ty    0)(scl   3) (nx 0) 0x000000000000c744 current_readline_init_include_level
[6459](sec  6)(fl 0x00)(ty    0)(scl   3) (nx 0) 0x000000000000c750 last_readline_init_file
[11283](sec  6)(fl 0x00)(ty    0)(scl   2) (nx 0) 0x000000000000c418 rl_readline_state
[11448](sec  2)(fl 0x00)(ty    0)(scl   2) (nx 0) 0x0000000000000de8 rl_gnu_readline_p
[11474](sec  2)(fl 0x00)(ty    0)(scl   2) (nx 0) 0x0000000000005fd0 rl_readline_name
[11592](sec  3)(fl 0x00)(ty    0)(scl   2) (nx 0) 0x0000000000043cd0 .refptr.rl_readline_state
[11948](sec  2)(fl 0x00)(ty    0)(scl   2) (nx 0) 0x0000000000000dec rl_readline_version

$ objdump.exe -t ./termcap/avrdude_pr1132v3dynamic.exe | grep readline
[2791](sec  1)(fl 0x00)(ty   20)(scl   3) (nx 0) 0x000000000004b510 buspirate_readline_noexit.constprop.0
[9465](sec  8)(fl 0x00)(ty    0)(scl   2) (nx 0) 0x0000000000000028 _head_libreadline8_dll
[9894](sec  8)(fl 0x00)(ty    0)(scl   2) (nx 0) 0x0000000000001b38 libreadline8_dll_iname

@mcuee
Copy link
Collaborator

mcuee commented Oct 20, 2022

The result libreadline based avrdude works fine. CTRL-D works as well.

PS C:\work\avr\avrdude_test\avrdude_bin> .\avrdude_pr1132v3static -c wiring -P COM5 -p m2560 -t

avrdude_pr1132v3static: AVR device initialized and ready to accept instructions
avrdude_pr1132v3static: device signature = 0x1e9801 (probably m2560)
avrdude> parms
>>> parms
Vtarget         : 0.0 V
SCK period      : 0.1 us

avrdude> (CTRL-D)
avrdude>
avrdude_pr1132v3static done.  Thank you.

It also works much better for my official Arduino Nano Every with buggy jtag2updi firmware in terms of progress bar, compared to git main. You can see the commits are working as expected.

  • Add space after the time in the progress bar, so any error message keeps visually separated from progress bar
  • (Most radical): Freeze (and disable) the progress bar at first error

Take note due to the padding size, it will still print out the progress bar, but then it will diable the progress bar because of the error.

In order to show the real results, I will not modify any outputs -- so you need to scroll to the right to see the command.

PS C:\work\avr\avrdude_test\avrdude_bin> mode COM12 baud=12 dtr=on > nul && mode COM12 baud=12 dtr=off > nul && echo "Hello World" | .\avrdude_git -c jtag2updi -P COM12 -p m4809 -U eeprom:w:-:r

avrdude_git: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.16s

avrdude_git: device signature = 0x1e9651 (probably m4809)
avrdude_git: reading input file <stdin> for eeprom
             with 12 bytes in 1 section within [0, 11]
             using 1 page and 52 pad bytes
avrdude_git: writing 12 bytes eeprom ...

Writing |                                                    | 0% 0.00savrdude_git error: bad response to write memory command: RSP_ILLEGAL_MEMORY_TYPE
Writing | ################################################## | 100% 0.01s

avrdude_git: 12 bytes of eeprom written
avrdude_git: verifying eeprom memory against <stdin>

Reading | ################################################## | 100% 0.16s

avrdude_git: 12 bytes of eeprom verified

avrdude_git done.  Thank you.

PS C:\work\avr\avrdude_test\avrdude_bin> mode COM12 baud=12 dtr=on > nul && mode COM12 baud=12 dtr=off > nul && echo "Hello World" | .\avrdude_pr1132v3static -c jtag2updi -P COM12 -p m4809 -U eeprom:w:-:r

avrdude_pr1132v3static: AVR device initialized and ready to accept instructions
avrdude_pr1132v3static: device signature = 0x1e9651 (probably m4809)
avrdude_pr1132v3static: reading input file <stdin> for eeprom
                        with 12 bytes in 1 section within [0, 11]
                        using 1 page and 52 pad bytes
avrdude_pr1132v3static: writing 12 bytes eeprom ...

Writing | -------------------------------------------------- | 0% 0.01 s

avrdude_pr1132v3static error: bad response to write memory command: RSP_ILLEGAL_MEMORY_TYPE
avrdude_pr1132v3static: 12 bytes of eeprom written
avrdude_pr1132v3static: verifying eeprom memory against <stdin>

Reading | ################################################## | 100% 0.16 s

avrdude_pr1132v3static: 12 bytes of eeprom verified

avrdude_pr1132v3static done.  Thank you.

@mcuee
Copy link
Collaborator

mcuee commented Oct 20, 2022

@stefanrueger

In the end, I think it is a good pull request and we need to fix the issue with macOS. We will need CMake experts like @mariusgreuel to help out in this case.

Windows is actually not too bad (MSVC build works, MSYS2 build works if people use minimum MSYS2 installation just for avrdude build, like github action build). Both will not use readline though. We can always wait for @mariusgreuel to help on the MSVC side. The MSYS2 issue with libreadline looks like a MSYS2 packaging issue of either readline or termcap.

@mcuee
Copy link
Collaborator

mcuee commented Oct 22, 2022

I figured out why macOS is using libedit in the end. It automatically pull in the SDK library /Library/Developer/CommandLineTools/SDKs/MacOSX13.0.sdk/usr/lib/libreadline.tbd.

//Path to a library.
HAVE_LIBELF:FILEPATH=/opt/homebrew/lib/libelf.a

//Path to a library.
HAVE_LIBFTDI:FILEPATH=/opt/homebrew/lib/libftdi.dylib

//Path to a library.
HAVE_LIBFTDI1:FILEPATH=/opt/homebrew/lib/libftdi1.dylib

//Path to a library.
HAVE_LIBHID:FILEPATH=/opt/homebrew/lib/libhid.dylib

//Path to a library.
HAVE_LIBHIDAPI:FILEPATH=/opt/homebrew/lib/libhidapi.dylib

//Path to a library.
HAVE_LIBREADLINE:FILEPATH=/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/lib/libreadline.tbd

//Path to a library.
HAVE_LIBUSB:FILEPATH=/opt/homebrew/lib/libusb.dylib

//Path to a library.
HAVE_LIBUSB_1_0:FILEPATH=/opt/homebrew/lib/libusb-1.0.dylib

//Path to a library.
HAVE_LIBUSB_WIN32:FILEPATH=HAVE_LIBUSB_WIN32-NOTFOUND

And apparently that libreadline.tbd is actually libedit.

cmake -D CMAKE_C_FLAGS=-I/opt/homebrew/include 
-D CMAKE_EXE_LINKER_FLAGS=-L/opt/homebrew/Cellar 
-D CMAKE_BUILD_TYPE=RelWithDebInfo 
-D CMAKE_VERBOSE_MAKEFILE=TRUE 
-B build_macos

cmake --build build_macos
...
[100%] Linking C executable avrdude
cd /Users/mcuee/build/avr/avrdude_test/avrdude_sr/build_macos/src && /opt/homebrew/Cellar/cmake/3.24.2/bin/cmake -E cmake_link_script CMakeFiles/avrdude.dir/link.txt --verbose=1
/Library/Developer/CommandLineTools/usr/bin/cc -I/opt/homebrew/include -O2 -g -DNDEBUG -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.0.sdk -mmacosx-version-min=12.6 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L/opt/homebrew/Cellar CMakeFiles/avrdude.dir/main.c.o CMakeFiles/avrdude.dir/term.c.o CMakeFiles/avrdude.dir/avrintel.c.o CMakeFiles/avrdude.dir/developer_opts.c.o CMakeFiles/avrdude.dir/whereami.c.o -o avrdude  libavrdude.a -lm /opt/homebrew/lib/libelf.a /opt/homebrew/lib/libusb.dylib /opt/homebrew/lib/libusb-1.0.dylib /opt/homebrew/lib/libhid.dylib /opt/homebrew/lib/libhidapi.dylib /opt/homebrew/lib/libftdi.dylib /opt/homebrew/lib/libftdi1.dylib /Library/Developer/CommandLineTools/SDKs/MacOSX13.0.sdk/usr/lib/libreadline.tbd 
[100%] Built target avrdude
/opt/homebrew/Cellar/cmake/3.24.2/bin/cmake -E cmake_progress_start /Users/mcuee/build/avr/avrdude_test/avrdude_sr/build_macos/CMakeFiles 0

mcuee@mcuees-Mac-mini avrdude_sr % otool -L ./build_macos/src/avrdude
./build_macos/src/avrdude:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
	/opt/homebrew/opt/libusb-compat/lib/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
	/opt/homebrew/opt/libusb/lib/libusb-1.0.0.dylib (compatibility version 4.0.0, current version 4.0.0)
	/opt/homebrew/opt/libhid/lib/libhid.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/homebrew/opt/hidapi/lib/libhidapi.0.dylib (compatibility version 0.0.0, current version 0.12.0)
	/opt/homebrew/opt/libftdi0/lib/libftdi.1.dylib (compatibility version 22.0.0, current version 22.0.0)
	/opt/homebrew/opt/libftdi/lib/libftdi1.2.dylib (compatibility version 2.0.0, current version 2.5.0)
	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)

Manually replacing the SDK readline with homebrew readline ( /opt/homebrew/lib/libreadline.dylib) and it will be okay. The libreadline version works well and CTRL-D works fine as EOF or quit.

mcuee@mcuees-Mac-mini avrdude_bin % otool -L avrdude_pr1132_libreadline 
avrdude_pr1132_libreadline:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
	/opt/homebrew/opt/libusb-compat/lib/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
	/opt/homebrew/opt/libusb/lib/libusb-1.0.0.dylib (compatibility version 4.0.0, current version 4.0.0)
	/opt/homebrew/opt/libhid/lib/libhid.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/homebrew/opt/hidapi/lib/libhidapi.0.dylib (compatibility version 0.0.0, current version 0.12.0)
	/opt/homebrew/opt/libftdi0/lib/libftdi.1.dylib (compatibility version 22.0.0, current version 22.0.0)
	/opt/homebrew/opt/libftdi/lib/libftdi1.2.dylib (compatibility version 2.0.0, current version 2.5.0)
	/opt/homebrew/opt/readline/lib/libreadline.8.dylib (compatibility version 8.2.0, current version 8.2.0)
mcuee@mcuees-Mac-mini avrdude_bin % otool -L avrdude_pr1132_libedit    
avrdude_pr1132_libedit:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
	/opt/homebrew/opt/libusb-compat/lib/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
	/opt/homebrew/opt/libusb/lib/libusb-1.0.0.dylib (compatibility version 4.0.0, current version 4.0.0)
	/opt/homebrew/opt/libhid/lib/libhid.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/homebrew/opt/hidapi/lib/libhidapi.0.dylib (compatibility version 0.0.0, current version 0.12.0)
	/opt/homebrew/opt/libftdi0/lib/libftdi.1.dylib (compatibility version 22.0.0, current version 22.0.0)
	/opt/homebrew/opt/libftdi/lib/libftdi1.2.dylib (compatibility version 2.0.0, current version 2.5.0)
	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)

@mcuee
Copy link
Collaborator

mcuee commented Oct 22, 2022

From the following it seems simple to force macOS to use libreadline and not libedit, unfortunately it does not work out of the box.
https://stackoverflow.com/questions/37548448/how-to-force-cmake-to-link-against-homebrew-keg-only-libraries

I tried the following build options and none of them worked.

export PKG_CONFIG_PATH="/opt/homebrew/opt/readline/lib/pkgconfig" && ./build.sh

LDFLAGS="-L/opt/homebrew/opt/readline/lib" && CPPFLAGS="-I/opt/homebrew/opt/readline/include" && ./build.sh

LDFLAGS="-L/opt/homebrew/opt/readline/lib" && CPPFLAGS="-I/opt/homebrew/opt/readline/include" && export PKG_CONFIG_PATH="/opt/homebrew/opt/readline/lib/pkgconfig" && ./build.sh

@mcuee
Copy link
Collaborator

mcuee commented Oct 22, 2022

@stefanrueger and @mariusgreuel

I have finally figured out a simple way to fix for macOS. There is no need to change CMake files. This works fine.

  1. force link readline (keg-only) by using `brew link readline --force"
  2. modify build.sh with the following changes.
mcuee@mcuees-Mac-mini avrdude_sr % git diff
diff --git a/build.sh b/build.sh
index bbb2f58..879c0a7 100755
--- a/build.sh
+++ b/build.sh
@@ -53,9 +53,9 @@ case "${ostype}" in
             # Apple M1 (may be new version of homebrew also)
             if [ -d /opt/homebrew ]  
             then
-                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/opt/homebrew/include -D CMAKE_EXE_LINKER_FLAGS=-L/opt/homebrew/Cellar"
+                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/opt/homebrew/include -D CMAKE_EXE_LINKER_FLAGS=-L/opt/homebrew/Cellar -D HAVE_LIBREADLINE:FILEPATH=/opt/homebrew/lib/libreadline.dylib"
             else
-                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/local/Cellar"
+                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/local/Cellar -D HAVE_LIBREADLINE:FILEPATH=/usr/local/lib/libreadline.dylib"
             fi
        fi
        ;;

@mcuee
Copy link
Collaborator

mcuee commented Oct 22, 2022

@stefanrueger
I am thinking you can merge the PR and we can deal with build.sh in another PR for macOS. MinGW MSYS2 is not a big issue as github action is okay. And the official Windows binary will be done using MSVC (without readline as of now).

@stefanrueger
Copy link
Collaborator Author

From my tests: Looks like only the github runners for linux-x86_64 and macos-x86_64 Linux actually define HAVE_LIBREADLINE. The others

  • linux (i386, i686, i686-linux-gnu, i386-linux-gnu)
  • linux (armhf, armhf, arm-linux-gnueabihf, arm-linux-gnueabihf)
  • linux (arm64, aarch64, aarch64-linux-gnu, aarch64-linux-gnu)
  • msvc (x86, Win32)
  • msvc (x64, x64)
  • msvc (arm64, ARM64)
  • mingw (mingw32, i686)
  • mingw (mingw64, x86_64)

do not set HAVE_LIBREADLINE for the CI tests. In particular, @mariusgreuel's readytoread() function in src/term.c will not have been compiled in the CI.

@mcuee
Copy link
Collaborator

mcuee commented Oct 23, 2022

From my tests: Looks like only the github runners for linux-x86_64 and macos-x86_64 Linux actually define HAVE_LIBREADLINE. The others do not set HAVE_LIBREADLINE for the CI tests. In particular, @mariusgreuel's readytoread() function in src/term.c will not have been compiled in the CI.

I believe the following will for sure not have libreadline since the library is not installed by github action

  • msvc (x86, Win32)
  • msvc (x64, x64)
  • msvc (arm64, ARM64)
  • mingw (mingw32, i686)
  • mingw (mingw64, x86_64)

MSYS2 mingw anyway has problems with libreadline build as of now due to the packaging issues with termcap.

Need to check why the following does not build with readline.

  • linux (i386, i686, i686-linux-gnu, i386-linux-gnu)
  • linux (armhf, armhf, arm-linux-gnueabihf, arm-linux-gnueabihf)
  • linux (arm64, aarch64, aarch64-linux-gnu, aarch64-linux-gnu)

@mcuee
Copy link
Collaborator

mcuee commented Oct 23, 2022

Need to check why the following does not build with readline.

  • linux (i386, i686, i686-linux-gnu, i386-linux-gnu)

@stefanrueger and @mariusgreuel
This one is strange. From the build log libreadline is installed. Need to check why HAVE_LIBREADLINE is not defined.l

  • linux (armhf, armhf, arm-linux-gnueabihf, arm-linux-gnueabihf)
  • linux (arm64, aarch64, aarch64-linux-gnu, aarch64-linux-gnu)

@stefanrueger and @mariusgreuel
Indeed libreadline is not installed for the ARM 32bit/64bit Linu build. We need to add that.

@mcuee
Copy link
Collaborator

mcuee commented Oct 23, 2022

@stefanrueger

You can try to add libreadline-dev in the github actions. It should work for Linux build now.
(Edit to add: this works and I have created a pull request).

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 916b7f1..2ac4313 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -49,6 +49,7 @@ jobs:
           libusb-1.0-0-dev
           libhidapi-dev
           libftdi1-dev
+          libreadline-dev
           texinfo
           texlive
           texi2html
@@ -110,6 +111,7 @@ jobs:
           libusb-1.0-0-dev:${{matrix.arch}}
           libhidapi-dev:${{matrix.arch}}
           libftdi1-dev:${{matrix.arch}}
+          libreadline-dev:${{matrix.arch}}
       - name: Configure
         run: >-
           cmake

@mcuee
Copy link
Collaborator

mcuee commented Oct 23, 2022

@stefanrueger
As for macOS, if we want to test readline, we need to change github action as well. I am not so familiar with github action, but something like the following. I will test this out first to see if it works or not.

Edit to add: this does not work. github action will fail. Need to study further.
https://github.com/mcuee/avrdude/actions/runs/3306074363/jobs/5456692962

@mcuee
Copy link
Collaborator

mcuee commented Oct 23, 2022

From my tests: Looks like only the github runners for linux-x86_64 and macos-x86_64 Linux actually define HAVE_LIBREADLINE. The others

  • linux (i386, i686, i686-linux-gnu, i386-linux-gnu)
  • linux (armhf, armhf, arm-linux-gnueabihf, arm-linux-gnueabihf)
  • linux (arm64, aarch64, aarch64-linux-gnu, aarch64-linux-gnu)
  • msvc (x86, Win32)
  • msvc (x64, x64)
  • msvc (arm64, ARM64)
  • mingw (mingw32, i686)
  • mingw (mingw64, x86_64)

do not set HAVE_LIBREADLINE for the CI tests. In particular, @mariusgreuel's readytoread() function in src/term.c will not have been compiled in the CI.

I tested in my avrdude fork and created a pull request #1146. for the three Linux build. Please review.

@mcuee
Copy link
Collaborator

mcuee commented Oct 23, 2022

Pull request #1148 created for MSYS2 ming32/mingw64. Interestingly it works in my avrdude fork test. Maybe my MSYS2 installation needs a revamp.
https://github.com/mcuee/avrdude/actions/runs/3306107484

@mcuee
Copy link
Collaborator

mcuee commented Oct 23, 2022

@stefanrueger As for macOS, if we want to test readline, we need to change github action as well. I am not so familiar with github action, but something like the following. I will test this out first to see if it works or not.

Edit to add: this does not work. github action will fail. Need to study further. https://github.com/mcuee/avrdude/actions/runs/3306074363/jobs/5456692962

Finally figured out the way for macOS, I have created pull request #1149 to add real readline (and not libedit) for macOS.

@stefanrueger stefanrueger merged commit 5b008a0 into avrdudes:main Oct 23, 2022
@mcuee
Copy link
Collaborator

mcuee commented Oct 24, 2022

Pull request #1148 created for MSYS2 ming32/mingw64. Interestingly it works in my avrdude fork test. Maybe my MSYS2 installation needs a revamp. https://github.com/mcuee/avrdude/actions/runs/3306107484

In the end it is not my MSYS2 installation problem.

Looking at the failed build log (https://github.com/avrdudes/avrdude/actions/runs/3308684165/jobs/5461248027), the issue is still the MSYS2 readline/termcap packaging issues mentioned here.

On my local machine I can fix the issue by rebuilding termcap as per the tips from MSYS2 github issue.

I'll see what I can do to correct this issue using github action.

@mcuee
Copy link
Collaborator

mcuee commented Oct 24, 2022

Finally figured out the way for macOS, I have created pull request #1149 to add real readline (and not libedit) for macOS.

I believe this works but I may have messed up the git thingy. I have to create a new PR.

@mcuee
Copy link
Collaborator

mcuee commented Oct 28, 2022

For those who want to test readline and static linking mingw32 Windows version of avrdude git main, you can try the binary mentioned here.

I did some simple test and it works well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
3 participants