Skip to content

Commit

Permalink
[Feature] support log more sql info to log file and pgrepaly can
Browse files Browse the repository at this point in the history
analyze poalrdb log file and monitor db system info.
1. support log search_path info to log file.
2. support log paremeters type for parameterization SQL. Only parameters
type is logged for extended protocol. Parameters type and definnitation
    of SQL are logged for PREPARED simple protocol.
3. Add pgreplay tool. currently, this version pgreplay can
support polardb audit log and workload replay monitor function
4. Support extension protocol of pg execution and normal execution
when prepare is not present on pgreplay.
  • Loading branch information
qinjingyuan authored and polardb-bot[bot] committed Oct 30, 2023
1 parent eda86b4 commit 6071c25
Show file tree
Hide file tree
Showing 34 changed files with 16,911 additions and 25 deletions.
33 changes: 33 additions & 0 deletions pgreplay/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Object files
*.o
*.obj

# Libraries
*.lib
*.a

# Shared objects (inc. Windows DLLs)
*.dll
*.so
*.so.*
*.dylib

# build products
pgreplay
pgreplay.exe
Makefile
# config.h
config.log
config.status

# test subdirectory
test/

# other versioning systems
.svn
CVS

# M4 cache
autom4te.cache/
.vscode
devtest
98 changes: 98 additions & 0 deletions pgreplay/CHANGELOG
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
Version 1.4.0

Enhancements:
- Add a replay option "-n" that provides a "dry run" without actually
connecting to the database.
Patch by Manuel Vives.

Bugfixes:
- Fix Makefile to call "install" portably.
It used to call "$(INSTALL) -D", which works on some, but not all systems
(MAC OS X is an example where it doesn't work).
Report by Jeff Doering.
- Fix crash after replay of DEALLOCATE ALL.
This caused a crash the next time prepared statements were deleted, for
example at the end of the database session.
Reported by Wan Shen Lim.

Version 1.3.0 Feb 20 2017

Enhancements:
- Accept CPPFLAGS from configure in Makefile.in.
Patch by Marko Kreen.
- Add command line option -X to specify extra connect string fragment.
Needed to specify unusual connect string options.
Patch by Marko Kreen.
- Introduce replay filter options with "-D database" and "-U username"
to filter for a database or user during parsing.
Patch by Gilles Darold.

Version 1.2.0 Aug 17 2012

Enhancements:
- Introduce replay option "-j" to jump ahead when all connections are idle.
This can speed up replay. The statistics will not include the skipped time,
but delay warnings will work as expected.
Idea and original patch by John Lumby.

Bugfixes:
- Fix failure to parse string constants like E'\\' where the backslash before
a quotation mark is backslash-quoted.
Bug discovered by Jeff Frost.

Version 1.1.0 Feb 09 2012

Enhancements:
- Respect environment variable DESTDIR in Makefile for RPM packagers.
- Improve execution delay reporting by introducing more intelligent time
steps when a report is written; every 10 seconds is way too spammy.
- Add documentation for interaction with pgFouine to the README.

Bugfixes:
- Fix incorrect assumption that everything that starts with a dollar
sign is a dollar quoted string. This used to trigger bogus "end of dollar
quote not found" error messages when $n placeholders are used in PREPARE
statements. Discovered and fixed by Todd Owen.
- When pgreplay waited for a response on a connection because it needed to
send the next command, it used to sleep for one millisecond before
polling the socket again. This proved to be too long on busy systems,
where replay started to lag behind. Now pgreplay will not sleep,
but keep polling until the response is there.

Version 1.0.0 Jun 03 2011

Bugfixes:
- Fix a connection and memory leak introduced by the new handling of FATAL
connection errors in 0.9.1.
Discovered by Denis Kostin.

Version 0.9.1 Feb 26 2011

Enhancements:
- Calculate parse and execution statistics and display them at the end
of the run.

Bugfixes:
- Use "=" instead of "==" in "test" comparisons in configure.
This improves portability.
- Change replay file timestamp to seconds after midnight of 2000-01-01
in local time. This makes the replay file format independent of time zones
and avoids problems with mktime(3) implementations that don't like
the UNIX epoch.
- Ignore string literals in filter_bad_statements during log file parsing.
This keeps the function from getting confused by the contents of the
string. Discovered by Josh Berkus.
- Correctly handle prepared statements without parameters.
Discovered by Grigorij Lipin.
- Fix a corner case bug in read_log_line that can cause data corruption
when parsing a stderr log. Discovered by Grigorij Lipin.
- Skip memory dumps in stderr log caused by "out of memory" errors
instead of gagging on them. Discovered by Grigorij Lipin.
- Don't gag if a connection attempt results in a FATAL error during replay.
This can for example happen if max_connections has been exceeded or if a
non-existant user is specified with "trust" authentication.
Discovered by Grigorij Lipin.

Version 0.9.0 Mar 19 2010

- first release
16 changes: 16 additions & 0 deletions pgreplay/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
FROM ubuntu:22.10

RUN TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update && \
apt-get install --no-install-recommends -y tzdata make gcc libc6-dev postgresql-14 libpq-dev postgresql-doc-14 git ca-certificates && \
apt-get clean && rm -rf /var/lib/apt/lists/*

WORKDIR /root
RUN git clone https://github.com/laurenz/pgreplay.git
WORKDIR /root/pgreplay
RUN ./configure --with-postgres=/usr/bin
RUN make
RUN make install
RUN ln -s /root/pgreplay/pgreplay /usr/local/bin

19 changes: 19 additions & 0 deletions pgreplay/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Portions Copyright (c) 2017-2023, CYBERTEC PostgreSQL International GmbH
Portions Copyright (c) 2010-2017, Magistrat der Stadt Wien

Permission to use, copy, modify, and distribute this software and its
documentation for any purpose, without fee, and without a written agreement
is hereby granted, provided that the above copyright notice and this paragraph
and the following two paragraphs appear in all copies.

IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE TO ANY PARTY FOR
DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING
LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION,
EVEN IF THE COPYRIGHT HOLDER HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

THE COPYRIGHT HOLDER SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS,
AND THE COPYRIGHT HOLDER HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE,
SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
64 changes: 64 additions & 0 deletions pgreplay/Makefile.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
CFLAGS = @CFLAGS@
CPPFLAGS = @CPPFLAGS@
LDFLAGS = @LDFLAGS@
LIBS = @LIBS@
CC = @CC@
EXTRA_OBJS = @EXTRA_OBJS@
prefix = @prefix@
exec_prefix = @exec_prefix@
bindir = @bindir@
datarootdir = @datarootdir@
datadir = @datadir@
mandir = @mandir@
docdir = @docdir@
htmldir = @htmldir@
INSTALL = @INSTALL@
VERSION = @PACKAGE_VERSION@
EXE = pgreplay
OBJS = parse.o replayitem.o main.o replayfile.o database.o $(EXTRA_OBJS)
HEADERS = config.h pgreplay.h

.PHONY: clean distclean tarball test install install_bin install_man install_html

.SUFFIXES: .o .c .h

$(EXE): $(OBJS) $(HEADERS)
$(CC) -o $@ $(CFLAGS) $(CPPFLAGS) $(OBJS) $(LDFLAGS) $(LIBS)

$(OBJS): $(HEADERS)

clean:
rm -f $(OBJS) $(EXE)

distclean: clean
rm -rf Makefile config.h config.h.in~ config.log config.cache config.status autom4te.cache test/runtest.sh pgreplay-$(VERSION).tar*

tarball: distclean
if tar --version 2>/dev/null | grep -q GNU; then : ;else echo "this requires GNU tar" 1>&2; exit 1; fi
ln -s "`pwd`" /tmp/pgreplay-$(VERSION)
tar -cf pgreplay-$(VERSION).tar -h -C /tmp --exclude .svn --exclude CVS --exclude pgreplay-$(VERSION).tar pgreplay-$(VERSION) --exclude test/testrun.c
rm /tmp/pgreplay-$(VERSION)
gzip -9 pgreplay-$(VERSION).tar

$(EXE).html: $(EXE).1
groff -Thtml -mman $< > $@

test: $(EXE) test/runtest.sh
cd test && ./runtest.sh

install: install_bin install_man

install_bin: $(EXE)
$(INSTALL) -d $(DESTDIR)$(bindir)
$(INSTALL) -m 0755 $< $(DESTDIR)$(bindir)/$<

install_man: $(EXE).1
$(INSTALL) -d $(DESTDIR)$(mandir)/man1
$(INSTALL) -m 0644 $< $(DESTDIR)$(mandir)/man1/$<

install_html: $(EXE).html
$(INSTALL) -d $(DESTDIR)$(htmldir)
$(INSTALL) -m 0644 $< $(DESTDIR)$(htmldir)/$<

.c.o:
$(CC) -c $(CFLAGS) -DVERSION='"$(VERSION)"' $<
Loading

0 comments on commit 6071c25

Please sign in to comment.