Python 3 support

[tfeldmann]

pysqlcipher cannot be installed with

sudo pip3 install pysqlcipher

because it fails with syntax errors for the python2 print statement.

[kalikaneko]

See some general comments on #11. Patches or reports beyond the print statement are welcome.

[rigglemania]

Because this was the only project preventing me from using Python 3 in one of my projects, I looked at what would be required to get this working in Python 3 and while it definitely required some work, I think I have a working port verified by being able to encrypt/decrypt a SQLite database in a test project. Due to the significance of the changes (including starting from scratch for the entire src directory), I think a new repo is the best approach (i.e. pysqlcipher3). I'll try to get a repo set up in the next few days with my changes, but here's an overview of what I had to do.

1. Fixed all print statements
2. Fixed some encoding issues in reading files due to changes in Python 3
3. Fixed issues with checking for the type of an instance (ListType and TupleType are no longer valid)
4. Grabbed the latest Python core source code and copied the files from /Modules/_sqlite/* into the /src directory of this project (deleting all existing files first). From the research I did, why the pysqlite repo seems to only support Python 2, pysqlite is the external name for the core sqlite3 library, which is why I grabbed the source files from there.
5. Merged changes from /Lib/sqlite3/* from the Python core source code into the /lib directory of this project
6. Renamed the extension in setup.py from pysqlcipher._sqlite to pysqlcipher._sqlite3 as in Python 3, naming conventions of the init method in module.c matter as the module names much match exactly.

Hopefully, I'm on the right track (I'm still relatively new to Python, but am an experienced software developer). Let me know if you have any questions or thoughts.

[tfeldmann]

@rigglemania This is the best thing I read all day. Thank you for your work.

[kalikaneko]

exciting! I wonder if the py3 version will be able to be run on python2.7 also.
waiting to see this new repo :)

[rigglemania]

No, unfortunately the python 3.4 version won't be able to be run on python 2.7. While some of the simple python 3 changes like the print statement could possibly be handled with the future module, the changes to the concrete types layer used in the .c and .h files seem like to big of changes. I'm making progress on the repo, but it's going a little bit slower because I'm actually trying to fully understand the setup.py file to clean it up where possible (I'm starting from scratch and including what is actually necessary). Unfortunately, the documentation on disttools is pretty weak so I'm finding I'm learning the most from actually digging through the source code.

[rigglemania]

@kalikaneko What's the purpose of the build_static command? Is this something that is really necessary and if so, how is the functionality used and how can it be tested. To me, it seems like we would be missing the static linking when compiling as described here: https://www.zetetic.net/sqlcipher/introduction/

[kalikaneko]

@rigglemania: it can be ignored, and I probably will remove it from setup. It was an attempt to compile the extension against a statically compiled sqlcipher, but that turned out to be a bad idea that ended nowhere (all the _build_extension method in MyBuildExt class can be ignored too). Currently, if you attempt to execute python setup.py build_static, you end up with a libsqlite.a artifact in the build folder, but I found that it cannot be loaded as an extension (which has to be dynamically linked against interpreter, is my reading. I think what would probably make sense is to first compile sqlcipher statically, and then link extension against that, but that's not what I ended up doing).

as I commented in other issue, I think the best option for a future release will be to follow upstream pysqlite steps and remove the amalgamation-as-default option, and to assume libsqlcipher is somewhere to be found in the system instead.

I'm a extension n00b myself, but I can try to help you with the porting.

Thinking about packaging this for debian/ubuntu and its eventual inclusion (with both py2 and py3 support) maybe one option will be to include two distinct source folders in the same repo, so both packages can be generated from the same tree, and then cleaning up setup.py so that it can be executed with both python versions. Will you be interested in co-maintaining the package?

[kalikaneko]

@rigglemania also, while cleaning setup.py it will make a lot of sense to rename classes so they are more meaningful.

One step on this direction is to clearly mark each of the two builders:
kalikaneko@c155277#diff-2eeaed663bd0d25b7e608891384b7298R378

[rigglemania]

I think we're on the same page as I've already done a lot of things, including some of the renames as well as switching the default to link against system lib rather than the amalgamation. I've got everything working in my local repo now (all unit tests passing) with the exception of the cross_bdist_wininst. Is this class currently working with Python 2.7?

Also, good idea on the different source folders idea to support both Python versions. I think I may be able to get that working without too much effort. I'm not sure about the tests files yet and if there are any Python 3 changes in them.

Finally, before I realized the static lib setup code wouldn't work, I actually had the same lib file being generated (at least the sizes were exactly the same) using the build_clib command with only around 10-20 lines. So if you do end up messing some more with creating the static lib, you should check out this command rather than copying/pasting for the _build_ext method.

[rigglemania]

So looking more at the cross_bdist_wininst file, I ended up deleting this file in my repo and removed the command from the setup.py file. Maybe it could work if building against the amalgamation, but since this isn't recommended for production, I don't see how building a Windows C extension on Linux linking against the Linux SqlCipher library would work.

I've got my repo up now: https://github.com/rigglemania/pysqlcipher3. It works against both Python 2 and Python 3. My testing consisted of both building linking against SQL Cipher (now the default behavior) and building the amalgamation (responsibility is now on user to provider source files similar to pysqlite). Then after installing against both Python versions, I ran all the unit tests, which passed (after a slight modification to a Python 2 test). I also tested creating a simple encrypted DB as described in the usage section. I still want to write a few unit tests for the libsqlcipher functionality, but other than that, I think the repo shouldn't need much more changes.

[molecular]

trying to get OpenBazaar to work which is using pysqlcipher. I'm running into this issue and am not sure how to get around it.

[kalikaneko]

@rigglemania this is awesome!
as you probably saw the cross_bdist_wininst file came from the pysqlite fork. I guess it might be useful to produce a self-contained extension for windows, but I agree that for now windows support might be a bit far.
I will set some time this week for testing your package, and if it looks so well as it seems we could release a 3.0 version in pypi, and the matching debian packages. I think we could have the 3.0 package published but not being the default relase for some time, so that users can adapt to the non-amalgamation by default.

@molecular the current status of pysqlcipher is only python 2.x support. Very likely there will a package available next week with py3 support thanks to rigglemania work!

[rigglemania]

@kalikaneko By removing cross_bdist_wininst.py, it doesn't mean that you can't create a self-contained extension for Windows. It just means that you have to build the extension on Windows and not Linux or Mac (which sounds unsafe to me anyways).

As for my repo, I published a package on PyPi as 'pysqlcipher3' (due to request to so). I was thinking that if you merged my changes into your repo, you could publish the same 'pysqlcipher' package on PyPi and it would work for both versions of python.

Finally, I just checked in some sqlcipher unit tests I wrote as well as a bug fix for the Python 2 version, so you'll want to grab the latest.

[yardenac]

Any updates on this? Should I just use @rigglemania's fork? https://github.com/rigglemania/pysqlcipher3

[csawyerYumaed]

still doesn't seem to support python3:

user@zoboFab-2  ~  pip3 install pysqlcipher
Collecting pysqlcipher
Downloading pysqlcipher-2.6.10.tar.gz (2.0MB)
100% |████████████████████████████████| 2.0MB 715kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/private/var/folders/n2/vz___1c91wx7qd4224njp0sm0000gn/T/pip-build-noifky2n/pysqlcipher/setup.py", line 64
print "CFLAGS", os.environ['CFLAGS']
^
SyntaxError: Missing parentheses in call to 'print'

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/n2/vz___1c91wx7qd4224njp0sm0000gn/T/pip-build-noifky2n/pysqlcipher/

[kalikaneko]

@csawyerYumaed you can use the python3-enabled fork by @rigglemania:

pip3 install pysqlcipher3

I have plans to merge back the fork, specially for submitting a unified package to debian, but I'm falling to find time for this.

[csawyerYumaed]

Thanks @kalikaneko I hope you find time soon! :)

[embray]

@kalikaneko Would you accept a pull request to reintegrate the Python 3 fork? IIUC it's not actually a huge diff to the pysqlcipher sources themselves.

[tomwwagstaff-product]

I'm working in an Anaconda environment. I've tried pip/pip3 install pysqlcipher3. It claims to complete successfully but then when I try to use it I get a ModuleNotFoundError

[iammrj]

same thing happening with me also using PyCharm @tomwwagstaff-product

[LefterisJP]

Seems the current maintainer of the fork removed it from pypi: rigglemania/pysqlcipher3#14

[rigglemania]

I added the fork back (but was forced to bump the version and publish as 1.0.3). However, I don't have time to actively maintain the project, and won't be addressing any issues that come up. So use at your own risk. I have no problems, though, if my fork gets merged back here where maybe it can get a bit more love.

[LefterisJP]

@kalikaneko have you had any time to work on merging back @rigglemania's fork in here? I guess we should just do it eventually.

I understand people don't have time. Is there any way I can help? Would be great to have 1 single repo for pysqlcipher.

[kalikaneko]

In reply to some transmogrified bits by [email protected] on the 2018-10-04 08:35:23:

@kalikaneko have you had any time to work on merging back @rigglemania's fork in here? I guess we should just do it eventually.

hi, no unfortunately I haven't had time to dedicate to this. focus in the projects I'm part of has moved elsewhere lately so I'm not a daily user of this package anymore. eventually I'll have to do it, because I'm running out of time to deprecate the py2 codebase that I maintain. and it'll be nice to have an unified debian package, too.

I understand people don't have time. Is there any way I can help? Would be great to have 1 single repo for pysqlcipher.

any working patches, or comments about what has been tried are welcome. I was considering either porting the codebase, or doing something more hacky like just merging two folders together, one for py27 and py3. dirty, but I've seen it done and in any case py2 is almost EOL.

…

-- We reject kings, presidents and voting. We believe in rough consensus and working code.

[snoack]

I just started using SQLCipher in Python 3. Due to the lack of compatible bindings, I went to set LD_PRELOAD=libsqlcipher.so.0 in a wrapper script that runs my Python script, and then just use the built-in sqlite3 module. That works because SQLCipher is essentially a fork of SQLite and therefore all symbols provided by libsqlite3.so are implemented under the same name in libsqlcipher.so. However, this approach is Linux-specific, then again pysqlcipher has only been used on Linux as well so far (according to the README).

Anyway, I'd still love to see some proper Python 3 bindings for SQLCipher, but I wonder whether porting the existing code base makes much sense, since there are already well-tested (and possibly more feature-complete) SQLite bindings in the upstream Python development tree that could be build as-is against SQLCipher.

Taking the latest code of the upstream sqlite3 module, packaging it as a standalone module, and building it against SQLCipher (instead of SQLite) seems to be trivial. In particular with Python 2 being past end-of-life now, it seems backporting is no longer a concern (except for making sure that it builds against a small range of recent Python 3.x releases).

As I make all of this sound so easy you might wonder why I didn't already do it myself. While the initial setup might be fairly straightforward, it still takes some ongoing effort keeping up with new Python releases, but probably less (or at the very least not any more) than maintaining the current pysqlcipher code base (let alone porting it to the latest version of Python). Moreover, at the moment I don't have much stake in SQLCipher other than a hobby project where the LD_PRELOAD approach works just fine (and as it is packaged for Debian, I avoid any dependencies that are not already included in Debian, and until some Python bindings for SQLCipher make it into a Debian release will take at least another few years).

[embray]

@snoack That's an interesting point, thanks. I haven't had any reason to work on this lately myself so I haven't touched it, but I didn't realize how much of this codebase was a forked off the Python sqlite3 module.

I agree that the approach you suggest seems to make the most sense.

[snoack]

but I didn't realize how much of this codebase was a forked off the Python sqlite3 module.

For reference, it was the other way around. Starting with Python 2.5, pysqlite was merged into Python. In fact development of pysqlite is dead since Aug, 2016.