value.toWire raises NotImplementedError

[donalm]

The 'quick usage example' on ldaptor's github homepage works for me (with minimal obvious modifications) on version 16 of ldaptor on Python 2.7

For version 19 of ldaptor, I get this error with python 3.6.5

Unhandled Error
Traceback (most recent call last):
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/twisted/python/log.py", line 103, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/twisted/python/log.py", line 86, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
    why = selectable.doRead()
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/twisted/internet/tcp.py", line 243, in doRead
    return self._dataReceived(data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/twisted/internet/tcp.py", line 249, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/ldaptor/protocols/ldap/ldapclient.py", line 70, in dataReceived
    if not o:
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/ldaptor/protocols/pureber.py", line 132, in __len__
    return len(self.toWire())
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/ldaptor/protocols/pureldap.py", line 122, in toWire
    return BERSequence(l).toWire()
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/ldaptor/protocols/pureber.py", line 312, in toWire
    r = b''.join(to_bytes(x) for x in self.data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/ldaptor/protocols/pureber.py", line 312, in <genexpr>
    r = b''.join(to_bytes(x) for x in self.data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/ldaptor/_encoder.py", line 19, in to_bytes
    return value.toWire()
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-3.6/lib/python3.6/site-packages/ldaptor/protocols/pureldap.py", line 139, in toWire
    raise NotImplementedError()
builtins.NotImplementedError: 

And the same error with version 19 of ldaptor on Python 2.7.14:

Unhandled Error
Traceback (most recent call last):
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/twisted/python/log.py", line 103, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/twisted/python/log.py", line 86, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
    why = selectable.doRead()
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/twisted/internet/tcp.py", line 243, in doRead
    return self._dataReceived(data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/twisted/internet/tcp.py", line 249, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/ldaptor/protocols/ldap/ldapclient.py", line 70, in dataReceived
    if not o:
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/ldaptor/protocols/pureber.py", line 132, in __len__
    return len(self.toWire())
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/ldaptor/protocols/pureldap.py", line 122, in toWire
    return BERSequence(l).toWire()
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/ldaptor/protocols/pureber.py", line 312, in toWire
    r = b''.join(to_bytes(x) for x in self.data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/ldaptor/protocols/pureber.py", line 312, in <genexpr>
    r = b''.join(to_bytes(x) for x in self.data)
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/ldaptor/_encoder.py", line 19, in to_bytes
    return value.toWire()
  File "/mnt/software/external/python_extras/0.8.0/platform-linux/python-2.7/lib/python2.7/site-packages/ldaptor/protocols/pureldap.py", line 139, in toWire
    raise NotImplementedError()
exceptions.NotImplementedError:

Tweaking the master branch ( d3c1919 ), I was able to suppress the error (in both Python interpreters) by modifying LDAPSearchResultReference in ldaptor/protocols/pureldap.py:

    def toWire(self):                                                           
        return b''

So I guess that's what the code is trying to convert to bytes. I'm not sure why I seem to be the first person to run into this. I don't think I'm doing anything unexpected.

Thanks!

[GrayAn]

Thanks, @donalm ! What LDAP server were you using when trying the quick start example?

It looks like I have broken LDAPSearchResultReference while porting the code for Python 3. It was not fully implemented before but certainly did not raise exceptions.

[donalm]

ActiveDirectory is providing our LDAP service. Thanks for looking at this @GrayAn

[donalm]

Also (maybe): ldaptor/protocols/pureldap.py :

1503c1503
<     oid = b'1.3.6.1.4.1.1466.20037'
---
>     oid = '1.3.6.1.4.1.1466.20037'
1528c1528
<     oid = b'1.3.6.1.4.1.1466.20037'
---
>     oid = '1.3.6.1.4.1.1466.20037'

Sorry to just dump this diff in here and run. I'll do some isolation testing with this and update the ticket.

UPDATE: I'm seeing this problem only in my Python 3 environment. When I add client.startTLS() after the connect call:

client = yield c.connect(basedn, overrides=overrides)
client.startTLS()

I get this ldaptor.protocols.ldap.ldapclient.LDAPStartTLSInvalidResponseName exception:

/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/python/reflect.py:448: DeprecationWarning: LDAPStartTLSInvalidResponseName.__str__ method is deprecated and will not be used for getting bytes representation in the future releases, use LDAPStartTLSInvalidResponseName.toWire instead
  return str(o)
/opt/pipeline/python3/lib/python3.6/traceback.py:151: DeprecationWarning: LDAPStartTLSInvalidResponseName.__str__ method is deprecated and will not be used for getting bytes representation in the future releases, use LDAPStartTLSInvalidResponseName.toWire instead
  return str(value)
[Failure instance: Traceback: <class 'ldaptor.protocols.ldap.ldapclient.LDAPStartTLSInvalidResponseName'>: <LDAPStartTLSInvalidResponseName instance at 0x7fb844bc6a68 with str error:
 Traceback (most recent call last):
  File "/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/mnt/software/dev/dmcmullan/ldap_test/bin/py/minimal3.py", line 16, in example
    client = yield client.startTLS()
  File "/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/scratch/geek/ldaptor/ldaptor/protocols/ldap/ldapclient.py", line 256, in _cbStartTLS
    raise LDAPStartTLSInvalidResponseName(msg.responseName)
ldaptor.protocols.ldap.ldapclient.LDAPStartTLSInvalidResponseName: <unprintable LDAPStartTLSInvalidResponseName object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/python/reflect.py", line 448, in safe_str
    return str(o)
TypeError: __str__ returned non-string (type bytes)
>
/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py:460:callback
/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py:568:_startRunCallbacks
/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py:654:_runCallbacks
/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py:1475:gotResult
--- <exception caught here> ---
/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/mnt/software/dev/dmcmullan/ldap_test/bin/py/minimal3.py:16:example
/mnt/software/dev/dmcmullan/ldap_test/bin/v3/lib/python3.6/site-packages/twisted/internet/defer.py:654:_runCallbacks
/scratch/geek/ldaptor/ldaptor/protocols/ldap/ldapclient.py:256:_cbStartTLS
]

The code works fine in Python 2. The diff I pasted above suppresses the error in Python 3.

Thanks again @GrayAn

[cwaldbieser]

@donalm - If I want to test a patch for the SearchResultReference issue, I have access to an AD environment, but I don't really know much about administering it. Is there some kind of explicit reference I can create in the directory that I can query for testing?

[donalm]

I hope you won't need to jump through too many hoops - the filterText that I'm using is just:
([email protected])
...to return the directory info for one user.

(I also don't administer the AD instance - I just query it).

[GrayAn]

@donalm Maybe you can try to test the latest changes against your AD instance? You can use this branch for it: startTLS, LDAPSearchResultReference and exception issues were fixed in it.