Mac users - 'tqdm' object has no attribute 'disable’

[gg4u]

If useful to Mac users, I met and solved this error when running segment_files_parallel() function,

Error was related to tqdm: 'tqdm' object has no attribute 'disable’ , probably linked to parallel processing.

I found comments in this repo:

3fon3fonov/exostriker#80

In the official tqdm repo, they suggest to bypass the issue with a wrapper:
tqdm/tqdm#487 (comment)

However, I was able to solve it by checking out install again with conda:

conda install tqdm

Conda said it was correctly installed, but (mysteriously) the issue was solved.

Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

My 2 cents : I suspect to have previously installed tqdm via pip. It may be useful to ensure packages are installed via conda (not sure about this...)

---

Update

Previous comment about conda install was NOT a solution.

Found again the same issue when running:

dataset.cluster_ids()

Now I have full trace-stack (below).

If I just re-run the function:

dataset.cluster_ids()

I get no error.

However, keeping in mind that:

• warning:
  If segmentation fails for a vocalisation it will be dropped from the database so that it doesn’t cause trouble downstream.
  in https://nilomr.github.io/pykanto/_build/html/_autosummary/pykanto.dataset.html?highlight=fail

• I had this issue when segmenting data: dataset.segment_into_units()

I wonder if there is a dataloss somewhere.

I don't know how to check for this, but for the sake of reproducibility and check of this error, this is the clustering I've got after running through all of the steps for Storm-petrel sample dataset (I kept parameters as their were in the tutorials):

https://nilomr.github.io/pykanto/_build/html/contents/interactive-app.html

(p.s. fabulous work!)

(_reduce_and_cluster_r pid=3295) Exception ignored in: <function tqdm.__del__ at 0x115653e50>
(_reduce_and_cluster_r pid=3295) Traceback (most recent call last):
(_reduce_and_cluster_r pid=3295)   File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 1162, in __del__
(_reduce_and_cluster_r pid=3295)     self.close()
(_reduce_and_cluster_r pid=3295)   File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 1281, in close
(_reduce_and_cluster_r pid=3295)     if self.disable:
(_reduce_and_cluster_r pid=3295) AttributeError: 'tqdm_asyncio' object has no attribute 'disable'

---------------------------------------------------------------------------
RayTaskError                              Traceback (most recent call last)
Cell In[44], line 1
----> 1 dataset.cluster_ids()

File ~/Sites/pykanto/pykanto/utils/compute.py:107, in timing.<locals>.wrap(*args, **kwargs)
    104 @wraps(f)
    105 def wrap(*args, **kwargs):
    106     start = time()
--> 107     output = f(*args, **kwargs)
    108     end = time()
    109     from pykanto.dataset import KantoData

File ~/Sites/pykanto/pykanto/dataset.py:768, in KantoData.cluster_ids(self, min_sample)
    751 @timing
    752 def cluster_ids(self, min_sample: int = 10) -> None:
    753     """
    754     Dimensionality reduction using UMAP + unsupervised clustering using
    755     HDBSCAN. This will fail if the sample size for an ID (grouping factor,
   (...)
    766 
    767     """
--> 768     df = reduce_and_cluster_parallel(
    769         self, min_sample=min_sample, num_cpus=self.parameters.num_cpus
    770     )
    772     if self.parameters.song_level:
    773         self.data = self.data.combine_first(df)

File ~/Sites/pykanto/pykanto/signal/cluster.py:290, in reduce_and_cluster_parallel(dataset, min_sample, num_cpus)
    280 obj_ids = [
    281     _reduce_and_cluster_r.remote(
    282         dataset_ref, i, song_level=song_level, min_sample=min_sample
    283     )
    284     for i in chunks
    285 ]
    286 pbar = {
    287     "desc": "Projecting and clustering vocalisations",
    288     "total": n_chunks,
    289 }
--> 290 dfls = [obj_id for obj_id in with_pbar(to_iterator(obj_ids), **pbar)]
    292 # Minimally check output
    293 try:

File ~/Sites/pykanto/pykanto/signal/cluster.py:290, in <listcomp>(.0)
    280 obj_ids = [
    281     _reduce_and_cluster_r.remote(
    282         dataset_ref, i, song_level=song_level, min_sample=min_sample
    283     )
    284     for i in chunks
    285 ]
    286 pbar = {
    287     "desc": "Projecting and clustering vocalisations",
    288     "total": n_chunks,
    289 }
--> 290 dfls = [obj_id for obj_id in with_pbar(to_iterator(obj_ids), **pbar)]
    292 # Minimally check output
    293 try:

File ~/Sites/miniconda3/lib/python3.9/site-packages/tqdm/notebook.py:259, in tqdm_notebook.__iter__(self)
    257 try:
    258     it = super(tqdm_notebook, self).__iter__()
--> 259     for obj in it:
    260         # return super(tqdm...) will not catch exception
    261         yield obj
    262 # NB: except ... [ as ...] breaks IPython async KeyboardInterrupt

File ~/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py:1195, in tqdm.__iter__(self)
   1192 time = self._time
   1194 try:
-> 1195     for obj in iterable:
   1196         yield obj
   1197         # Update and possibly print the progressbar.
   1198         # Note: does not call self.update(1) for speed optimisation.

File ~/Sites/pykanto/pykanto/utils/compute.py:39, in to_iterator(obj_ids, breaks)
     37 done, obj_ids = ray.wait(obj_ids)
     38 if breaks:
---> 39     yield ray.get(done[0])
     40 else:
     41     try:

File ~/Sites/miniconda3/lib/python3.9/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    103     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    104         return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File ~/Sites/miniconda3/lib/python3.9/site-packages/ray/_private/worker.py:2380, in get(object_refs, timeout)
   2378     worker.core_worker.dump_object_store_memory_usage()
   2379 if isinstance(value, RayTaskError):
-> 2380     raise value.as_instanceof_cause()
   2381 else:
   2382     raise value

RayTaskError: ray::_reduce_and_cluster_r() (pid=3295, ip=127.0.0.1)
  File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 273, in _reduce_and_cluster_r
    return [
  File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 274, in <listcomp>
    reduce_and_cluster(
  File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 211, in reduce_and_cluster
    embedding, _ = umap_reduce(
  File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 94, in umap_reduce
    embedding = reducer.fit_transform(data)
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 2772, in fit_transform
    self.fit(X, y)
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 2684, in fit
    self.embedding_, aux_data = self._fit_embed_data(
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 2717, in _fit_embed_data
    return simplicial_set_embedding(
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 1156, in simplicial_set_embedding
    embedding = optimize_layout_euclidean(
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/layouts.py", line 338, in optimize_layout_euclidean
    for n in tqdm(range(n_epochs), **tqdm_kwds):
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 562, in __new__
    with cls.get_lock():  # also constructs lock if non-existent
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 661, in get_lock
    cls._lock = TqdmDefaultWriteLock()
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 99, in __init__
    cls.create_mp_lock()
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 123, in create_mp_lock
    cls.mp_lock = RLock()
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/context.py", line 73, in RLock
    return RLock(ctx=self.get_context())
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/synchronize.py", line 187, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/synchronize.py", line 80, in __init__
    register(self._semlock.name, "semaphore")
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/resource_tracker.py", line 147, in register
    self._send('REGISTER', name, rtype)
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/resource_tracker.py", line 154, in _send
    self.ensure_running()
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/resource_tracker.py", line 124, in ensure_running
    signal.pthread_sigmask(signal.SIG_UNBLOCK, _IGNORED_SIGNALS)
  File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/signal.py", line 69, in pthread_sigmask
    sigs_set = _signal.pthread_sigmask(how, mask)
ray.exceptions.TaskCancelledError: Task: TaskID(c7528efcb2fd36edffffffffffffffffffffffff01000000) was cancelled

[nilomr]

Hi @gg4u, many thanks for reporting this issue! I haven't been able to reproduce this in macOS - tests are passing. I will look further into it when I have some time, and in the meantime maybe someone else will run into the same.

[gg4u]

Pleased to have give a little help, @nilomr !