Trouble using Documentstores other than elastic #1368

advaitdeshmukh · 2021-08-24T06:16:14Z

advaitdeshmukh
Aug 24, 2021

I am trying to get the docker compose demo to work dense retrieval(which worked for my usecase on colab). The poblem is when I run the docker compose file with only haystack_ui and haystack_api images, it gets into a weird loop where haystack_api worker keeps timing out and it runs all over again. I did change the necessary yaml pipeline file but being a complete newbie to docker, I'm probably making some major mistake. Attaching my docker-compse.yaml and pipelines.yaml below.

docker-compose.yml.txt
pipelines.yaml.txt

Answered by oryx1729

Sep 10, 2021

Hi @advaitdeshmukh, I'm closing this thread. Let's continue the discussion on #1411.

View full answer

oryx1729 · 2021-08-31T13:19:34Z

oryx1729
Aug 31, 2021
Collaborator

Hi @advaitdeshmukh, it seems that the docker image for the API is not using your custom pipelines.yaml.

You can replace rest_api/pipeline/pipelines.yaml with your own file. This custom file should then be mounted on the docker image. You can use the following docker-compose.yml:

version: "3"
services:
  haystack-api:
    build:
      context: .
      dockerfile: Dockerfile
    image: "deepset/haystack-cpu:latest"
    pull_policy: always
    volumes:
      - ./rest_api/pipeline:/home/user/rest_api/pipeline
    ports:
      - 8000:8000
    restart: always
    command: "/bin/bash -c 'sleep 15 && gunicorn rest_api.application:app -b 0.0.0.0 -k uvicorn.workers.UvicornWorker --workers 1 --timeout 180'"
  ui:
    build:
      context: ui
      dockerfile: Dockerfile
    ports:
      - 8501:8501
    environment:
      - API_ENDPOINT=http://haystack-api:8000
      - EVAL_FILE=eval_labels_example.csv

If you still face issues, please share the complete logs of docker-compose up.

5 replies

advaitdeshmukh Sep 1, 2021
Author

Hey, thanks for the reply and sorry for the delayed response. This docker-compose definitely uses the new pipelines now. However now I'm running into an issue with the in memory sqlite3 (which i'm guessing faiss is using). The error appears right as i upload my document. adding the logs here. How do you think I proceed here?

haystack-api_1  | [2021-09-01 06:47:50 +0000] [10] [INFO] Application startup complete.
haystack-api_1  | 09/01/2021 06:49:07 - INFO - haystack -   {"request": {"query": "my example query", "filters": null, "top_k_retriever": 3, "top_k_reader": 3}, "response": {"answers": [], "query": "my example query", "node_id": "Reader"}, "time": "1.39"}
haystack-api_1  | huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
haystack-api_1  | To disable this warning, you can either:
haystack-api_1  | 	- Avoid using `tokenizers` before the fork if possible
haystack-api_1  | 	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
100%|██████████| 1/1 [00:00<00:00,  2.11docs/s]
haystack-api_1  | [2021-09-01 06:49:16 +0000] [10] [ERROR] Exception in ASGI application
haystack-api_1  | Traceback (most recent call last):
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
haystack-api_1  |     result = await app(self.scope, self.receive, self.send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
haystack-api_1  |     return await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 208, in __call__
haystack-api_1  |     await super().__call__(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
haystack-api_1  |     await self.middleware_stack(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
haystack-api_1  |     raise exc from None
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
haystack-api_1  |     await self.app(scope, receive, _send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__
haystack-api_1  |     await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
haystack-api_1  |     raise exc from None
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
haystack-api_1  |     await self.app(scope, receive, sender)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__
haystack-api_1  |     await route.handle(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle
haystack-api_1  |     await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app
haystack-api_1  |     response = await func(request)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 227, in app
haystack-api_1  |     dependant=dependant, values=values, is_coroutine=is_coroutine
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 161, in run_endpoint_function
haystack-api_1  |     return await run_in_threadpool(dependant.call, **values)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool
haystack-api_1  |     return await loop.run_in_executor(None, func, *args)
haystack-api_1  |   File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
haystack-api_1  |     result = self.fn(*self.args, **self.kwargs)
haystack-api_1  |   File "/home/user/rest_api/controller/file_upload.py", line 71, in file_upload
haystack-api_1  |     meta=file_metas,
haystack-api_1  |   File "/home/user/haystack/pipeline.py", line 272, in run
haystack-api_1  |     raise Exception(f"Exception while running node `{node_id}` with input `{node_input}`: {e}, full stack trace: {tb}")
haystack-api_1  | Exception: Exception while running node `FAISSDocumentStore` with input `{'documents': [{'text': "my document text", 'meta': {'name': 'doc.pdf', '_split_id': 69}}], 'root_node': 'File', 'remove_whitespace': None, 'remove_empty_lines': None, 'remove_header_footer': None, 'node_id': 'FAISSDocumentStore'}`: (sqlite3.OperationalError) no such table: meta
haystack-api_1  | [SQL: DELETE FROM meta WHERE meta.document_id = ?]
haystack-api_1  | [parameters: ('d594e0f71dafa2c04a2b20f2948b19da',)]
haystack-api_1  | (Background on this error at: https://sqlalche.me/e/14/e3q8), full stack trace: Traceback (most recent call last):
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1772, in _execute_context
haystack-api_1  |     cursor, statement, parameters, context
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
haystack-api_1  |     cursor.execute(statement, parameters)
haystack-api_1  | sqlite3.OperationalError: no such table: meta
haystack-api_1  | 
haystack-api_1  | The above exception was the direct cause of the following exception:
haystack-api_1  | 
haystack-api_1  | Traceback (most recent call last):
haystack-api_1  |   File "/home/user/haystack/pipeline.py", line 269, in run
haystack-api_1  |     node_output, stream_id = self.graph.nodes[node_id]["component"].run(**node_input)
haystack-api_1  |   File "/home/user/haystack/document_store/base.py", line 290, in run
haystack-api_1  |     self.write_documents(documents=documents, index=index)
haystack-api_1  |   File "/home/user/haystack/document_store/faiss.py", line 198, in write_documents
haystack-api_1  |     duplicate_documents=duplicate_documents)
haystack-api_1  |   File "/home/user/haystack/document_store/sql.py", line 313, in write_documents
haystack-api_1  |     self.session.query(MetaORM).filter_by(document_id=doc.id).delete()
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3139, in delete
haystack-api_1  |     execution_options={"synchronize_session": synchronize_session},
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
haystack-api_1  |     result = conn._execute_20(statement, params or {}, execution_options)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20
haystack-api_1  |     return meth(self, args_10style, kwargs_10style, execution_options)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 324, in _execute_on_connection
haystack-api_1  |     self, multiparams, params, execution_options
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1462, in _execute_clauseelement
haystack-api_1  |     cache_hit=cache_hit,
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1815, in _execute_context
haystack-api_1  |     e, statement, parameters, cursor, context
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1996, in _handle_dbapi_exception
haystack-api_1  |     sqlalchemy_exception, with_traceback=exc_info[2], from_=e
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
haystack-api_1  |     raise exception
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1772, in _execute_context
haystack-api_1  |     cursor, statement, parameters, context
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
haystack-api_1  |     cursor.execute(statement, parameters)
haystack-api_1  | sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: meta
haystack-api_1  | [SQL: DELETE FROM meta WHERE meta.document_id = ?]
haystack-api_1  | [parameters: ('d594e0f71dafa2c04a2b20f2948b19da',)]
haystack-api_1  | (Background on this error at: https://sqlalche.me/e/14/e3q8)
haystack-api_1  | 
ui_1            | 2021-09-01 06:49:16.276 Traceback (most recent call last):
ui_1            |   File "/usr/local/lib/python3.7/site-packages/streamlit/script_runner.py", line 350, in _run_script
ui_1            |     exec(code, module.__dict__)
ui_1            |   File "/home/user/webapp.py", line 72, in <module>
ui_1            |     raw_json = upload_doc(data_file)
ui_1            |   File "/home/user/utils.py", line 62, in upload_doc
ui_1            |     response_raw = requests.post(url, files=files).json()
ui_1            |   File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 910, in json
ui_1            |     return complexjson.loads(self.text, **kwargs)
ui_1            |   File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
ui_1            |     return _default_decoder.decode(s)
ui_1            |   File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
ui_1            |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
ui_1            |   File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
ui_1            |     raise JSONDecodeError("Expecting value", s, err.value) from None
ui_1            | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ui_1            |

oryx1729 Sep 2, 2021
Collaborator

Hi @advaitdeshmukh, by default, the FAISSDocumentStore is using in-memory SQLite. Try switching that to a file-based SQLite?

You can set that in the pipelines.yaml.

  - name: FAISSDocumentStore
    type: FAISSDocumentStore
    params:
      faiss_index_factory_str: Flat
      sql_url: sqlite:///test.db

advaitdeshmukh Sep 2, 2021
Author

Hey, based on the similar suggestion here (#1221 (comment)) I've already tried that out. My containers and gui run now but weirdly enough i dont get any responses when i run my query anymore. Will add the screenshot and logs by running that once more. Am I supposed to make any other changes while adding the sql_url that i might be missing out on?
Something that confuses me here is that when i use my python interpreter with a faiss/milvus documentstore it seems to work perfectly(running colab style code). The moment i try to run the api (via docker or uvicorn command) these errors pop up. not sure if that info helps to narrow down on any issue though.

logs:

haystack-api_1  | [2021-09-02 10:03:54 +0000] [24] [INFO] Started server process [24]
haystack-api_1  | [2021-09-02 10:03:54 +0000] [24] [INFO] Waiting for application startup.
haystack-api_1  | [2021-09-02 10:03:54 +0000] [24] [INFO] Application startup complete.
haystack-api_1  | 09/02/2021 10:03:54 - ERROR - farm.data_handler.processor -   There were 1 errors during preprocessing at positions: {0}
haystack-api_1  | [2021-09-02 10:03:54 +0000] [24] [ERROR] Exception in ASGI application
haystack-api_1  | Traceback (most recent call last):
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
haystack-api_1  |     result = await app(self.scope, self.receive, self.send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
haystack-api_1  |     return await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 208, in __call__
haystack-api_1  |     await super().__call__(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
haystack-api_1  |     await self.middleware_stack(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
haystack-api_1  |     raise exc from None
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
haystack-api_1  |     await self.app(scope, receive, _send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__
haystack-api_1  |     await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
haystack-api_1  |     raise exc from None
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
haystack-api_1  |     await self.app(scope, receive, sender)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__
haystack-api_1  |     await route.handle(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle
haystack-api_1  |     await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app
haystack-api_1  |     response = await func(request)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 227, in app
haystack-api_1  |     dependant=dependant, values=values, is_coroutine=is_coroutine
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 161, in run_endpoint_function
haystack-api_1  |     return await run_in_threadpool(dependant.call, **values)
haystack-api_1  |   File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool
haystack-api_1  |     return await loop.run_in_executor(None, func, *args)
haystack-api_1  |   File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
haystack-api_1  |     result = self.fn(*self.args, **self.kwargs)
haystack-api_1  |   File "/home/user/rest_api/controller/search.py", line 55, in query
haystack-api_1  |     result = _process_request(PIPELINE, request)
haystack-api_1  |   File "/home/user/rest_api/controller/search.py", line 75, in _process_request
haystack-api_1  |     top_k_reader=request.top_k_reader)
haystack-api_1  |   File "/home/user/haystack/pipeline.py", line 272, in run
haystack-api_1  |     raise Exception(f"Exception while running node `{node_id}` with input `{node_input}`: {e}, full stack trace: {tb}")
haystack-api_1  | Exception: Exception while running node `DPRetriever` with input `{'root_node': 'Query', 'query': 'How many PGEs were distributed?', 'filters': {}, 'top_k_retriever': 3, 'top_k_reader': 3, 'node_id': 'DPRetriever'}`: object of type 'NoneType' has no len(), full stack trace: Traceback (most recent call last):
haystack-api_1  |   File "/home/user/haystack/pipeline.py", line 269, in run
haystack-api_1  |     node_output, stream_id = self.graph.nodes[node_id]["component"].run(**node_input)
haystack-api_1  |   File "/home/user/haystack/retriever/base.py", line 181, in run
haystack-api_1  |     output, stream = run_query_timed(**kwargs)
haystack-api_1  |   File "/home/user/haystack/retriever/base.py", line 43, in wrapper
haystack-api_1  |     ret = fn(*args, **kwargs)
haystack-api_1  |   File "/home/user/haystack/retriever/base.py", line 198, in run_query
haystack-api_1  |     documents = self.retrieve(query=query, filters=filters, top_k=top_k_retriever, index=index)
haystack-api_1  |   File "/home/user/haystack/retriever/dense.py", line 196, in retrieve
haystack-api_1  |     query_emb = self.embed_queries(texts=[query])
haystack-api_1  |   File "/home/user/haystack/retriever/dense.py", line 262, in embed_queries
haystack-api_1  |     result = self._get_predictions(queries)["query"]
haystack-api_1  |   File "/home/user/haystack/retriever/dense.py", line 229, in _get_predictions
haystack-api_1  |     if len(dataset) == 1:
haystack-api_1  | TypeError: object of type 'NoneType' has no len()
haystack-api_1  | 
haystack-api_1  | 09/02/2021 10:03:56 - INFO - haystack -   {"request": {"query": "How many PGEs were distributed?", "filters": null, "top_k_retriever": 3, "top_k_reader": 3}, "response": {"answers": [], "query": "How many PGEs were distributed?", "node_id": "Reader"}, "time": "2.00"}

The above query is automatically run on boot after streamlit connects to the api. Adding our document and first time manually running the query below:

haystack-api_1  | huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
haystack-api_1  | To disable this warning, you can either:
haystack-api_1  | 	- Avoid using `tokenizers` before the fork if possible
haystack-api_1  | 	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
100%|██████████| 1/1 [00:00<00:00,  1.63docs/s]huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
haystack-api_1  | To disable this warning, you can either:
haystack-api_1  | 	- Avoid using `tokenizers` before the fork if possible
haystack-api_1  | 	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
haystack-api_1  |

As mentioned above, we get no response. What might be the mistake that i am making here?

oryx1729 Sep 2, 2021
Collaborator

Can you verify if you're using a recent version for the docker image of the api & the ui?

You could also try running the API without Docker from the root dir: uvicorn rest_api.application:app & then do a query: curl --request POST --url 'http://127.0.0.1:8000/query' -H "Content-Type: application/json" --data '{"query": "Why is API not working?"}'.

advaitdeshmukh Sep 2, 2021
Author

Hey, My images look up to date to me. Also, the problem persists even when i run the api directly without docker as above. Although now i seem to get empty responses a lot( I still used streamlit to upload the doc).

09/02/2021 18:42:17 - INFO - haystack -   {"request": {"query": "How many PGEs were distributed?", "filters": null, "top_k_retriever": null, "top_k_reader": null}, "response": {"answers": [], "query": "How many PGEs were distributed?", "node_id": "Reader"}, "time": "0.03"}
INFO:     127.0.0.1:58324 - "POST //query HTTP/1.1" 200 OK

oryx1729 · 2021-09-10T10:54:01Z

oryx1729
Sep 10, 2021
Collaborator

Hi @advaitdeshmukh, I'm closing this thread. Let's continue the discussion on #1411.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trouble using Documentstores other than elastic #1368

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Trouble using Documentstores other than elastic #1368

advaitdeshmukh Aug 24, 2021

Replies: 2 comments · 5 replies

oryx1729 Aug 31, 2021 Collaborator

advaitdeshmukh Sep 1, 2021 Author

oryx1729 Sep 2, 2021 Collaborator

advaitdeshmukh Sep 2, 2021 Author

oryx1729 Sep 2, 2021 Collaborator

advaitdeshmukh Sep 2, 2021 Author

oryx1729 Sep 10, 2021 Collaborator

advaitdeshmukh
Aug 24, 2021

Replies: 2 comments 5 replies

oryx1729
Aug 31, 2021
Collaborator

advaitdeshmukh Sep 1, 2021
Author

oryx1729 Sep 2, 2021
Collaborator

advaitdeshmukh Sep 2, 2021
Author

oryx1729 Sep 2, 2021
Collaborator

advaitdeshmukh Sep 2, 2021
Author

oryx1729
Sep 10, 2021
Collaborator