Issue using Unstructured hosted API

[sanjayc2]

Hi,

I am trying to use the Unstructured file converter to read a local pdf file and convert it into Haystack Document. I had originally gotten a free tier from Unstructured, and used docker to use Unstructured locally. This worked fine, until I used up the free part..

Now, Unstructured is not supporting the free tier any longer. So, I'm using the hosted version. I believe the call below is the correct one to make; for the api key, I am using my personal key ID (not the free tier ID!) which I got after creating an Unstructured account. I thought all I needed was the api key but the below code errors out:

converter = UnstructuredFileConverter(api_key = Secret.from_env_var("UNSTRUCTURED_API_KEY", strict=False), progress_bar=True)
docs = converter.run(paths=[path])

When I run this, and provide the api key the error I get is:

Converting files to Haystack Documents: 0it [00:00, ?it/s]INFO: Failed to process a request due to connection error - [Errno 11001] getaddrinfo failed. Attempting retry number 1 after sleep.
ERROR: Failed to partition the document.

I then thought I need to provide the api url in addition to the api key, but I've tried both the Unstructured Workflow Endpoint and Unstructured Partition Endpoint (from my Unstructured account) urls in turn, and the code is still erroring out. In one case I got:

Converting files to Haystack Documents: 0it [00:00, ?it/s]INFO: HTTP Request: GET https://api.unstructuredapp.io/general/docs "HTTP/1.1 200 OK"
INFO: HTTP Request: POST https://api.unstructuredapp.io/general/v0/general "HTTP/1.1 400 Bad Request"
ERROR: Failed to partition set 2.

In the other case, I got:

Converting files to Haystack Documents: 0it [00:00, ?it/s]INFO: HTTP Request: GET https://platform.unstructuredapp.io/general/docs "HTTP/1.1 404 Not Found"
ERROR: Failed to partition the document.
ERROR: Server responded with 404 - {"detail":"Not Found"}

I would really appreciate your help resolving this so that I can use Unstructured. Thanks in advance.

Sanjay

[sjrl]

Hey @sanjayc2 sorry to here you have been having issues using the hosted version of Unstructured.

I'm not too familiar with the self hosted version, but I did want to point you towards this PR deepset-ai/haystack-core-integrations#1416 to show that our unstructured-fileconverter-haystack integration only works with unstructured-client<0.30.0. I'm only bringing this up in case it's relevant for the Docker image you are using to self host unstructured.

Also to better help with debugging your code could you share the full code snippet you run?

So what paths do you send in this line? docs = converter.run(paths=[path])