Error on running on parallel

[jrpereirajr]

Hi!

I have set up a Proof of Concept (PoC) that calls the dbt run command after ingesting some data. The ingestion process is executed using an IRIS Interoperability Production. Since this production invokes its hosts asynchronously, multiple dbt run commands are issued concurrently.

To provide a visual representation of this production, please refer to the following image:

In order to increase throughput, I have adjusted the pool size of the host that issues the dbt run command to allow concurrent calls. However, after making this adjustment, dbt-iris started logging error messages like the following:

<Table 'customer1_dbt.stg__ordem_servico__dbt_tmp' already exists>

I think the cause is dbt-iris always try to create a new table with the same name, because there are many dbt process running in parallel due serveral interoperability BOs running in parallel as well.

I would like to know if this indicates a misuse of dbt and dbt-iris or if there is an underlying issue.

Thanks!

[bdeboe]

Hi @jrpereirajr !

can you share a little more about the transformations envisaged by your dbt project?
Is it correct that that table is meant to be specific to your process, and should not be "seen" by other dbt run sessions that might be running concurrently?

[jrpereirajr]

Hi Benjamin!

I didn't created that table. I think it was created by the dbt-iris plugin because I found this file in the directory target/run:

/* create_view_as */
create table "customer1_dbt"."stg_ordem_servico__dbt_tmp" as
    
with

source as (
    select 
        OriginalId,
...

Note that the name of the table is the same presented in the error message - customer1_dbt.stg_ordem_servico__dbt_tmp.

This is the model SQL file that generated the statement mentioned early:

{{
    config(
        materialized='view'
    )
}}

with

source as (
    select 
        OriginalId,
        SessionId,
        Codigo
    from {{source('shift_lis', 'Order')}}
),

renamed as (
    select
        OriginalId as EntityId,
        SessionId,
        Codigo
        current_timestamp(3) as LastModified
    from source
)

select * from renamed

Thank you!

[bdeboe]

Hi,

it looks like running multiple jobs concurrently may not be an intended pattern, at least from (not finding much in the dbt docs) this thread: dbt-labs/dbt-core#2881
At the bottom there is a reference to how the Snowflake adapter deals with the issue by supporting the invocation ID as a suffix for naming these tables, as part of a macro. I wonder if that is something we could consider for the IRIS one as well @daimor ?

[daimor]

Yes, I think we can make this too

[fernandolferreira]

Hi @daimor , do you have any news about this issue?

[daimor]

Sorry, I did not have a chance to look yet.

[jrpereirajr]

Hi guys! Do you have any updates?

[daimor]

I've released a new version, also with upgrade to the latest dbt
please check it

[jrpereirajr]

Hi @daimor.

Thank you for your response.

After installing version v1.8.17, bash was not able to find dbt command. By issuing a dbt command, I've getting this error:
bash: dbt: command not found

This was I used to test it into a container:

ARG IMAGE=containers.intersystems.com/intersystems/irishealth-ml:2023.1.4.580.0
FROM $IMAGE

USER root

RUN apt-get update
RUN apt-get install -y python3.10-venv nano curl

COPY requirements.txt ./

RUN pip install -r ./requirements.txt
...

requirements.txt:

dbt-iris
numpy

After downgrading to version v1.5.7, it went back to work:

dbt-iris==1.5.7
numpy

Did I miss something?

[daimor]

dbt-core, in the latest version not a functional requirement for dbt-iris itself
and it can be installed separately (and I think supposed to be installed this way)
I'll update package to have it in dependencies for compatibility

[jrpereirajr]

Hi @daimor.

Thank you for installing instructions. By installing dbt-core first and then dbt-iris I was able to run dbt again.

However, after installing dbt 1.8.18 I'm now getting the following error on concurrent executions:

As you can see, I opened two shell sessions and tried to execute the same dbt run command at the same time, but got a <Table or view not found> error.

[bdeboe]

any chance you'd be able to show more of the generated SQL? You can use dbt compile to see those in case you weren't already tracking them

[jrpereirajr]

Hi @bdeboe,

Sorry for the delay, I was on PTO.
Are you looking for the content created within target folder? If so, have you a secure way to send it for privacy reasons?

[bdeboe]

I think it would be helpful to verify if the generated SQL uses different table names for both invocations, as I believe @daimor intended with his change. If not, we need to look into the model file to see if perhaps the materialization option isn't what the code expects it to be (could well be the code that needs to change, of course).
And in case it does use different table names, we should check which table is being dropped earlier than expected and triggering that error.

[jrpereirajr]

Hi @bdeboe

I sent you the files into your e-mail.

Thanks!