Connection to AWS DynamoDB

KNIME Extensions
1

Is there a node to scan or query AWS DynamoDB? Or any workaround that allow query to DynamoDB?

2

Hello @anguslou,
currently we do not yet have DynamoDB nodes, but they are already in the works! I think with the next feature release 4.2 in July they will be available. In the meantime, you may be able to use the Simba JDBC drivers, but they are not free.
Kind regards
Alexander

3 Likes
3

That is fantastic. Thank you.

4

Hi,
I just wanted to let you know that the DynamoDB nodes have now been released in KNIME Labs: https://kni.me/e/2gQk2-iaVv_3CDPv. Please give them a try and let us know if you have feedback!
Kind regards,
Alexander

2 Likes
5

Looking good. Will feedback for any problems.

6

I got an error message when trying to open 01 Data Access/ 06 Zip and Remote Files/ 06 Amazon S3 meets DynamoDB:

Editor initialization failed: org.knime.workbench.editor.WorkflowEditor. Site is incorrect.

I tried to use these nodes myself, working well for Describe Table, Batch Get, Query, etc. But cannot figure out DynamoDB Scan. Don’t know how to add to Filter & Projection.

7

Hi @anguslou,
You can find out more about filter and projection expressions here. There is documentation for projection expressions and a filter expression is a condition expression.
Kind regards,
Alexander

1 Like
8

Thanks Alexander,

I understand dynamoDB expressions, but I don’t know how to put it into Amazon DynamoDB Scan Node. I try to press the buttons there but the placeholder or value is not updated into the node. Any document or video can help me to there?

9

Hi,
ah, I think I know what the problem is. By default there is already an empty line in the list of name and value mapping lists. It is just not really visible. You can click at the top of the list to edit them. With the “-” button you can remove a mapping. The placeholders can then be used in your projection or filter expression.
Kind regards,
Alexander

10

Eventually I get there. So far all the DynamoDB nodes work perfectly as expected.

11

Hi @anguslou,
Thanks for the feedback and apologies for the slightly unintuitive user interface. I think if the table with the mappings had lines between the cells, it might be more obvious where to enter the values. Maybe this can be fixed in the future.
Kind regards,
Alexander

12

It will help much if the 06_Amazon_S3_meets_DynamoDB at EXAMPLE (knime@hub.knime.com) works. But I cannot run it and get an error message:

Execute failed: Directory ‘/knime-wf-hacking-days/newFolder/’ of output file does not exist

The Amazon Authentication and Amazon S3 Connector (Labs) nodes are “green-light”, but others are either yellow or red. Amazon DynamoDB Create Table node is even execution failed: Request is missing Authentication token.

13

I’d changed to my Amazon Authentication and revised the S3 path, then it works well. Thank you.

2 Likes
14

Hi Alexander,
I have been using Knime to connect to AWS for some weeks now without a problem. recently the data in one of the tables exceeded the 1MB, and as explained in the description of the “dynamodb scan” it will not retrieve more than 1MB and does not provide the “lastevaluatedkey” .

my questions, are there any suggestion tricks to get the full results back? by using this node or other nodes? Thanks in advance.

Bashar

15

Hello Bashar,
The node should retrieve more than 1MB. The thing with the 1MB batches is that if you have a table attribute (i.e. column) that does not occur at all in the first 1MB batch, then the attribute will be ignored in all future batches. Do you see the behavior that KNIME ignores everything after 1MB? Because that would be a bug.
Kind regards,
Alexander

16

Hello Alexander, Thank you for your response!
I resolved this by specifying the column names I need in the “Projections” and it is resolved now. I believe it had to do with an attribute that does not occur in all records. so far I am getting the data I need without any problem. if this issue arises again I shall drop another note here!
Thanks again,
Bashar

2 Likes