Feature request: Apache Drill connector without S3

[julioasotodv]

Hi,

First of all, thank you all for Falcon. It's awesome.

However, it looks like the Apache Drill connector only works with S3 buckets. Nonetheless, Drill itself is able to query from lots of other places (such as a local filesystem, HDFS, MongoDB...).

Adding generic support for Apache Drill (without the need for S3 credentials) would be great. It would expand Falcon capabilities a lot with just one connector.

Thank you!

[n-riesco]

@julioasotodv sorry for the late reply. At the moment, the connector for Apache Drill doesn't implement some the functionality other SQL connectors have in Falcon.

I've had a quick look at the documentation, and the missing functionality is readily available in Apache Drill, so this is an easy first-time contribution.

If no one has a go at implementing the missing functionality, I'll do it myself next month.

[julioasotodv]

@n-riesco TBH I'm not that familiar with how Falcon works itself (I do not have much spare time neither...)

I will try to poke around a little bit

[n-riesco]

@julioasotodv Great! If you need any help, please, post here.

The main change is to add the functions tables and schemas to https://github.com/plotly/falcon-sql-client/blob/master/backend/persistent/datastores/ApacheDrill.js and update SQL_DIALECTS_USING_EDITOR and PREVIEW_QUERY in https://github.com/plotly/falcon-sql-client/blob/master/app/constants/constants.js .

We've got some documentation on implementing a connector in https://github.com/plotly/falcon-sql-client/blob/master/NEW_CONNECTION.md .

And see the instructions to build Falcon in https://github.com/plotly/falcon-sql-client/blob/master/CONTRIBUTING.md ; basically:

$ git clone https://github.com/plotly/falcon-sql-client falcon-sql-client
$ cd falcon-sql-client
$ yarn install
$ yarn run rebuild:modules:electron
$ yarn run build
$ yarn run start

---

If, at some point, you don't find the time to complete a PR, please, let us know, so that someone else can continue the work.

Thank you for doing this!

[shannonlal]

@julioasotodv If you have any issues with setting up the environment please let me know. I can jump a skype call and give you a hand

[n-riesco]

@julioasotodv did you have a chance work on this feature? If you haven't, @kndungu is interested in implementing it.

[julioasotodv]

Hi, unfortunately I did not have any time at all :(

Also, I found out that my js is extremely rusty 😅

So yeah, @kndungu can go ahead anytime :)

[sidd1990]

Hi all,

I am interested to contribute. I have a similar experience of developing the connector. But, I am new to open source contribution. Need help!

[shannonlal]

@sidd1990 Which connector are you interested in adding. I can help you with the connector and UI part and walk you through the integration