bug: WebHDFS's url parameter should be changed (delegation_token -> delegation)

[jason-heo]

Describe the bug

Hello.

This issue was originally reports to databendlabs/databend#16742

But I think this repository is better place to resolve it.

When I run INSERT INTO in bendsql, OpenDAL requests WebHDFS url like this:

https://<hostname>:<port>/webhdfs/v1/path/to/xxx.parquet?op=CREATE&...&delegation_token=<token>

I think delegation_token should be renamed to delegation

Above url gives 401 error. But When I test &delegation=<token> with curl, it worked well.

You can see not delegation_token but delegation on WebHDFS REST API -> Authentication

I've tested using curl and confirm that following url worked well.

$ curl -i "https://<hostname>:<port>/webhdfs/v1/path/to?op=LISTSTATUS&delegation=<token>"

Steps to Reproduce

Sorry, I couldn't write OpenDAL source code in Rust.

Expected Behavior

url parameter delegation_token should be renamed to delegation

Additional Context

No response

Are you willing to submit a PR to fix this bug?

• Yes, I would like to submit a PR.

[Xuanwo]

Hi, @jason-heo, thank you for this report. The name here is following the naming pattern with webhdfs API: https://hadoop.apache.org/docs/r3.3.2/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Delegation_Token_Operations

Using delegation_token here makes it easier to differentiate from the future addition of oauth2_access_token.

[jason-heo]

@Xuanwo

Hello, Xuanwo

Thank you for your reply :)

The page on Delegation Token Operations explains how to obtain a delegation token.

One we have thetoken, I think it should be passed as the delgation parameter in URL.

curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?delegation=<TOKEN>&op=..."

This is shonw in the Authentication.

Please let me know I have misunderstood you.

Thanks.

[Xuanwo]

Hi, @jason-heo. The parameter is not a one-to-one mapping from OpenDAL to the underlying storage services. I personally find delegation_token clear and easy to use.

I understand that using URL format can be a bit confusing; perhaps we could add some compatibility tricks on the Databend side.

I will take a look over databendlabs/databend#16742 and implement this change for you.

[jason-heo]

The parameter is not a one-to-one mapping from OpenDAL to the underlying storage services

Oh, sorry. I didn't know that.

Saw your comment on Databend issue.

Let me close this issue.

Thank you @Xuanwo