Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Initial SDK #2

Merged
merged 27 commits into from
Jan 3, 2024
Merged

Initial SDK #2

merged 27 commits into from
Jan 3, 2024

Conversation

maaz-munir
Copy link
Contributor

@maaz-munir maaz-munir commented Nov 23, 2023

This PR contains the SDK for the SERP API.

This is how the UI would look like for someone using the SDK,

image

serp/req.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
serp/defaults.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
serp/valid.go Outdated Show resolved Hide resolved
@maaz-munir
Copy link
Contributor Author

image

This would be the UI for someone using google_search as a source via the SDK.

Since the API returns a different structure of data for the parse parameter I am returning back an interface. Generics dont apply for methods.

Will clean up the response.go file as well later.

Copy link
Contributor

@UmanShahzad UmanShahzad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're getting close!

oxylabs/common.go Outdated Show resolved Hide resolved
oxylabs/common.go Outdated Show resolved Hide resolved
serp/req.go Outdated Show resolved Hide resolved
serp/req.go Outdated Show resolved Hide resolved
serp/response.go Show resolved Hide resolved
serp/bing.go Outdated Show resolved Hide resolved
serp/bing.go Outdated Show resolved Hide resolved
serp/baidu.go Outdated Show resolved Hide resolved
serp/baidu.go Show resolved Hide resolved
serp/baidu.go Outdated Show resolved Hide resolved
@UmanShahzad UmanShahzad changed the title SDK setup Initial SDK Dec 3, 2023
@maaz-munir
Copy link
Contributor Author

  • Changed interface based approach to a single response structure. Had to circumvent via a custom unmarshall function.
  • Added remaining google sources.

Copy link
Contributor

@UmanShahzad UmanShahzad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rapidly release the next portion of this - async handler?

Then the proxy wrapper.

Then I think it's enough for the MVP and showcase!

We can get the above 2 done in this week I'm sure.

serp/yandex_async.go Outdated Show resolved Hide resolved
serp/yandex_async.go Outdated Show resolved Hide resolved
serp/yandex_async.go Outdated Show resolved Hide resolved
serp/yandex_async.go Outdated Show resolved Hide resolved
serp/yandex_async.go Outdated Show resolved Hide resolved
@maaz-munir
Copy link
Contributor Author

  • Improved error handling by using a channel for the error within the go-func
  • Added two funcs for google
  • There seems to be a problem where if you pass in Parse= true it results in the job having the status faulted. I tried this via postman with the push pull method and it has the same problem. It could be a problem with the async api, I will check it again later.

@UmanShahzad
Copy link
Contributor

FYI @Ryouku

@maaz-munir
Copy link
Contributor Author

A preview of what the UI would look like for someone using the proxy endpoint for the serp api
image

Copy link
Contributor

@UmanShahzad UmanShahzad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good direction!

Let's fix the remaining stuff, add the README.md and docs on top of the funcs, and we're good for an MVP launch possibly. I'll give it a spin myself near the end.

serp/async_helpers.go Outdated Show resolved Hide resolved
serp/client.go Outdated Show resolved Hide resolved
serp/proxy_endpint.go Outdated Show resolved Hide resolved
Copy link
Contributor

@UmanShahzad UmanShahzad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment needs to be on top of the actual function, one for each, following Golang styles for comments.

proxy/proxy_helpers.go Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
Copy link
Contributor

@UmanShahzad UmanShahzad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maaz-munir is this ready for me to do an end-to-end review from a developer's perspective?

@UmanShahzad UmanShahzad merged commit fea1101 into master Jan 3, 2024
@UmanShahzad UmanShahzad deleted the sdk-setup branch January 3, 2024 21:56
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants