Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

remove s3 output location requirement from hub class init #5081

Merged
merged 6 commits into from
Mar 12, 2025
Merged

remove s3 output location requirement from hub class init #5081

merged 6 commits into from
Mar 12, 2025

Conversation

bencrabtree
Copy link
Collaborator

Issue #, if available:

Description of changes:

Removes the requirement that Hubs had to have the S3OutputPath field set in order to use the class interface. Hubs created outside of this class would usually not have that field, causing customers to error out when using

Testing done:

Removes a bunch of unused functions, and adds unit tests for this functionality

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)
  • If adding any dependency in requirements.txt files, I have spell checked and ensured they exist in PyPi

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@bencrabtree bencrabtree requested a review from a team as a code owner March 10, 2025 16:35
@bencrabtree bencrabtree requested a review from nileshvd March 10, 2025 16:35
evakravi
evakravi reviewed Mar 10, 2025
View reviewed changes
src/sagemaker/jumpstart/hub/hub.py Outdated

return self._sagemaker_session.create_hub(
hub_name=self.hub_name,
hub_description=description,
hub_display_name=display_name,
hub_search_keywords=search_keywords,
s3_storage_config={"S3OutputPath": self.hub_storage_location.get_uri()},
s3_storage_config={
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we make a dedicated function to get the s3 output path?

Copy link
Collaborator Author

@bencrabtree bencrabtree Mar 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could, though I don't think it's necessary as this logic is simple. I unit test both paths

evakravi
evakravi previously approved these changes Mar 10, 2025
View reviewed changes
@nargokul nargokul merged commit a282892 into aws:master Mar 12, 2025
14 checks passed
mollyheamazon pushed a commit to mollyheamazon/sagemaker-python-sdk that referenced this pull request Mar 14, 2025
@bencrabtree @nargokul @mollyheamazon
remove s3 output location requirement from hub class init (aws#5081)
5e8e894 
* remove s3 output location requirement from hub class init

* fix integ test hub

* lint

* fix test

---------

Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com>
evakravi pushed a commit to evakravi/sagemaker-python-sdk that referenced this pull request Mar 20, 2025
@bencrabtree @nargokul @sagemaker-bot
remove s3 output location requirement from hub class init (aws#5081)
6408c0c 
* remove s3 output location requirement from hub class init

* fix integ test hub

* lint

* fix test

---------

Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com>
mollyheamazon pushed a commit to mollyheamazon/sagemaker-python-sdk that referenced this pull request Apr 8, 2025
@bencrabtree @nargokul @mollyheamazon
remove s3 output location requirement from hub class init (aws#5081)
3a55a15 
* remove s3 output location requirement from hub class init

* fix integ test hub

* lint

* fix test

---------

Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com>
mollyheamazon added a commit to mollyheamazon/sagemaker-python-sdk that referenced this pull request Apr 8, 2025
@mollyheamazon
parent fb22b91
4495904 
author Molly He <mollyhe@amazon.com> 1740529460 -0800
committer Molly He <mollyhe@amazon.com> 1744156034 -0700

parent fb22b91
author Molly He <mollyhe@amazon.com> 1740529460 -0800
committer Molly He <mollyhe@amazon.com> 1744155953 -0700

parent fb22b91
author Molly He <mollyhe@amazon.com> 1740529460 -0800
committer Molly He <mollyhe@amazon.com> 1744155703 -0700

parent fb22b91
author Molly He <mollyhe@amazon.com> 1740529460 -0800
committer Molly He <mollyhe@amazon.com> 1744155683 -0700

Test py312 ci support

Test py312 ci support

first commit

test to point to personal stack

changed tox.ini

Revert "changed tox.ini"

This reverts commit 696cb47.

Revert "Revert "changed tox.ini""

This reverts commit 5d35bd0.

Revert "Revert "Revert "changed tox.ini"""

This reverts commit 2ab95fb.

Revert "Revert "changed tox.ini""

This reverts commit 5d35bd0.

Revert "changed tox.ini"

This reverts commit 696cb47.

Revert "test to point to personal stack"

This reverts commit 49de844.

Revert "first commit"

This reverts commit 1374184.

add pyproject.toml

add py312 to ci

bump numpy version

numpy version change

add py312

upgrade pip version

add setuptools wheel to tox.ini

add pyyaml version constraint, remove py312 from docstring because there is no py312 image yet

update pyyaml version constraint

update pyyaml version constraint

deprecate py38

bump scipy

bump tensorflow and tensorboard

bump dill

bump apache-airflow to ensure dill and greenlet version

remove constraint for apache-airflow

remove constraint for apache-airflow

bump torch version

bump torchvision version

new tests

try changing some tests

update model trainer test

fix test_pipeline

add constraint to apache in tox

Fix key error in _send_metrics() (aws#5068)

Co-authored-by: pintaoz <pintaoz@amazon.com>

Use sagemaker session's s3_resource in download_folder (aws#5064)

Co-authored-by: pintaoz <pintaoz@amazon.com>

Fix error when there is no session to call _create_model_request() (aws#5062)

* Fix error when there is no session to call _create_model_request()

* Fix codestyle

---------

Co-authored-by: pintaoz <pintaoz@amazon.com>

Ensure Model.is_repack() returns a boolean (aws#5060)

* Ensure Model.is_repack() returns a boolean

* update test

---------

Co-authored-by: pintaoz <pintaoz@amazon.com>

feat: Make DistributedConfig Extensible (aws#5039)

* feat: Make DistributedConfig Extensible

* pylint

* Include none types when creating config jsons for safer reference

* fix: update test to account for changes

* format

* Add integ test

* pylint

* prepare release v2.240.0

* update development version to v2.240.1.dev0

* Fix key error in _send_metrics() (aws#5068)

Co-authored-by: pintaoz <pintaoz@amazon.com>

* fix: Added check for the presence of model package group before creating one (aws#5063)

Co-authored-by: Keshav Chandak <chakesh@amazon.com>

* Use sagemaker session's s3_resource in download_folder (aws#5064)

Co-authored-by: pintaoz <pintaoz@amazon.com>

* remove union

* fix merge artifact

* Change dir path to distributed_drivers

* update paths

---------

Co-authored-by: ci <ci>
Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com>
Co-authored-by: pintaoz <pintaoz@amazon.com>
Co-authored-by: Keshav Chandak <keshav.chandak1995@gmail.com>
Co-authored-by: Keshav Chandak <chakesh@amazon.com>

Skip tests with deprecated instance type (aws#5077)

Co-authored-by: pintaoz <pintaoz@amazon.com>

pipeline definition function doc update (aws#5074)

Co-authored-by: Rohan Gujarathi <gujrohan@amazon.com>

feat: add integ tests for training JumpStart models in private hub (aws#5076)

* feat: add integ tests for training JumpStart models in private hub

* fixed formatting

* remove unused imports

* fix unused imports

* fix unit test failure and fix bug around versioning

* fix formatting

* fix unit tests

* fix model_uri usage issue

* fix some formatting

* separate private hub setup code

* add try catch block

* fix flake8 issue so except clause is not bare

* black formatting

fix: resolve infinite loop in _find_config on Windows systems (aws#4970)

* fix: resolve Windows path handling in _find_config

* Replace Path.match("/") with Path.anchor comparison
* Fix infinite loop in _studio.py path traversal

* test: Add tests for the new root path exploration

* Fix formatting style

* Fixed line to long

* Fix docstyle by running black manually

* Fix testcase with \\ when running on non-windows machines

* Fix formatting style

* cleanup unused import

change: update image_uri_configs  03-11-2025 07:18:09 PST

Fixing Pytorch training python version in tests (aws#5084)

* Fixing Pytorch training python version in tests

* Updating Inference test handling

remove s3 output location requirement from hub class init (aws#5081)

* remove s3 output location requirement from hub class init

* fix integ test hub

* lint

* fix test

---------

Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com>

fix: Prevent RunContext overlap between test_run tests (aws#5083)

Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com>

remove py38 from unit testing

fix integ test by bumping py38 to py39 for PyTorch

change framework_version that supports py39 in integ tests

remove py38 from unit testing

Update estimator.py
@mollyheamazon mollyheamazon mentioned this pull request Apr 9, 2025
Open
10 tasks
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@malav-shastri malav-shastri malav-shastri approved these changes

@nargokul nargokul nargokul approved these changes

@Narrohag Narrohag Narrohag approved these changes

@evakravi evakravi evakravi left review comments

@nileshvd nileshvd Awaiting requested review from nileshvd nileshvd is a code owner automatically assigned from aws/sagemaker-ml-frameworks

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@bencrabtree @malav-shastri @evakravi @nargokul @Narrohag