Skip to content

Commit

Permalink
Multi: Updates to SageMaker workflow READMEs and metadata (awsdocs#5389)
Browse files Browse the repository at this point in the history
  • Loading branch information
rlhagerm authored Sep 21, 2023
1 parent eb04ae4 commit b4c74eb
Show file tree
Hide file tree
Showing 16 changed files with 478 additions and 417 deletions.
3 changes: 3 additions & 0 deletions .doc_gen/metadata/sagemaker_metadata.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,9 @@ sagemaker_Scenario_GettingStarted:
sagemaker_Scenario_Pipelines:
title: Get started with &SM; geospatial jobs in a pipeline using an &AWS; SDK
title_abbrev: Get started with geospatial jobs and pipelines
guide_topic:
title: Create and run SageMaker pipelines using &AWS; SDKs on Community.aws
url: https://community.aws/posts/create-and-run-sagemaker-pipelines-using-aws-sdks
synopsis_list:
- Set up resources for a pipeline.
- Set up a pipeline that executes a geospatial job.
Expand Down
2 changes: 0 additions & 2 deletions .github/pre_validate/pre_validate.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,6 @@
'sample_cert.pem',
'sample_private_key.pem',
'sample_saml_metadata.xml',
'GeoSpatialPipeline.json',
'latlongtest.csv',
}

# Media file types.
Expand Down
49 changes: 33 additions & 16 deletions dotnetv3/SageMaker/Scenarios/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,49 @@

This scenario demonstrates how to work with Amazon SageMaker pipelines and geospatial jobs.

### Amazon SageMaker Pipelines
A [SageMaker pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines.html) is a series of
interconnected steps that can be used to automate machine learning workflows. You can create and run pipelines from SageMaker Studio by using Python, but you can also do this by using AWS SDKs in other
interconnected steps that can be used to automate machine learning workflows. Pipelines use interconnected steps and shared parameters to support repeatable workflows that can be customized for your specific use case. You can create and run pipelines from SageMaker Studio using Python, but you can also do this by using AWS SDKs in other
languages. Using the SDKs, you can create and run SageMaker pipelines and also monitor operations for them.

### Pipeline steps
This example pipeline includes an [AWS Lambda step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda)
and a [callback step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback).
Both steps are processed by the same example Lambda function.
### Explore the scenario
This example scenario demonstrates using AWS Lambda and Amazon Simple Queue Service (Amazon SQS) as part of an Amazon SageMaker pipeline. The pipeline itself executes a geospatial job to reverse geocode a sample set of coordinates into human-readable addresses. Input and output files are located in an Amazon Simple Storage Service (Amazon S3) bucket.

This Lambda code is included as part of this example, with the following functionality:
- Starts the SageMaker Vector Enrichment Job with the provided job configuration.
![Workflow image](../../../workflows/sagemaker_pipelines/resources/workflow.png)

When you run the example console application, you can execute the following steps:

- Create the AWS resources and roles needed for the pipeline.
- Create the AWS Lambda function.
- Create the SageMaker pipeline.
- Upload an input file into an S3 bucket.
- Execute the pipeline and monitor its status.
- Display some output from the output file.
- Clean up the pipeline resources.

#### Pipeline steps
[Pipeline steps](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html) define the actions and relationships of the pipeline operations. The pipeline in this example includes an [AWS Lambda step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda)
and a [callback step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback).
Both steps are processed by the same example Lambda function.

The Lambda function handler is included as part of the example, with the following functionality:
- Starts a [SageMaker Vector Enrichment Job](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) with the provided job configuration.
- Processes Amazon SQS queue messages from the SageMaker pipeline.
- Starts the export function with the provided export configuration.
- Processes Amazon SQS queue messages from the SageMaker pipeline.
- Completes the pipeline when the export is complete.

![Pipeline image](../Images/Pipeline.PNG)
![Pipeline image](../../../workflows/sagemaker_pipelines/resources/pipeline.png)

### Pipeline parameters
#### Pipeline parameters
The example pipeline uses [parameters](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html) that you can reference throughout the steps. You can also use the parameters to change
values between runs. In this example, the parameters are used to set the Amazon Simple Storage Service (Amazon S3)
locations for the input and output files, along with the identifiers for the role and queue to use in the pipeline.
The example demonstrates how to set and access these parameters.
values between runs and control the input and output setting. In this example, the parameters are used to set the Amazon Simple Storage Service (Amazon S3)
locations for the input and output files, along with the identifiers for the role and queue to use in the pipeline.
The example demonstrates how to set and access these parameters before executing the pipeline using an SDK.

### Geospatial jobs
#### Geospatial jobs
A SageMaker pipeline can be used for model training, setup, testing, or validation. This example uses a simple job
for demonstration purposes: a [Vector Enrichment Job (VEJ)](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) that processes a set of coordinates to produce human-readable
addresses powered by Amazon Location Service. Other types of jobs could be substituted in the pipeline instead.
for demonstration purposes: a [Vector Enrichment Job (VEJ)](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) that processes a set of coordinates to produce human-readable
addresses powered by Amazon Location Service. Other types of jobs can be substituted in the pipeline instead.

## ⚠ Important

Expand Down
9 changes: 9 additions & 0 deletions dotnetv3/SageMaker/Scenarios/SageMakerScenario.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
</PropertyGroup>

<ItemGroup>
<Content Include="..\..\..\resources\sample_files\GeoSpatialPipeline.json" Link="GeoSpatialPipeline.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
<Content Include="settings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
Expand All @@ -21,6 +24,12 @@
<None Remove="settings.local.json" />
</ItemGroup>

<ItemGroup>
<None Include="..\..\..\resources\sample_files\latlongtest.csv" Link="latlongtest.csv">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>

<ItemGroup>
<PackageReference Include="AWSSDK.EC2" Version="3.7.137.1" />
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.2" />
Expand Down
54 changes: 35 additions & 19 deletions javav2/usecases/workflow_sagemaker_pipes/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,49 @@

This scenario demonstrates how to work with Amazon SageMaker pipelines and geospatial jobs.

A [SageMaker pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines.html) is a series of
interconnected steps that can be used to automate machine learning workflows. You can create and run pipelines from SageMaker Studio by using Python, but you can also do this by using AWS SDKs in other
### Amazon SageMaker Pipelines
A [SageMaker pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines.html) is a series of
interconnected steps that can be used to automate machine learning workflows. Pipelines use interconnected steps and shared parameters to support repeatable workflows that can be customized for your specific use case. You can create and run pipelines from SageMaker Studio using Python, but you can also do this by using AWS SDKs in other
languages. Using the SDKs, you can create and run SageMaker pipelines and also monitor operations for them.

### Pipeline steps
This example pipeline includes an [AWS Lambda step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda)
and a [callback step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback).
Both steps are processed by the same example Lambda function.
### Explore the scenario
This example scenario demonstrates using AWS Lambda and Amazon Simple Queue Service (Amazon SQS) as part of an Amazon SageMaker pipeline. The pipeline itself executes a geospatial job to reverse geocode a sample set of coordinates into human-readable addresses. Input and output files are located in an Amazon Simple Storage Service (Amazon S3) bucket.

This Lambda code is included as part of this example, with the following functionality:
- Starts the SageMaker Vector Enrichment Job with the provided job configuration.
![Workflow image](../../../workflows/sagemaker_pipelines/resources/workflow.png)

When you run the example console application, you can execute the following steps:

- Create the AWS resources and roles needed for the pipeline.
- Create the AWS Lambda function.
- Create the SageMaker pipeline.
- Upload an input file into an S3 bucket.
- Execute the pipeline and monitor its status.
- Display some output from the output file.
- Clean up the pipeline resources.

#### Pipeline steps
[Pipeline steps](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html) define the actions and relationships of the pipeline operations. The pipeline in this example includes an [AWS Lambda step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda)
and a [callback step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback).
Both steps are processed by the same example Lambda function.

The Lambda function handler is included as part of the example, with the following functionality:
- Starts a [SageMaker Vector Enrichment Job](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) with the provided job configuration.
- Processes Amazon SQS queue messages from the SageMaker pipeline.
- Starts the export function with the provided export configuration.
- Processes Amazon Simple Queue Service (Amazon SQS) messages from the SageMaker pipeline.
- Completes the pipeline when the export is complete.

![AWS Tracking Application](images/pipes.png)
![Pipeline image](../../../workflows/sagemaker_pipelines/resources/pipeline.png)

### Pipeline parameters
#### Pipeline parameters
The example pipeline uses [parameters](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html) that you can reference throughout the steps. You can also use the parameters to change
values between runs. In this example, the parameters are used to set the Amazon Simple Storage Service (Amazon S3)
locations for the input and output files, along with the identifiers for the role and queue to use in the pipeline.
The example demonstrates how to set and access these parameters.
values between runs and control the input and output setting. In this example, the parameters are used to set the Amazon Simple Storage Service (Amazon S3)
locations for the input and output files, along with the identifiers for the role and queue to use in the pipeline.
The example demonstrates how to set and access these parameters before executing the pipeline using an SDK.

### Geospatial jobs
#### Geospatial jobs
A SageMaker pipeline can be used for model training, setup, testing, or validation. This example uses a simple job
for demonstration purposes: a [Vector Enrichment Job (VEJ)](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) that processes a set of coordinates to produce human-readable
addresses powered by Amazon Location Service. Other types of jobs could be substituted in the pipeline instead.

for demonstration purposes: a [Vector Enrichment Job (VEJ)](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) that processes a set of coordinates to produce human-readable
addresses powered by Amazon Location Service. Other types of jobs can be substituted in the pipeline instead.
## ⚠ Important

* Running this code might result in charges to your AWS account.
Expand Down Expand Up @@ -58,7 +74,7 @@ You must download and use these files to successfully run this code example:
+ GeoSpatialPipeline.json
+ latlongtest.csv

These files are located on GitHub in this folder [sample_files](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/resources/sample_files).
These files are located on GitHub in this [folder](../../../workflows/sagemaker_pipelines/resources).

### Java Lambda Function

Expand Down
Binary file not shown.
53 changes: 35 additions & 18 deletions kotlin/usecases/workflow_sagemaker_pipes/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,49 @@

This scenario demonstrates how to work with Amazon SageMaker pipelines and geospatial jobs.

A [SageMaker pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines.html) is a series of
interconnected steps that can be used to automate machine learning workflows. You can create and run pipelines from SageMaker Studio by using Python, but you can also do this by using AWS SDKs in other
### Amazon SageMaker Pipelines
A [SageMaker pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines.html) is a series of
interconnected steps that can be used to automate machine learning workflows. Pipelines use interconnected steps and shared parameters to support repeatable workflows that can be customized for your specific use case. You can create and run pipelines from SageMaker Studio using Python, but you can also do this by using AWS SDKs in other
languages. Using the SDKs, you can create and run SageMaker pipelines and also monitor operations for them.

### Pipeline steps
This example pipeline includes an [AWS Lambda step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda)
and a [callback step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback).
Both steps are processed by the same example Lambda function.
### Explore the scenario
This example scenario demonstrates using AWS Lambda and Amazon Simple Queue Service (Amazon SQS) as part of an Amazon SageMaker pipeline. The pipeline itself executes a geospatial job to reverse geocode a sample set of coordinates into human-readable addresses. Input and output files are located in an Amazon Simple Storage Service (Amazon S3) bucket.

This Lambda code is included as part of this example, with the following functionality:
- Starts the SageMaker Vector Enrichment Job with the provided job configuration.
![Workflow image](../../../workflows/sagemaker_pipelines/resources/workflow.png)

When you run the example console application, you can execute the following steps:

- Create the AWS resources and roles needed for the pipeline.
- Create the AWS Lambda function.
- Create the SageMaker pipeline.
- Upload an input file into an S3 bucket.
- Execute the pipeline and monitor its status.
- Display some output from the output file.
- Clean up the pipeline resources.

#### Pipeline steps
[Pipeline steps](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html) define the actions and relationships of the pipeline operations. The pipeline in this example includes an [AWS Lambda step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda)
and a [callback step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback).
Both steps are processed by the same example Lambda function.

The Lambda function handler is included as part of the example, with the following functionality:
- Starts a [SageMaker Vector Enrichment Job](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) with the provided job configuration.
- Processes Amazon SQS queue messages from the SageMaker pipeline.
- Starts the export function with the provided export configuration.
- Processes Amazon Simple Queue Service (Amazon SQS) messages from the SageMaker pipeline.
- Completes the pipeline when the export is complete.

![AWS App](images/pipes.png)
![Pipeline image](../../../workflows/sagemaker_pipelines/resources/pipeline.png)

### Pipeline parameters
#### Pipeline parameters
The example pipeline uses [parameters](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html) that you can reference throughout the steps. You can also use the parameters to change
values between runs. In this example, the parameters are used to set the Amazon Simple Storage Service (Amazon S3)
locations for the input and output files, along with the identifiers for the role and queue to use in the pipeline.
The example demonstrates how to set and access these parameters.
values between runs and control the input and output setting. In this example, the parameters are used to set the Amazon Simple Storage Service (Amazon S3)
locations for the input and output files, along with the identifiers for the role and queue to use in the pipeline.
The example demonstrates how to set and access these parameters before executing the pipeline using an SDK.

### Geospatial jobs
#### Geospatial jobs
A SageMaker pipeline can be used for model training, setup, testing, or validation. This example uses a simple job
for demonstration purposes: a [Vector Enrichment Job (VEJ)](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) that processes a set of coordinates to produce human-readable
addresses powered by Amazon Location Service. Other types of jobs could be substituted in the pipeline instead.
for demonstration purposes: a [Vector Enrichment Job (VEJ)](https://docs.aws.amazon.com/sagemaker/latest/dg/geospatial-vej.html) that processes a set of coordinates to produce human-readable
addresses powered by Amazon Location Service. Other types of jobs can be substituted in the pipeline instead.

## ⚠ Important

Expand Down Expand Up @@ -59,7 +76,7 @@ To successfully run this code example, you must download and use the following f
+ GeoSpatialPipeline.json
+ latlongtest.csv

These files are located on GitHub in [sample_files](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/resources/sample_files).
These files are located on GitHub in this [folder](../../../workflows/sagemaker_pipelines/resources).

### Kotlin Lambda function

Expand Down
Binary file not shown.
Loading

0 comments on commit b4c74eb

Please sign in to comment.