Docs on ZenML setup

.gitbook.yaml

@@ -6,12 +6,14 @@ structure:
 
 redirects:
   how-to/customize-docker-builds/use-code-repositories-to-speed-up-docker-build-times: how-to/customize-docker-builds/how-to-reuse-builds.md
-  reference/migration-guide/README.md: how-to/manage-the-zenml-server/migration-guide/migration-guide.md
-  reference/migration-guide/migration-zero-twenty.md: how-to/manage-the-zenml-server/migration-guide/migration-zero-twenty.md
-  reference/migration-guide/migration-zero-thirty.md: how-to/manage-the-zenml-server/migration-guide/migration-zero-thirty.md
-  reference/migration-guide/migration-zero-forty.md: how-to/manage-the-zenml-server/migration-guide/migration-zero-forty.md
-  reference/migration-guide/migration-zero-sixty.md: how-to/manage-the-zenml-server/migration-guide/migration-zero-sixty.md
+  reference/migration-guide: how-to/manage-the-zenml-server/migration-guide/migration-guide.md
+  reference/migration-guide/migration-zero-twenty: how-to/manage-the-zenml-server/migration-guide/migration-zero-twenty.md
+  reference/migration-guide/migration-zero-thirty: how-to/manage-the-zenml-server/migration-guide/migration-zero-thirty.md
+  reference/migration-guide/migration-zero-forty: how-to/manage-the-zenml-server/migration-guide/migration-zero-forty.md
+  reference/migration-guide/migration-zero-sixty: how-to/manage-the-zenml-server/migration-guide/migration-zero-sixty.md
+
+  getting-started/deploying-zenml/manage-the-deployed-services/upgrade-the-version-of-the-zenml-server: how-to/manage-the-zenml-server/upgrade-zenml-server.md
+  getting-started/deploying-zenml/manage-the-deployed-services/troubleshoot-your-deployed-server: how-to/manage-the-zenml-server/troubleshoot-your-deployed-server.md
+  how-to/stack-deployment/implement-a-custom-integration: how-to/contribute-to-zenml/implement-a-custom-integration.md
+  how-to/setting-up-a-project-repository/best-practices: how-to/setting-up-a-project-repository/set-up-repository.md
 
-  getting-started/deploying-zenml/manage-the-deployed-services/upgrade-the-version-of-the-zenml-server.md: how-to/manage-the-zenml-server/upgrade-zenml-server.md
-  getting-started/deploying-zenml/manage-the-deployed-services/troubleshoot-your-deployed-server.md: how-to/manage-the-zenml-server/troubleshoot-your-deployed-server.md
-  how-to/stack-deployment/implement-a-custom-integration.md: how-to/contribute-to-zenml/implement-a-custom-integration.md

docs/book/how-to/customize-docker-builds/how-to-reuse-builds.md

@@ -33,7 +33,7 @@ While reusing Docker builds is useful, it can be limited. This is because specif
 
 ## Use the artifact store to upload your code
 
-You can also let ZenML use the artifact store to upload your code. This is the default behaviour if no code repository is detected and the `allow_download_from_artifact_store` flag is not set to `False` in your `DockerSettings`.
+You can also let ZenML use the artifact store to upload your code. This is the default behavior if no code repository is detected and the `allow_download_from_artifact_store` flag is not set to `False` in your `DockerSettings`.
 
 ## Use code repositories to speed up Docker build times
 

docs/book/how-to/customize-docker-builds/how-to-use-a-private-pypi-repository.md

@@ -0,0 +1,33 @@
+---
+description: How to use a private PyPI repository.
+---
+
+# How to use a private PyPI repository
+
+For packages that require authentication, you may need to take additional steps:
+
+1. Use environment variables to store credentials securely.
+2. Configure pip or poetry to use these credentials when installing packages.
+3. Consider using custom Docker images that have the necessary authentication setup.
+
+Here's an example of how you might set up authentication using environment variables:
+
+```python
+import os
+from zenml.config import DockerSettings
+
+os.environ['PIP_EXTRA_INDEX_URL'] = f"https://{os.environ['PYPI_USERNAME']}:{os.environ['PYPI_PASSWORD']}@your-private-pypi-server.com/simple"
+
+docker_settings = DockerSettings(requirements=["my-private-package==1.0.0"])
+
+@pipeline(settings={"docker": docker_settings})
+def my_pipeline(...):
+    ...
+```
+
+Note: Be cautious with handling credentials. Always use secure methods to manage
+and distribute authentication information within your team.
+<!-- For scarf -->
+<figure><img alt="ZenML Scarf" referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" /></figure>
+
+

docs/book/how-to/manage-the-zenml-server/best-practices-upgrading-zenml.md

@@ -42,7 +42,7 @@ ZenML Pro comes with multi-tenancy which makes it easy for you to have multiple
 
 ## Upgrading your code
 
-Sometimes, you might have to upgrade your code to work with a new version of ZenML. This is true especially when you are moving from a really old version to a new major version. The following tips might help, in addition to everything you've learnt in this document so far.
+Sometimes, you might have to upgrade your code to work with a new version of ZenML. This is true especially when you are moving from a really old version to a new major version. The following tips might help, in addition to everything you've learned in this document so far.
 
 ### Testing and Compatibility
 

docs/book/how-to/setting-up-a-project-repository/README.md

@@ -2,11 +2,92 @@
 description: Setting your team up for success with a project repository.
 ---
 
-# 😸 Setting up a project repository
+# 😸 Setting up a Well-Architected ZenML Project
 
-ZenML code typically lives in a `git` repository. Setting this repository up correctly can make a huge impact on collaboration and
-getting the maximum out of your ZenML deployment. This section walks users through some of the options available to create a project
-repository with ZenML.
+Welcome to the guide on setting up a well-architected ZenML project. This section will provide you with a comprehensive overview of best practices, strategies, and considerations for structuring your ZenML projects to ensure scalability, maintainability, and collaboration within your team.
+
+## The Importance of a Well-Architected Project
+
+A well-architected ZenML project is crucial for the success of your machine learning operations (MLOps). It provides a solid foundation for your team to develop, deploy, and maintain ML models efficiently. By following best practices and leveraging ZenML's features, you can create a robust and flexible MLOps pipeline that scales with your needs.
+
+## Key Components of a Well-Architected ZenML Project
+
+### Repository Structure
+
+A clean and organized repository structure is essential for any ZenML project. This includes:
+
+- Proper folder organization for pipelines, steps, and configurations
+- Clear separation of concerns between different components
+- Consistent naming conventions
+
+Learn more about setting up your repository in the [Set up repository guide](./best-practices.md).
+
+### Version Control and Collaboration
+
+Integrating your ZenML project with version control systems like Git is crucial for team collaboration and code management. This allows for:
+
+- Makes creating pipeline builds faster, as you can leverage the same image and [have ZenML download code from your repository](../../how-to/customize-docker-builds/how-to-reuse-builds.md#use-code-repositories-to-speed-up-docker-build-times).
+- Easy tracking of changes
+- Collaboration among team members
+
+Discover how to connect your Git repository in the [Set up a repository guide](./best-practices.md).
+
+### Stacks, Models, and Pipelines
+
+Understanding the relationship between stacks, models, and pipelines is key to designing an efficient ZenML project:
+- Stacks: Define your infrastructure and tool configurations
+- Models: Represent your machine learning models and their metadata
+- Pipelines: Encapsulate your ML workflows
+
+Learn about organizing these components in the [Organizing Stacks, Pipelines, and Models guide](./stacks-pipelines-models.md).
+
+### Access Management and Roles
+
+Proper access management ensures that team members have the right permissions and responsibilities:
+
+- Define roles such as data scientists, MLOps engineers, and infrastructure managers
+- Set up service connectors and manage authorizations
+- Establish processes for pipeline maintenance and server upgrades
+- Leverage [Teams in ZenML Pro](../../getting-started/zenml-pro/zenml-pro.md) to assign roles and permissions to a group of users, to mimic your real-world team roles.
+TODO: update when teams is merged
+
+Explore access management strategies in the [Access Management and Roles guide](./access-management-and-roles.md).
+
+### Shared Components and Libraries
+
+Leverage shared components and libraries to promote code reuse and standardization across your team:
+
+- Custom flavors, steps, and materializers
+- Shared private wheels for internal distribution
+- Handling authentication for specific libraries
+
+Find out more about sharing code in the [Shared Libraries and Logic for Teams guide](./shared_components_for_teams.md).
+
+### Project Templates
+
+Utilize project templates to kickstart your ZenML projects and ensure consistency:
+
+- Use pre-made templates for common use cases
+- Create custom templates tailored to your team's needs
+
+Learn about using and creating project templates in the [Project Templates guide](./project-templates.md).
+
+### Migration and Maintenance
+
+As your project evolves, you may need to migrate existing codebases or upgrade your ZenML server:
+
+- Strategies for migrating legacy code to newer ZenML versions
+- Best practices for upgrading ZenML servers
+
+Discover migration strategies and maintenance best practices in the [Migration and Maintenance guide](../../how-to/manage-the-zenml-server/best-practices-upgrading-zenml.md#upgrading-your-code).
+
+## Getting Started
+
+To begin building your well-architected ZenML project, start by exploring the guides in this section. Each guide provides in-depth information on specific aspects of project setup and management.
+
+Remember, a well-architected project is an ongoing process. Regularly review and refine your project structure, processes, and practices to ensure they continue to meet your team's evolving needs.
+
+By following these guidelines and leveraging ZenML's powerful features, you'll be well on your way to creating a robust, scalable, and collaborative MLOps environment.
 
 <figure><img src="../../.gitbook/assets/Remote_with_code_repository.png" alt=""><figcaption><p>A visual representation of how the code repository fits into the general ZenML architecture.</p></figcaption></figure>
 

docs/book/how-to/setting-up-a-project-repository/access-management.md

@@ -0,0 +1,82 @@
+- Write about Access Management and roles within a ZenML server (data scientist, stacks/infra engineer) etc
+    - Write on  how to set up service connectors and give authorization to which team members
+    - write about who migrates and maintains pipelines (link to the right place from [Include best practices for upgrading ZenML servers](../../how-to/manage-the-zenml-server/best-practices-upgrading-zenml.md))
+    - Write about who upgrades the zenml server and how to do it (link to the right place from [Include best practices for upgrading ZenML servers](../../how-to/manage-the-zenml-server/best-practices-upgrading-zenml.md))
+
+# Access Management and Roles in ZenML
+
+Effective access management is crucial for maintaining security and efficiency in your ZenML projects. This guide will help you understand the different roles within a ZenML server and how to manage access for your team members.
+
+## Typical Roles in an ML Project
+
+In an ML project, you will typically have the following roles:
+
+- Data Scientists: Primarily work on developing and running pipelines.
+- MLOps Platform Engineers: Manage the infrastructure and stack components.
+- Project Owners: Oversee the entire ZenML deployment and manage user access.
+
+The above is an estimation of roles that you might have in your team. In your case, the names might be different or there might be more roles, but you can relate the responbilities we discuss in this document to your own project loosely.
+
+{% hint style="info" %}
+You can create Roles in ZenML Pro with a given set of permissions and assign them to either Users or Teams that represent your real-world team structure. Sign up for a free trial to try it yourself: https://cloud.zenml.io/
+{% endhint %} TODO: add link to roles in Pro
+
+## Service Connectors: Gateways to External Services
+
+Service connectors are how different cloud services are integrated with ZenML. They are used to abstract away the credentials and other configurations needed to access these services.
+
+Ideally, you would want that only the MLOps Platform Engineers have access for creating and managing connectors. This is because they are closest to your infrastructure and can make informed decisions about what authentication mechanisms to use and more.
+
+Other team members can use connectors to create stack components that talk to the external services but should not have to worry about setting them and shouldn't have access to the credentials used to configure them.
+
+Learn more about the best practices in managing credentials and recommended roles in our [Managing Stacks and Components guide](../stack-deployment/README.md).
+
+
+## ZenML Server Upgrades
+
+The decision to upgrade your ZenML server is usually taken by your Project Owners after consulting with all the teams using the server. This is because there might be teams with conflicting requirements and moving to a new version of ZenML (that might come with upgrades to certain libraries) can break code for some users.
+
+{% hint style="info" %}
+You can choose to have different servers for different teams and that can alleviate some of the pressure to upgrade if you have multiple teams using the same server. ZenML Pro offers multi-tenancy out of the box, for situations like these. Sign up for a free trial to try it yourself: https://cloud.zenml.io/
+TODO: add link to tenants in Pro
+{% endhint %}
+
+Performing the upgrade itself is a task that typically falls on the MLOps Platform Engineers. They should:
+
+- ensure that all data is backed up before performing the upgrade
+- no service disruption or downtime happens during the upgrade
+
+and more. Read in detail about the best practices for upgrading your ZenML server in the [Best Practices for Upgrading ZenML Servers](../manage-the-zenml-server/best-practices-upgrading-zenml.md) guide.
+
+
+## Pipeline Migration and Maintenance
+
+When you upgrade to a new version of ZenML, you might have to test if your code works as expected and if the syntax is up to date with what ZenML expects. Although we do our best to make new releases compatible with older versions, there might be some breaking changes that you might have to address.
+
+The pipeline code itself is typically owned by the Data Scientist, but the Platform Engineer is responsible for making sure that new changes can be tested in a safe environment without impacting existing workflows. This involves setting up a new server and doing a staged upgrade and other strategies.
+
+The Data Scientist should also check out the release notes, and the migration guide where applicable when upgrading the code. Read more about the best practices for upgrading your ZenML server and your code in the [Best Practices for Upgrading ZenML Servers](../manage-the-zenml-server/best-practices-upgrading-zenml.md) guide.
+
+
+## Best Practices for Access Management
+
+Apart from the role-specific tasks we discussed so far, there are some general best practices you should follow to ensure a secure and well-managed ZenML environment that supports collaboration while maintaining proper access controls.
+
+- Regular Audits: Conduct periodic reviews of user access and permissions.
+- Role-Based Access Control (RBAC): Implement RBAC to streamline permission management.
+- Least Privilege: Grant minimal necessary permissions to each role.
+- Documentation: Maintain clear documentation of roles, responsibilities, and access policies.
+
+{% hint style="info" %}
+The Role-Based Access Control (RBAC) and assigning of permissions is only available for ZenML Pro users.
+{% endhint %}
+
+By following these guidelines, you can ensure a secure and well-managed ZenML environment that supports collaboration while maintaining proper access controls.
+
+
+
+
+<!-- For scarf -->
+<figure><img alt="ZenML Scarf" referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" /></figure>
+
+

docs/book/how-to/setting-up-a-project-repository/create-your-own-template.md

@@ -0,0 +1,48 @@
+---
+description: How to create your own ZenML template.
+---
+
+# Create your own ZenML template
+
+Creating your own ZenML template is a great way to standardize and share your ML workflows across different projects or teams. ZenML uses [Copier](https://copier.readthedocs.io/en/stable/) to manage its project templates. Copier is a library that allows you to generate projects from templates. It's simple, versatile, and powerful.
+
+Here's a step-by-step guide on how to create your own ZenML template:
+
+1. **Create a new repository for your template.** This will be the place where you store all the code and configuration files for your template.
+2. **Define your ML workflows as ZenML steps and pipelines.** You can start by copying the code from one of the existing ZenML templates (like the [starter template](https://github.com/zenml-io/template-starter)) and modifying it to fit your needs.
+3. **Create a `copier.yml` file.** This file is used by Copier to define the template's parameters and their default values. You can learn more about this config file [in the copier docs](https://copier.readthedocs.io/en/stable/creating/).
+4. **Test your template.** You can use the `copier` command-line tool to generate a new project from your template and check if everything works as expected:
+
+```bash
+copier copy https://github.com/your-username/your-template.git your-project
+```
+
+Replace `https://github.com/your-username/your-template.git` with the URL of your template repository, and `your-project` with the name of the new project you want to create.
+
+5. **Use your template with ZenML.** Once your template is ready, you can use it with the `zenml init` command:
+
+```bash
+zenml init --template https://github.com/your-username/your-template.git
+```
+
+Replace `https://github.com/your-username/your-template.git` with the URL of your template repository.
+
+If you want to use a specific version of your template, you can use the `--template-tag` option to specify the git tag of the version you want to use:
+
+```bash
+zenml init --template https://github.com/your-username/your-template.git --template-tag v1.0.0
+```
+
+Replace `v1.0.0` with the git tag of the version you want to use.
+
+That's it! Now you have your own ZenML project template that you can use to quickly set up new ML projects. Remember to keep your template up-to-date with the latest best practices and changes in your ML workflows.
+
+Our [Production Guide](../../user-guide/production-guide/README.md) documentation is built around the `E2E Batch` project template codes. Most examples will be based on it, so we highly recommend you to install the `e2e_batch` template with `--template-with-defaults` flag before diving deeper into this documentation section, so you can follow this guide along using your own local environment.
+
+```bash
+mkdir e2e_batch
+cd e2e_batch
+zenml init --template e2e_batch --template-with-defaults
+```
+
+<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>