README.md

Serving Working Group

Discuss and enhance the support of inference serving for accelerated workloads in Kubernetes. Make Kubernetes the natural choice for hosting production inference reliably, and improve all serving workloads along the way.

The charter defines the scope and governance of the Serving Working Group.

Stakeholder SIGs

• SIG Apps
• SIG Architecture
• SIG Autoscaling
• SIG Instrumentation
• SIG Network
• SIG Node
• SIG Scheduling
• SIG Storage

Meetings

Joining the mailing list for the group will typically add invites for the following meetings to your calendar.

• WG Serving Weekly Meeting (Calendar): Wednesdays at 9:00 PT (Pacific Time) (weekly). Convert to your timezone.
  • Meeting notes and Agenda.
  • Meeting recordings.

Organizers

• Eduardo Arango (@ArangoGutierrez), NVIDIA
• Jiaxin Shan (@Jeffwan), Bytedance
• Sergey Kanzhelev (@SergeyKanzhelev), Google
• Yuan Tang (@terrytangyuan), Red Hat

Contact

• Slack: #wg-serving
• Mailing list
• Open Community Issues/PRs
• Steering Committee Liaison: Maciej Szulik (@soltysh)

Workstreams

The WG Serving will operate in several workstreams with different areas of focus. The following workstreams are currently active:

• Autoscaling
• MultiHost/MultiNode
• Orchestration
• DRA

Sponsored Subprojects

• LLM Instance Gateway
• Serving Catalog

How to ...

subscribe to WG events

1. Join the [email protected] and [email protected] for needed permissions.
2. You should have meeting invites in your inbox after some delay. Make sure to fish them out of Junk/Spam and accept the invite so it will be added to your calendar.
3. Import WG Serving Shared Calendar to see WG events.

schedule an ad-hoc or workstream meeting

1. Get access to the WG Serving Shared Calendar. Ask chairs in #wg-serving slack chat.
2. Create a meeting notes document or find existing one. Make sure it is shared as following:
   • Shared with [email protected] for commenting
   • Shared with [email protected] for editing
3. Create an event. Make sure to create it in the shared calendar.
   • Use this zoom link: https://zoom.us/j/9955436256?pwd=Z2FQWU1jeDZkVC9RRTN4TlZyZTBHZz09
   • Invite [email protected]
   • Invite [email protected] so the event will appear in https://www.kubernetes.dev/resources/calendar/
   • Mention the meeting notes document in the description
4. Make sure one of the chairs has accepted the invite. If not - ask chairs for a host key to claim a host during the meeting.
5. Start a recording of the meeting (record to the cloud).
6. You can end the recording explicitly or it will be finished when the meeting will end and all participants will leave it

upload meeting recording to YouTube (for chairs)

1. Download the video file from zoom account [email protected]
2. Log in to YouTube under the kubernetes-wg-serving account (use switch account)
3. Upload video to YouTube
   • Name should follow the template: [WG Serving] name of the meeting 20240516.
   • Add meeting notes and agenda document to the meeting description
   • Add video to k8s-owned YouTube playlist:
     • Once video if uploaded, open the video page, click +Save button under the video, select the checkbox WG Serving.
4. Add link to the video to the meeting notes document
5. In zoom account change the title of the recording - add (uploaded) suffix