✖️ Toolkit v0.2 -- Old discussion

[flanakin]

🎯 Primary goals

Ingest and normalize cost data for Enterprise Agreement and Microsoft Customer Agreement accounts to a normalized schema in FinOps hubs.

The schema will be an interim release that will be updated when the FinOps Open Cost and Usage Specification (FOCUS) 1.0 is released. We will do our best to align with the direction FOCUS 1.0 is going to minimize change.

🙋‍♀️ How you can help

1. Are we missing anything you'd expect in this release?
2. Is there anything you feel is more critical before this release?
3. Thinking about going past v0.2, what do you want to see next?

📋 Milestone tasks: Toolkit v0.2
🌱 Working branch: features/exports

[helderpinto]

Is schema normalization between EA/MCA-E/MCA/CSP Azure Plan/MOSA, etc. in the Azure Cost Management roadmap?

If answer is "yes", isn't this release maybe superfluous if a few months later ACM normalization is available to the public, or is this also an effort that contributes to the ACM normalization roadmap itself?

If answer is "no", we should maybe make the schema normalization bits available as a module customers would use independently from finops-hub and integrate into their FinOps pipelines.

[flanakin]

We'll do many things in FinOps hubs that are on the Cost Management backlog. When there's a clear, near-term ETA, we'll likely wait. But in cases like normalization where we don't have an ETA, we need to unblock ourselves. This is more critical for FinOps hubs due to the requirement to merge data across account types. When Cost Management supports normalization, we'll switch over to that. We just need to unblock ourselves to make progress (like every organization has done so far 😜).

Totally agree about making normalization available outside of FinOps hubs. We've talked about 3 ways but haven't closed anything:

1. Create a separate Data Factory pipeline template as part of the FinOps toolkit.
2. Document how to set it up and provide the schema files so it's repeatable and customizable without getting into a template.
3. Deploy FinOps hubs in an integration mode where we can trigger external pipelines/webhooks when ingestion is complete.

3 (integration mode) is already part of our roadmap and that sets people up for other things that are coming as well. This is the ideal path.

2 (manual setup) is an obvious one that's also easy to do.

1 (pipeline template) is easy enough, but there is a question whether this would be used vs. 2 (manual setup). Are you likely to deploy a standalone pipeline for normalization without customizing it first and integrating it into an existing solution? Maybe. I'm not sure. This is where I'd love the community's feedback.

Do you think we're missing any other options? Fwiw, I'm just referring to what we do in FinOps hubs here, but we've also talked about a few other options for how to enable FOCUS specifically.

Side note: It's "Microsoft" Cost Management 😉

[helderpinto]

Thanks for all the details you shared, @flanakin. It is very much appreciated. Among the three proposed ways for schema normalization, I see 1) and 3) as the most interesting ones, avoiding customers to reinvent the wheel when no additional customization is needed besides normalization. For example, there are customers who have under the same tenant EA and CSP subscriptions, or who are planning to migrate from EA to MCA soon. Providing them with a ready-made solution that allows for bringing all their consumption data into the same schema saves them from unnecessary efforts. I am not sure about whether option 1) or 3) should be prioritized - I need to know more about what the "integration mode" is.

[flanakin]

Update: We're splitting this into 2 releases. Adopting a normalized schema will come in 0.2 with some other Power BI updates and other features like managed exports and cross-tenant/remote hubs will stay in this release, but be renamed to 0.3.

[flanakin]

Closing this since the conversation has been focused on the normalized schema, which was shipped in 0.2.