MAC address is not persistent on Rebooting the system.

[swe12345]

On rebooting the system expecting that the newly assigned MAC address must persist

Steps:

1. Assign a valid MAC address xx:xx:xx:xx:xx:xx to the system using the curl command
   curl -k -H "X-Auth-Token:${bmc_token}" -X PATCH -D patch.txt -d '{"MACAddress":"xx:xx:xx:xx:xx:xx"}' https://$bmc/redfish/v1/Managers/bmc/EthrnetInterfaces/eth0
2. Now GET and verify if the MAC address has changed using
   curl -k -H "X-Auth-Token:${bmc_token}" -X GET  https://${bmc}/redfish/v1/Managers/bmc/EthernetInterfaces/eth0
3. Now reboot the BMC and verify if the MAC address remains the same

Expectation:

Expecting that the MAC address should persist on rebooting the system.

@sunharis @raviteja-b

[williamspatrick]

Does your machine have configured a place to store the MAC? Or are you expecting it to be saved in u-boot?

[swe12345]

@ratagupt @wak-google

[prkatti1]

The MAC address we assign was persistent on reboot in earlier builds, this issue we are noticing is only on recent builds.
We are expecting the same behavior.

[wak-google]

Specifically, what version of phosphor-networkd are you using?

[wak-google]

Also, did you possibly enable https://github.com/openbmc/phosphor-networkd/blob/master/meson_options.txt#L13 in your config?

[swe12345]

We are using the lastest networkd. @raviteja-b @sunharis Can you please answer Wak's questions.

[raviteja-b]

Also, did you possibly enable https://github.com/openbmc/phosphor-networkd/blob/master/meson_options.txt#L13 in your config?

yes, enabled persist-mac meson flag.
@wak-google do we need to persist user-configured MAC address after reboot? what's the right behaviour here

[wak-google]

Are you sure you have 9caa64d

[raviteja-b]

Are you sure you have 9caa64d

yes @wak-google

[mdmillerii]

Have you checked the u-boot environment to see if the MAC has been set? fw_printenv ethaddr eth1addr for the first and second interface known to u-boot. Also you can check the u-boot prompt with printenv to verify it is looking in the same place for its environment.

[prkatti1]

Do we have any update here? We are expecting MAC address to be persistent on reboot like earlier.

[williamspatrick]

Do we have any update here? We are expecting MAC address to be persistent on reboot like earlier.

Seems likely no one has further ideas and you will need to debug and fix it?

[prkatti1]

Do we have any update here? We are expecting MAC address to be persistent on reboot like earlier.

Seems likely no one has further ideas and you will need to debug and fix it?

I'd love to fix, need some info why the earlier function is changed? which commit has introduced this issue? as we are seeing this issue very recently.

[williamspatrick]

I'd love to fix, need some info why the earlier function is changed? which commit has introduced this issue? as we are seeing this issue very recently.

If we knew this, the fix would probably be obvious. You’re welcome to experiment with git bisect.

[mdmillerii]

Have you performed any of the troubleshooting i suggested to help isolate which step might be failing? What were the results?

[zevweiss]

@prkatti1, how recently did you notice the behavior change? We did recently (OpenBMC incorporated it in this kernel update) add this patch affecting the ftgmac100 MAC-address initialization path; I'm not sure offhand if/how it could be causing the problem you're seeing, but it might be worth testing a revert to see if it changes anything?

[swe12345]

We are seeing this issue from April onwards. @zevweiss

[swe12345]

MAC address was persistent on rebooting the system for the past 7 years, seeing this change only from April onwards @zevweiss

[zevweiss]

Okay, well that's significantly before the patch I linked went in, so presumably not the cause (for @swe12345 at least -- does your "we" also include @prkatti1? Not sure if you're at the same employer). As suggested above, investigating with git bisect could help narrow down exactly when the change was introduced; I don't know of any other changes off the top of my head that seem potentially related.

[swe12345]

After rebooting the machine seeing that printenv in uboot environment is updated with the original value and not the new changed MAC address.

===On setting the MAC address
ast# printenv ethaddr eth1addr eth2addr
ethaddr=aa:e2:84:14:xx:xx
eth1addr=08:94:ef:82:yy:yy

Error: "eth2addr" not defined

ast#

====After reboot
ast# printenv ethaddr eth1addr eth2addr
ethaddr=08:94:ef:82:xx:xx
eth1addr=08:94:ef:82:yy:yy

Error: "eth2addr" not defined

ast#

[mdmillerii]

So the u-boot environment is not being updated when the change is made (it's not reverting back much more likely it's not being set, unless there is a mismatched size causing a checksum error and revert to b side)

Next suggested debug step is confirm the u-boot environment with fw_getenv matches the one in u-boot.

After that it's trace phosphor-networkd invoking fw_setenv starting with the meson option pointed out earlier and following through the code.

[swe12345]

@zevweiss @mdmillerii
Checked the value of fw_printenv on the BMC
======After changing MAC
root@xxxxx:~# fw_printenv|grep eth
eth1addr=08:94:ef:82:xx:xx
ethaddr=aa:e2:84:14:yy:yy

=====On rebooting the system
root@xxxxxx:~# fw_printenv|grep eth
ethaddr=08:94:ef:82:xx:xx
eth1addr=08:94:ef:82:yy:yy

[mdmillerii]

Check if either your u-boot or a script is extracting it from vpd even when already set. You can read at u-boot prompt after setting before reboot.

[swe12345]

On changing the MAC address seeing that the VPD is not updated with the updated value.
root@xxxxxx:# vpd-tool -i |grep MAC
"MACAddress": "08:94:ef:82:xx:xx",
"MACAddress": "08:94:ef:82:yy:yy",
root@xxxxxx:#

And on rebooting the system too the MAC remains same(original MAC) in the VPD.

[mdmillerii]

There is no code I'm aware of to update VPD for a MAC change. It's expected that any such change from shell or redfish would be administratively assigned and not the manufacturer assigned address which would be expected in the VPD.

I suppose the recent change to support the kernel to read the MAC from nvmem eg I2C exposes we don't have a path to record the administrator override address in phosphor networkd.

Systemd networkd can use a .link file to store the permanent and desired MAC address to use but code will need to be submitted to gerrit to manage to manage the file.

If you have a script taking the value from vpd and assigning the Mac then it needs to be written with only once logic. Note I don't recall seeing which platform you are building throughout this issue.

Also, please review the markdown syntax when posting log or code snippets. The documentation is linked just under the editor text box {Markdown is supported.](https://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax)