Resource offer cleanup sometimes removes the wrong offers

[bgins]

General Description

In our current implementation, the solver removes a resource offer when a resource provider disconnects:

lilypad/pkg/solver/server.go

Lines 163 to 173 in da719da

	func (solverServer *solverServer) disconnectCB(connParams http.WSConnectionParams) {
	if connParams.Type == "ResourceProvider" {
	metricsDashboard.TrackNodeConnectionEvent(metricsDashboard.NodeConnectionParams{
	Event: "Disconnect",
	ID: connParams.ID,
	CountryCode: connParams.CountryCode,
	IP: connParams.IP,
	})
	solverServer.controller.removeResourceOfferByResourceProvider(connParams.ID)
	}
	}

The removeResourceOfferByResourceProvider implementation removes a single resource offer:

lilypad/pkg/solver/controller.go

Lines 420 to 423 in da719da

	err = controller.store.RemoveResourceOffer(resourceOffers[0].ID)
	if err != nil {
	return err
	}

We store the resource offers in a map:

lilypad/pkg/solver/store/memory/store.go

Line 15 in da719da

resourceOfferMap map[string]*data.ResourceOfferContainer

When a job is complete, we mark the resource offer with a ResultsAccepted[3] state, but we do not remove it from the map.

Maps are unordered in Go, and a result removeResourceOfferByResourceProvider removes a random resource offer which may or may not be the active, unmatched resource offer.

In addition, resource providers may submit multiple resource offers. Cleaning up a single resource offer may not be enough in general.

Which system(s) or functionality does this affect

The changes to fix this issue will affect the solver and resource providers.

Describe the changes, and how this affects/ interacts with each system

We can fix this issue by removing all resource offers in a DealNegotiating[0] state. These are the resource offers that are unmatched, and we would like to remove them so they do not get matched when a resource provider is not connected to the solver.

We should avoid removing offers in any other state. Offers in a DealAgreed[1] and ResultsSubmitted[2] are in flight and we should be especially careful to preserve them.