Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock when generating encryption key #107

Merged
merged 10 commits into from
Aug 16, 2018
Merged
2 changes: 1 addition & 1 deletion config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package config

import (
"time"
"time"

"github.com/Peripli/service-manager/api"
"github.com/Peripli/service-manager/pkg/env"
Expand Down
5 changes: 4 additions & 1 deletion pkg/sm/sm.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,9 @@ func (smb *ServiceManagerBuilder) Build() *ServiceManager {
}

func initializeSecureStorage(secureStorage storage.Security) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just thinking out loud - should we explicitly fail if this method actually attempts to generate a new key (len(encryptionKey) == 0) and we already have data in the database? Is there a sane way to validate this? Because we would be in a very messed up situation if SM starts in such state

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on line

return fmt.Errorf("Could not generate encryption key: %v", err)

wrap error with %v - its fine but in some places in SM we use %s and in others we do it with %v - should we try to be consistent?

if err := secureStorage.Lock(); err != nil {
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to return an error if database is locked. I think it should be:

  1. Try to lock database.
  2. If it is already locked, then wait some time (2 seconds for example)
  3. Try to lock again
  4. Repeat several times(5 for example)
  5. If it couldn't acquire the lock, then panic
  6. If lock is acquired then check for the encryption key. If it is already there then do nothing, otherwise generate one

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears that advisory locks block until a lock can be acquired:
https://medium.com/@codeflows/application-level-locking-with-postgres-advisory-locks-c29759eeb7a7

}
keyFetcher := secureStorage.Fetcher()
encryptionKey, err := keyFetcher.GetEncryptionKey()
if err != nil {
Expand All @@ -172,7 +175,7 @@ func initializeSecureStorage(secureStorage storage.Security) error {
}
logrus.Debug("Successfully generated new encryption key")
}
return nil
return secureStorage.Unlock()
}

func handleInterrupts(ctx context.Context, cancelFunc context.CancelFunc) {
Expand Down
9 changes: 7 additions & 2 deletions storage/interfaces.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,14 @@ type Credentials interface {
}

// Security interface for encryption key operations
type Security interface{
type Security interface {
// Lock locks the storage so that only one process can manipulate the encryption key.
// Returns an error if the process has already acquired the lock
Lock() error
// Unlock releases the acquired lock.
Unlock() error
// Fetcher provides means to obtain the encryption key
Fetcher() security.KeyFetcher
// Setter provides means to change the encryption key
Setter() security.KeySetter
}
}
31 changes: 31 additions & 0 deletions storage/postgres/security.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,40 @@ import (
"github.com/sirupsen/logrus"
)

const securityLockIndex = 111

type securityStorage struct {
db *sqlx.DB
encryptionKey []byte
isLocked bool
}

// Lock acquires a database lock so that only one process can manipulate the encryption key.
// Returns an error if the process has already acquired the lock
func (s *securityStorage) Lock() error {
if s.isLocked {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to not rely on a second inmemory lock and to directly rely on the response from the db query?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a mutex here to safeguard from future possible raceconditions

return fmt.Errorf("Lock is already acquired")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why error? The fact that the lock is locked, means its locked and if we decided that we do blocking select we should stick with that in all situations (even when in the same node)

}
_, err := s.db.Exec("SELECT pg_advisory_lock($1)", securityLockIndex)
if err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.rre this

return err
}
s.isLocked = true
return nil
}

// Unlock releases the database lock.
func (s *securityStorage) Unlock() error {
if !s.isLocked {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again i would prefer to drop the inmemory lock

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a mutex here to safeguard from future possible raceconditions

return nil
}

_, err := s.db.Exec("SELECT pg_advisory_unlock($1)", securityLockIndex)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.rre

if err != nil {
return err
}
s.isLocked = false
return nil
}

// Fetcher returns a KeyFetcher configured to fetch a key from the database
Expand Down
76 changes: 75 additions & 1 deletion storage/postgres/security_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,20 @@ import (
"crypto/rand"
"database/sql"
"fmt"
"time"
"testing"
"time"

"github.com/DATA-DOG/go-sqlmock"
"github.com/Peripli/service-manager/security"
"github.com/jmoiron/sqlx"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
func TestAPostgresStorage(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably it would make sense to add some tests in sm_test.go (talking to a real db and validating basic scenarios)

  1. failure occurs if the first key is missing from application.yml/env/flags
  2. things happen on startup if first key is present
    2.1. second key is generated
    2.2 locking works (lock twice, verify first is ok, second returns correct db response or waits and eventually when 1st unlocks, 2cd proceeds)

RegisterFailHandler(Fail)
RunSpecs(t, "Postgres Storage Suite")
}


var _ = Describe("Security", func() {

Expand Down Expand Up @@ -159,4 +165,72 @@ var _ = Describe("Security", func() {
})
})
})

Describe("Locking", func() {
var mockdb *sql.DB
var mock sqlmock.Sqlmock
var storage *securityStorage
envEncryptionKey := make([]byte, 32)

JustBeforeEach(func() {
storage = &securityStorage{
db: sqlx.NewDb(mockdb, "sqlmock"),
encryptionKey: envEncryptionKey,
isLocked: false,
}
})
BeforeEach(func() {
mockdb, mock, _ = sqlmock.New()
rand.Read(envEncryptionKey)
})
AfterEach(func() {
mockdb.Close()
})

Describe("Lock", func() {

Context("When lock is already acquired", func() {
It("Should return an error", func() {
storage.isLocked = true
err := storage.Lock()
Expect(err).ToNot(BeNil())
})
})

Context("When lock is not yet acquired", func() {
AfterEach(func() {
storage.Unlock()
})
BeforeEach(func() {
mock.ExpectExec("SELECT").WillReturnResult(sqlmock.NewResult(int64(1), int64(1)))
})
It("Should acquire lock", func() {
err := storage.Lock()
Expect(err).To(BeNil())
Expect(storage.isLocked).To(Equal(true))
})
})
})

Describe("Unlock", func() {
Context("When lock is not acquired", func() {
It("Should return nil", func() {
storage.isLocked = false
err := storage.Unlock()
Expect(err).To(BeNil())
})
})
Context("When lock is acquired", func() {
BeforeEach(func() {
mock.ExpectExec("SELECT").WillReturnResult(sqlmock.NewResult(int64(1), int64(1)))
})
It("Should release lock", func() {
storage.isLocked = true
err := storage.Unlock()
Expect(err).To(BeNil())
Expect(storage.isLocked).To(Equal(false))
})
})
})
})
})
2 changes: 1 addition & 1 deletion storage/postgres/storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ func (storage *postgresStorage) Credentials() storage.Credentials {

func (storage *postgresStorage) Security() storage.Security{
storage.checkOpen()
return &securityStorage{storage.db, storage.encryptionKey}
return &securityStorage{storage.db, storage.encryptionKey, false}
}

func (storage *postgresStorage) Open(uri string, encryptionKey []byte) error {
Expand Down