Merge branch 'main' into computed-insert

CHANGELOG.md

@@ -27,6 +27,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
  - #3327, Fix slow responses on schema cache reloads - @steve-chavez
  - #3340, Log when the LISTEN channel gets a notification - @steve-chavez
  - #3345, Fix in-database configuration values not loading for `pgrst.server_trace_header` and `pgrst.server_cors_allowed_origins` - @laurenceisla
+ - #3361, Clarify PGRST204(column not found) error message - @steve-chavez
 
 ### Deprecated
 

docs/references/errors.rst

@@ -259,7 +259,7 @@ Related to the HTTP request elements.
 Group 2 - Schema Cache
 ~~~~~~~~~~~~~~~~~~~~~~
 
-Related to a :ref:`stale schema cache <stale_schema>`. Most of the time, these errors are solved by :ref:`reloading the schema cache <schema_reloading>`.
+Related to a :ref:`schema_cache`. Most of the time, these errors are solved by :ref:`schema_reloading`.
 
 +---------------+-------------+-------------------------------------------------------------+
 | Code | HTTP status | Description |

docs/references/schema_cache.rst

@@ -3,46 +3,29 @@
 Schema Cache
 ============
 
-Some PostgREST features need metadata from the database schema. Getting this metadata requires expensive queries. To avoid repeating this work, PostgREST uses a schema cache.
-
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| Feature | Required Metadata |
-+============================================+===============================================================================+
-| :ref:`resource_embedding` | Foreign key constraints |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`Functions <functions>` | Function signature (parameters, return type, volatility and |
-| | `overloading <https://www.postgresql.org/docs/current/xfunc-overload.html>`_) |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`Upserts <upsert>` | Primary keys |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`Insertions <insert>` | Primary keys (optional: only if the Location header is requested) |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`OPTIONS requests <options_requests>` | View INSTEAD OF TRIGGERS and primary keys |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`open-api` | Table columns, primary keys and foreign keys |
-+ +-------------------------------------------------------------------------------+
-| | View columns and INSTEAD OF TRIGGERS |
-+ +-------------------------------------------------------------------------------+
-| | Function signature |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-
-.. _stale_schema:
-
-Stale Schema Cache
-------------------
-
-One operational problem that comes with a cache is that it can go stale. This can happen for PostgREST when you make changes to the metadata before mentioned. Requests that depend on the metadata will fail.
-
-You can solve this by reloading the cache manually or automatically.
+PostgREST requires metadata from the database schema to provide a REST API that abstracts SQL details. One example of this is the interface for :ref:`resource_embedding`.
 
-.. note::
- If you are using :ref:`in_db_config`, a schema reload will always :ref:`reload the configuration<config_reloading>` as well.
+Getting this metadata requires expensive queries. To avoid repeating this work, PostgREST uses a schema cache.
 
 .. _schema_reloading:
 
 Schema Cache Reloading
 ----------------------
 
+To not let the schema cache go stale (happens when you make changes to the database), you need to reload it.
+
+You can do this with UNIX signals or with PostgreSQL notifications. It's also possible to do this automatically using `event triggers <https://www.postgresql.org/docs/current/event-trigger-definition.html>`_.
+
+.. note::
+
+ - If you are using the :ref:`in_db_config`, a schema cache reload will :ref:`reload the configuration<config_reloading>` as well.
+ - There’s no downtime when reloading the schema cache. The reloading will happen on a background thread while serving requests.
+
+.. _schema_reloading_signals:
+
+Schema Cache Reloading with Unix Signals
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 To manually reload the cache without restarting the PostgREST server, send a SIGUSR1 signal to the server process.
 
 .. code:: bash
@@ -59,8 +42,6 @@ For docker you can do:
  # or in docker-compose
  docker-compose kill -s SIGUSR1 <service>
 
-There’s no downtime when reloading the schema cache. The reloading will happen on a background thread while serving requests.
-
 .. _schema_reloading_notify:
 
 Schema Cache Reloading with NOTIFY
@@ -81,7 +62,7 @@ The ``pgrst`` notification channel is enabled by default. For configuring the ch
 Automatic Schema Cache Reloading
 --------------------------------
 
-You can do automatic schema cache reloading in a pure SQL way and forget about stale schema cache errors. For this use an `event trigger <https://www.postgresql.org/docs/current/event-trigger-definition.html>`_ and ``NOTIFY``.
+You can do automatic reloading and forget there is a schema cache. For this use an `event trigger <https://www.postgresql.org/docs/current/event-trigger-definition.html>`_ and ``NOTIFY``.
 
 .. code-block:: postgres
 

nix/tools/docs.nix

@@ -59,6 +59,24 @@ let
  '';
 
  render =
+ let
+ pdflatex = texlive.combine {
+ inherit (texlive)
+ amsmath
+ booktabs
+ cancel
+ gensymb
+ mathdots
+ multirow
+ pgf
+ pgf-blur
+ scheme-basic
+ siunitx
+ standalone
+ yhmath
+ ;
+ };
+ in
  checkedShellScript
  {
  name = "postgrest-docs-render";
@@ -67,7 +85,7 @@ let
  withTmpDir = true;
  }
  ''
- ${texlive.combined.scheme-full}/bin/pdflatex -halt-on-error -output-directory="$tmpdir" db.tex
+ ${pdflatex}/bin/pdflatex -halt-on-error -output-directory="$tmpdir" db.tex
  ${imagemagick}/bin/convert -density 300 "$tmpdir/db.pdf" ../_static/db.png
  '';
 

src/PostgREST/ApiRequest.hs

@@ -95,10 +95,10 @@ data DbAction
  = ActRelationRead {dbActQi :: QualifiedIdentifier, actHeadersOnly :: Bool}
  | ActRelationMut {dbActQi :: QualifiedIdentifier, actMutation :: Mutation}
  | ActRoutine {dbActQi :: QualifiedIdentifier, actInvMethod :: InvokeMethod}
+ | ActSchemaRead Schema Bool
 
 data Action
  = ActDb DbAction
- | ActSchemaRead Schema Bool
  | ActRelationInfo QualifiedIdentifier
  | ActRoutineInfo QualifiedIdentifier
  | ActSchemaInfo
@@ -190,8 +190,8 @@ getAction resource schema method =
  (ResourceRelation rel, "DELETE") -> Right . ActDb $ ActRelationMut (qi rel) MutationDelete
  (ResourceRelation rel, "OPTIONS") -> Right $ ActRelationInfo (qi rel)
 
- (ResourceSchema, "HEAD") -> Right $ ActSchemaRead schema True
- (ResourceSchema, "GET") -> Right $ ActSchemaRead schema False
+ (ResourceSchema, "HEAD") -> Right . ActDb $ ActSchemaRead schema True
+ (ResourceSchema, "GET") -> Right . ActDb $ ActSchemaRead schema False
  (ResourceSchema, "OPTIONS") -> Right ActSchemaInfo
 
  _ -> Left $ UnsupportedMethod method

src/PostgREST/App.hs

@@ -9,7 +9,6 @@ Some of its functionality includes:
 - Producing HTTP Headers according to RFCs.
 - Content Negotiation
 -}
-{-# LANGUAGE NamedFieldPuns #-}
 {-# LANGUAGE RecordWildCards #-}
 module PostgREST.App
  ( postgrest
@@ -24,7 +23,6 @@ import Data.String (IsString (..))
 import Network.Wai.Handler.Warp (defaultSettings, setHost, setPort,
  setServerName)
 
-import qualified Data.HashMap.Strict as HM
 import qualified Data.Text.Encoding as T
 import qualified Hasql.Transaction.Sessions as SQL
 import qualified Network.Wai as Wai
@@ -54,7 +52,6 @@ import PostgREST.Query (DbHandler)
 import PostgREST.Response.Performance (ServerTiming (..),
  serverTimingHeader)
 import PostgREST.SchemaCache (SchemaCache (..))
-import PostgREST.SchemaCache.Routine (Routine (..))
 import PostgREST.Version (docsVersion, prettyVersion)
 
 import qualified Data.ByteString.Char8 as BS
@@ -172,14 +169,11 @@ handleRequest AuthResult{..} conf appState authenticated prepared pgVer apiReq@A
  case iAction of
  ActDb dbAct -> do
  (planTime', plan) <- withTiming $ liftEither $ Plan.actionPlan dbAct conf apiReq sCache
- (txTime', resultSet) <- withTiming $ runQuery (planIsoLvl plan) (planFunSettings plan) (Plan.pTxMode plan) $ Query.actionQuery plan conf apiReq pgVer
- (respTime', pgrst) <- withTiming $ liftEither $ Response.actionResponse plan (dbActQi dbAct) apiReq resultSet
- return $ pgrstResponse (ServerTiming jwtTime parseTime planTime' txTime' respTime') pgrst
-
- ActSchemaRead tSchema headersOnly -> do
- (planTime', iPlan) <- withTiming $ liftEither $ Plan.inspectPlan apiReq headersOnly tSchema
- (txTime', oaiResult) <- withTiming $ runQuery roleIsoLvl mempty (Plan.ipTxmode iPlan) $ Query.openApiQuery iPlan conf sCache pgVer
- (respTime', pgrst) <- withTiming $ liftEither $ Response.openApiResponse iPlan (T.decodeUtf8 prettyVersion, docsVersion) oaiResult conf sCache iSchema iNegotiatedByProfile
+ (txTime', queryResult) <- withTiming $ runDbHandler appState conf (Plan.planIsoLvl conf authRole plan) (Plan.planTxMode plan) authenticated prepared observer $ do
+ Query.setPgLocals plan conf authClaims authRole apiReq
+ Query.runPreReq conf
+ Query.actionQuery plan conf apiReq pgVer sCache
+ (respTime', pgrst) <- withTiming $ liftEither $ Response.actionResponse queryResult (dbActQi dbAct) apiReq (T.decodeUtf8 prettyVersion, docsVersion) conf sCache iSchema iNegotiatedByProfile
  return $ pgrstResponse (ServerTiming jwtTime parseTime planTime' txTime' respTime') pgrst
 
  ActRelationInfo identifier -> do
@@ -196,20 +190,6 @@ handleRequest AuthResult{..} conf appState authenticated prepared pgVer apiReq@A
  return $ pgrstResponse (ServerTiming jwtTime parseTime Nothing Nothing respTime') pgrst
 
  where
- roleSettings = fromMaybe mempty (HM.lookup authRole $ configRoleSettings conf)
- roleIsoLvl = HM.findWithDefault SQL.ReadCommitted authRole $ configRoleIsoLvl conf
- runQuery isoLvl funcSets mode query =
- runDbHandler appState conf isoLvl mode authenticated prepared observer $ do
- Query.setPgLocals conf authClaims authRole (HM.toList roleSettings) funcSets apiReq
- Query.runPreReq conf
- query
-
- planIsoLvl (Plan.CallReadPlan{crProc}) = fromMaybe roleIsoLvl $ pdIsoLvl crProc
- planIsoLvl _ = roleIsoLvl
-
- planFunSettings (Plan.CallReadPlan{crProc}) = pdFuncSettings crProc
- planFunSettings _ = mempty
-
  pgrstResponse :: ServerTiming -> Response.PgrstResponse -> Wai.Response
  pgrstResponse timing (Response.PgrstResponse st hdrs bod) = Wai.responseLBS st (hdrs ++ ([serverTimingHeader timing | configServerTimingEnabled conf])) bod
 

src/PostgREST/Error.hs

@@ -249,7 +249,7 @@ instance JSON.ToJSON ApiRequestError where
  (Just "Try renaming the parameters or the function itself in the database so function overloading can be resolved")
 
  toJSON (ColumnNotFound relName colName) = toJsonPgrstError
- SchemaCacheErrorCode04 ("Column '" <> colName <> "' of relation '" <> relName <> "' does not exist") Nothing Nothing
+ SchemaCacheErrorCode04 ("Could not find the '" <> colName <> "' column of '" <> relName <> "' in the schema cache") Nothing Nothing
 
 -- |
 -- If no relationship is found then:

src/PostgREST/Plan.hs

@@ -18,9 +18,12 @@ resource.
 module PostgREST.Plan
  ( actionPlan
  , ActionPlan(..)
+ , DbActionPlan(..)
  , InspectPlan(..)
  , inspectPlan
  , callReadPlan
+ , planTxMode
+ , planIsoLvl
  ) where
 
 import qualified Data.ByteString.Lazy as LBS
@@ -90,7 +93,7 @@ import Protolude hiding (from)
 -- Setup for doctests
 -- >>> import Data.Ranged.Ranges (fullRange)
 
-data ActionPlan
+data DbActionPlan
  = WrappedReadPlan
  { wrReadPlan :: ReadPlanTree
  , pTxMode :: SQL.Mode
@@ -123,31 +126,46 @@ data InspectPlan = InspectPlan {
  , ipSchema :: Schema
  }
 
+data ActionPlan = Db DbActionPlan | MaybeDb InspectPlan
+
+planTxMode :: ActionPlan -> SQL.Mode
+planTxMode (Db x) = pTxMode x
+planTxMode (MaybeDb x) = ipTxmode x
+
+planIsoLvl :: AppConfig -> ByteString -> ActionPlan -> SQL.IsolationLevel
+planIsoLvl AppConfig{configRoleIsoLvl} role actPlan = case actPlan of
+ Db CallReadPlan{crProc} -> fromMaybe roleIsoLvl $ pdIsoLvl crProc
+ _ -> roleIsoLvl
+ where
+ roleIsoLvl = HM.findWithDefault SQL.ReadCommitted role configRoleIsoLvl
+
 actionPlan :: DbAction -> AppConfig -> ApiRequest -> SchemaCache -> Either Error ActionPlan
 actionPlan dbAct conf apiReq sCache = case dbAct of
  ActRelationRead identifier headersOnly ->
- wrappedReadPlan identifier conf sCache apiReq headersOnly
+ Db <$> wrappedReadPlan identifier conf sCache apiReq headersOnly
  ActRelationMut identifier mut ->
- mutateReadPlan mut apiReq identifier conf sCache
+ Db <$> mutateReadPlan mut apiReq identifier conf sCache
  ActRoutine identifier invMethod ->
- callReadPlan identifier conf sCache apiReq invMethod
+ Db <$> callReadPlan identifier conf sCache apiReq invMethod
+ ActSchemaRead tSchema headersOnly ->
+ MaybeDb <$> inspectPlan apiReq headersOnly tSchema
 
-wrappedReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> Bool -> Either Error ActionPlan
+wrappedReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> Bool -> Either Error DbActionPlan
 wrappedReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{..},..} headersOnly = do
  rPlan <- readPlan identifier conf sCache apiRequest
  (handler, mediaType) <- mapLeft ApiRequestError $ negotiateContent conf apiRequest identifier iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan)
  if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right ()
  return $ WrappedReadPlan rPlan SQL.Read handler mediaType headersOnly
 
-mutateReadPlan :: Mutation -> ApiRequest -> QualifiedIdentifier -> AppConfig -> SchemaCache -> Either Error ActionPlan
+mutateReadPlan :: Mutation -> ApiRequest -> QualifiedIdentifier -> AppConfig -> SchemaCache -> Either Error DbActionPlan
 mutateReadPlan mutation apiRequest@ApiRequest{iPreferences=Preferences{..},..} identifier conf sCache = do
  rPlan <- readPlan identifier conf sCache apiRequest
  mPlan <- mutatePlan mutation identifier apiRequest sCache rPlan
  if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right ()
  (handler, mediaType) <- mapLeft ApiRequestError $ negotiateContent conf apiRequest identifier iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan)
  return $ MutateReadPlan rPlan mPlan SQL.Write handler mediaType mutation
 
-callReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> InvokeMethod -> Either Error ActionPlan
+callReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> InvokeMethod -> Either Error DbActionPlan
 callReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{..},..} invMethod = do
  let paramKeys = case invMethod of
  InvRead _ -> S.fromList $ fst <$> qsParams'