Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Do not review] Single resource #4787

Open
wants to merge 118 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
47c370f
Raw resources in the Lake and Resource Id integer map
SergeyGaluzo Sep 27, 2024
5aca029
perf tweeks
SergeyGaluzo Oct 2, 2024
de5a156
Merge history separation with separated raw resources (#4685)
SergeyGaluzo Oct 17, 2024
e37ffd4
Merge resource id map
SergeyGaluzo Oct 17, 2024
5d8d03a
Fix after merge
SergeyGaluzo Oct 18, 2024
cc94008
Fix after merge
SergeyGaluzo Oct 18, 2024
d6ce380
adjustments
SergeyGaluzo Oct 18, 2024
2950bcf
tests
SergeyGaluzo Oct 18, 2024
8e74731
Merge from main
SergeyGaluzo Oct 18, 2024
c3ccf69
Merge fixes
SergeyGaluzo Oct 18, 2024
b0ed1eb
Ignore NULL reference resource tyoe
SergeyGaluzo Oct 18, 2024
51a942d
test fix
SergeyGaluzo Oct 18, 2024
38e869b
default number of columns = 13
SergeyGaluzo Oct 19, 2024
bb535af
commented 84 diff
SergeyGaluzo Oct 19, 2024
7060321
set end date on enqueue
SergeyGaluzo Oct 22, 2024
427cdfa
rename secondary to adls
SergeyGaluzo Oct 22, 2024
02a545f
defrag formatting
SergeyGaluzo Oct 27, 2024
fbbaf20
set timeout to 0 for getting frag
SergeyGaluzo Oct 28, 2024
3f5dcd9
isnull on input param
SergeyGaluzo Oct 28, 2024
e6e3fe9
Merge branch 'main' into users/sergal/rawmaplakeandhistoryandresource…
SergeyGaluzo Oct 29, 2024
5569ea1
after merge from main
SergeyGaluzo Oct 30, 2024
2648ecd
Merge branch 'users/sergal/rawmaplakeandhistoryandresourceidmap' into…
SergeyGaluzo Oct 31, 2024
037061d
Reference separation
SergeyGaluzo Oct 31, 2024
99c6620
file id
SergeyGaluzo Nov 1, 2024
0b226bf
small things
SergeyGaluzo Nov 1, 2024
7f286f9
Do not write raw resource
SergeyGaluzo Nov 7, 2024
873583e
Added missing file id in history insert
SergeyGaluzo Nov 18, 2024
ce28891
hard delete tests
SergeyGaluzo Nov 19, 2024
848195c
change capture
SergeyGaluzo Nov 19, 2024
61b1c06
FK
SergeyGaluzo Nov 19, 2024
65b21dc
Comments
SergeyGaluzo Nov 19, 2024
2b28ed4
resource list lake and 768
SergeyGaluzo Nov 20, 2024
6c46e43
no lake on list
SergeyGaluzo Nov 20, 2024
2d8a618
no extra lines
SergeyGaluzo Nov 20, 2024
38e6c45
small cleanup
SergeyGaluzo Nov 21, 2024
90132fb
84 in sync
SergeyGaluzo Nov 21, 2024
77d4950
diff
SergeyGaluzo Nov 22, 2024
7f17c96
GetResources back
SergeyGaluzo Nov 25, 2024
8b660a0
Fix diff
SergeyGaluzo Dec 4, 2024
9992090
Use ResourceListLake correctly
SergeyGaluzo Dec 4, 2024
daa5c53
Added @ResourceListLake to update resource search params
SergeyGaluzo Dec 5, 2024
cfa461c
tweeking
SergeyGaluzo Dec 11, 2024
5373db8
fix diff
SergeyGaluzo Dec 11, 2024
ad7576a
resourceids
SergeyGaluzo Dec 11, 2024
8867758
filter out null ref resourec type
SergeyGaluzo Dec 12, 2024
e56f599
handle search params updates in parallel with data move
SergeyGaluzo Dec 12, 2024
c59a7ce
Added id join in include queries
SergeyGaluzo Dec 12, 2024
83c812a
84 diff fix
SergeyGaluzo Dec 13, 2024
3019c0f
no CR on IsResourceRef
SergeyGaluzo Dec 14, 2024
46892b1
Removed not needed alias
SergeyGaluzo Dec 14, 2024
4d65b4d
no alias
SergeyGaluzo Dec 15, 2024
73d02be
fixed RespurceIns trigger
SergeyGaluzo Dec 16, 2024
4bbb393
Add ResourceTbl history update into temporary ResourceUpd trigger.
SergeyGaluzo Dec 17, 2024
a79c4c3
Final hard delete logic
SergeyGaluzo Dec 17, 2024
730260b
final delete invisible history
SergeyGaluzo Dec 17, 2024
9197191
DataLakeStore flag
SergeyGaluzo Dec 17, 2024
4183235
names
SergeyGaluzo Dec 18, 2024
0347536
names
SergeyGaluzo Dec 18, 2024
ba9415b
Current resource view
SergeyGaluzo Dec 18, 2024
88f831b
nicer
SergeyGaluzo Dec 18, 2024
039a148
spelling
SergeyGaluzo Dec 18, 2024
37b83b0
84 -> 85
SergeyGaluzo Dec 18, 2024
b975e1a
85
SergeyGaluzo Dec 18, 2024
a7f7ea3
merge from CurrentResource branch
SergeyGaluzo Dec 18, 2024
2f34299
85
SergeyGaluzo Dec 18, 2024
0389733
Changed order of columns for clustered key on resource tables to supp…
SergeyGaluzo Dec 20, 2024
826d08b
DELETE inside loop
SergeyGaluzo Dec 20, 2024
bd453b2
old views always and proc to move
SergeyGaluzo Dec 20, 2024
cb5c967
Merge from main after 84
SergeyGaluzo Dec 20, 2024
e6c5268
optimized stored procedure
SergeyGaluzo Dec 21, 2024
f5764a4
export
SergeyGaluzo Dec 22, 2024
cde0918
Simple sort value logic
SergeyGaluzo Dec 23, 2024
5840834
plus
SergeyGaluzo Dec 23, 2024
c9450ff
Merge from filed count branch
SergeyGaluzo Dec 23, 2024
22174f7
11->12 plus comments
SergeyGaluzo Dec 23, 2024
ff69585
null check
SergeyGaluzo Dec 24, 2024
a8b7de2
adls message
SergeyGaluzo Dec 24, 2024
53d5930
message
SergeyGaluzo Dec 24, 2024
ccdc3e5
log
SergeyGaluzo Dec 24, 2024
8b35734
2 columns back
SergeyGaluzo Dec 25, 2024
b17416b
exporter with file/blob API
SergeyGaluzo Dec 26, 2024
567b839
group read
SergeyGaluzo Dec 26, 2024
d8fc0d7
Accomodate CurrentResource view
SergeyGaluzo Dec 27, 2024
0f00915
Switched search to set file reads
SergeyGaluzo Dec 27, 2024
1a8d6d4
moved large logic outside of reader
SergeyGaluzo Dec 27, 2024
ba9bfc0
refactored to use set reads
SergeyGaluzo Dec 27, 2024
0b633b3
alarms
SergeyGaluzo Dec 27, 2024
b751b14
Corrected != null on SqlBytes
SergeyGaluzo Dec 27, 2024
24bcfaf
removed redundant select
SergeyGaluzo Dec 27, 2024
7d5368e
check for adls client is null after returning empty
SergeyGaluzo Dec 28, 2024
3db2a4d
name typo
SergeyGaluzo Dec 28, 2024
e5f833f
Removed file system
SergeyGaluzo Dec 28, 2024
747e1d9
Hard delete check for invisible resource
SergeyGaluzo Dec 30, 2024
dad9d90
hard delete fix
SergeyGaluzo Jan 1, 2025
890d5ca
move alter inside try/catch
SergeyGaluzo Jan 2, 2025
d00ee7d
Added ResourceTbl to update trigger
SergeyGaluzo Jan 3, 2025
7212ebc
Added diag
SergeyGaluzo Jan 3, 2025
55b4bba
Added checks of Tbls
SergeyGaluzo Jan 3, 2025
9a01079
sending zero raw resource for reindex
SergeyGaluzo Jan 3, 2025
6983e72
Configurable sleep
SergeyGaluzo Jan 3, 2025
5dfcfac
constraint name correction
SergeyGaluzo Jan 5, 2025
88cbc06
rebatched
SergeyGaluzo Jan 6, 2025
daa1ad9
removed commit with failure call
SergeyGaluzo Jan 6, 2025
3913db5
Removed redundant tostring
SergeyGaluzo Jan 9, 2025
a16381d
ensure not null
SergeyGaluzo Jan 9, 2025
806f877
more ensure
SergeyGaluzo Jan 9, 2025
b2e2d7e
more ensure
SergeyGaluzo Jan 9, 2025
49bbdbd
import tests with raw in lake
SergeyGaluzo Jan 10, 2025
5f44cc7
minus data lake
SergeyGaluzo Jan 12, 2025
99a8c7a
minus data lake
SergeyGaluzo Jan 12, 2025
609ff9e
Removed not used stored proc
SergeyGaluzo Jan 13, 2025
983a5da
reverted formatting change
SergeyGaluzo Jan 13, 2025
feb8562
comments
SergeyGaluzo Jan 13, 2025
a309c92
Removed item2
SergeyGaluzo Jan 13, 2025
f850188
Cient -> Client
SergeyGaluzo Jan 13, 2025
24430d0
removed local var
SergeyGaluzo Jan 13, 2025
9e1d8a6
revert disable indexes
SergeyGaluzo Jan 14, 2025
02c47a5
Single resource per blob
SergeyGaluzo Jan 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3,928 changes: 3,928 additions & 0 deletions src/Microsoft.Health.Fhir.SqlServer/Features/Schema/Migrations/85.diff.sql

Large diffs are not rendered by default.

6,360 changes: 6,360 additions & 0 deletions src/Microsoft.Health.Fhir.SqlServer/Features/Schema/Migrations/85.sql

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -94,5 +94,6 @@ public enum SchemaVersion
V82 = 82,
V83 = 83,
V84 = 84,
V85 = 85,
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ namespace Microsoft.Health.Fhir.SqlServer.Features.Schema
{
public static class SchemaVersionConstants
{
public const int Min = (int)SchemaVersion.V83;
public const int Max = (int)SchemaVersion.V84;
public const int Min = (int)SchemaVersion.V84;
public const int Max = (int)SchemaVersion.V85;
public const int MinForUpgrade = (int)SchemaVersion.V83; // this is used for upgrade tests only
public const int SearchParameterStatusSchemaVersion = (int)SchemaVersion.V6;
public const int SupportForReferencesWithMissingTypeVersion = (int)SchemaVersion.V7;
Expand All @@ -33,6 +33,7 @@ public static class SchemaVersionConstants
public const int Merge = (int)SchemaVersion.V50;
public const int IncrementalImport = (int)SchemaVersion.V53;
public const int LakePrerequisite = (int)SchemaVersion.V84;
public const int Lake = (int)SchemaVersion.V85;

// It is currently used in Azure Healthcare APIs.
public const int ParameterizedRemovePartitionFromResourceChangesVersion = (int)SchemaVersion.V21;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,12 @@ CREATE SEQUENCE dbo.ResourceSurrogateIdUniquifierSequence
CYCLE
CACHE 1000000
GO
CREATE SEQUENCE dbo.ResourceIdIntMapSequence
AS int
START WITH 0
INCREMENT BY 1
MINVALUE 0
MAXVALUE 79999
CYCLE
CACHE 1000000
GO
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ Go

INSERT INTO dbo.SchemaVersion
VALUES
(84, 'started')
(85, 'started')

Go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
CREATE PROCEDURE dbo.AssignResourceIdInts @Count int, @FirstIdInt bigint OUT
AS
set nocount on
DECLARE @SP varchar(100) = 'AssignResourceIdInts'
,@Mode varchar(200) = 'Cnt='+convert(varchar,@Count)
,@st datetime = getUTCdate()
,@FirstValueVar sql_variant
,@LastValueVar sql_variant
,@SequenceRangeFirstValue int

BEGIN TRY
SET @FirstValueVar = NULL
WHILE @FirstValueVar IS NULL
BEGIN
EXECUTE sys.sp_sequence_get_range @sequence_name = 'dbo.ResourceIdIntMapSequence', @range_size = @Count, @range_first_value = @FirstValueVar OUT, @range_last_value = @LastValueVar OUT
SET @SequenceRangeFirstValue = convert(int,@FirstValueVar)
IF @SequenceRangeFirstValue > convert(int,@LastValueVar)
SET @FirstValueVar = NULL
END

SET @FirstIdInt = datediff_big(millisecond,'0001-01-01',sysUTCdatetime()) * 80000 + @SequenceRangeFirstValue
END TRY
BEGIN CATCH
IF error_number() = 1750 THROW -- Real error is before 1750, cannot trap in SQL.
IF @@trancount > 0 ROLLBACK TRANSACTION
EXECUTE dbo.LogEvent @Process=@SP,@Mode=@Mode,@Status='Error';
THROW
END CATCH
GO
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
CREATE PROCEDURE dbo.CaptureResourceIdsForChanges @Resources dbo.ResourceList READONLY
CREATE PROCEDURE dbo.CaptureResourceIdsForChanges @Resources dbo.ResourceList READONLY, @ResourcesLake dbo.ResourceListLake READONLY
AS
set nocount on
-- This procedure is intended to be called from the MergeResources procedure and relies on its transaction logic
INSERT INTO dbo.ResourceChangeData
( ResourceId, ResourceTypeId, ResourceVersion, ResourceChangeTypeId )
SELECT ResourceId, ResourceTypeId, Version, CASE WHEN IsDeleted = 1 THEN 2 WHEN Version > 1 THEN 1 ELSE 0 END
FROM @Resources
FROM (SELECT ResourceId, ResourceTypeId, Version, IsHistory, IsDeleted FROM @Resources UNION ALL SELECT ResourceId, ResourceTypeId, Version, IsHistory, IsDeleted FROM @ResourcesLake) A
WHERE IsHistory = 0
GO
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,17 @@ BEGIN TRY
END
,MatchedVersion = isnull(D.Version,0)
,MatchedRawResource = D.RawResource
,MatchedFileId = D.FileId
,MatchedOffsetInFile = D.OffsetInFile
-- ResourceIndex allows to deal with more than one late arrival per resource
FROM (SELECT TOP (@DummyTop) *, ResourceIndex = convert(int,row_number() OVER (PARTITION BY ResourceTypeId, ResourceId ORDER BY ResourceSurrogateId DESC)) FROM @ResourceDateKeys) A
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId_Version) WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId AND B.Version > 0 AND B.ResourceSurrogateId < A.ResourceSurrogateId ORDER BY B.ResourceSurrogateId DESC) L -- lower
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId_Version) WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId AND B.Version > 0 AND B.ResourceSurrogateId > A.ResourceSurrogateId ORDER BY B.ResourceSurrogateId) U -- upper
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId_Version) WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId AND B.Version < 0 ORDER BY B.Version) M -- minus
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId_Version) WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId AND B.ResourceSurrogateId BETWEEN A.ResourceSurrogateId AND A.ResourceSurrogateId + 79999) D -- date
FROM (SELECT TOP (@DummyTop) A.*, M.ResourceIdInt, ResourceIndex = convert(int,row_number() OVER (PARTITION BY A.ResourceTypeId, A.ResourceId ORDER BY ResourceSurrogateId DESC))
FROM @ResourceDateKeys A
LEFT OUTER JOIN dbo.ResourceIdIntMap M WITH (INDEX = U_ResourceIdIntMap_ResourceId_ResourceTypeId) ON M.ResourceTypeId = A.ResourceTypeId AND M.ResourceId = A.ResourceId
) A
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceIdInt = A.ResourceIdInt AND B.Version > 0 AND B.ResourceSurrogateId < A.ResourceSurrogateId ORDER BY B.ResourceSurrogateId DESC) L -- lower
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceIdInt = A.ResourceIdInt AND B.Version > 0 AND B.ResourceSurrogateId > A.ResourceSurrogateId ORDER BY B.ResourceSurrogateId) U -- upper
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceIdInt = A.ResourceIdInt AND B.Version < 0 ORDER BY B.Version) M -- minus
OUTER APPLY (SELECT TOP 1 * FROM dbo.Resource B WHERE B.ResourceTypeId = A.ResourceTypeId AND B.ResourceIdInt = A.ResourceIdInt AND B.ResourceSurrogateId BETWEEN A.ResourceSurrogateId AND A.ResourceSurrogateId + 79999) D -- date
OPTION (MAXDOP 1, OPTIMIZE FOR (@DummyTop = 1))

EXECUTE dbo.LogEvent @Process=@SP,@Mode=@Mode,@Status='End',@Start=@st,@Rows=@@rowcount
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ set nocount on
DECLARE @st datetime = getUTCdate()
,@SP varchar(100) = 'GetResources'
,@InputRows int
,@DummyTop bigint = 9223372036854775807
,@NotNullVersionExists bit
,@NullVersionExists bit
,@MinRT smallint
Expand All @@ -22,57 +21,69 @@ BEGIN TRY
SELECT B.ResourceTypeId
,B.ResourceId
,ResourceSurrogateId
,B.Version
,C.Version
,IsDeleted
,IsHistory
,RawResource
,IsRawResourceMetaSet
,SearchParamHash
FROM (SELECT TOP (@DummyTop) * FROM @ResourceKeys) A
JOIN dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId_Version) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId AND B.Version = A.Version
OPTION (MAXDOP 1, OPTIMIZE FOR (@DummyTop = 1))
,FileId
,OffsetInFile
FROM (SELECT * FROM @ResourceKeys) A
INNER LOOP JOIN dbo.ResourceIdIntMap B WITH (INDEX = U_ResourceIdIntMap_ResourceId_ResourceTypeId) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId
INNER LOOP JOIN dbo.Resource C ON C.ResourceTypeId = A.ResourceTypeId AND C.ResourceIdInt = B.ResourceIdInt AND C.Version = A.Version
OPTION (MAXDOP 1)
ELSE
SELECT *
FROM (SELECT B.ResourceTypeId
,B.ResourceId
,ResourceSurrogateId
,B.Version
,C.Version
,IsDeleted
,IsHistory
,RawResource
,IsRawResourceMetaSet
,SearchParamHash
FROM (SELECT TOP (@DummyTop) * FROM @ResourceKeys WHERE Version IS NOT NULL) A
JOIN dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId_Version) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId AND B.Version = A.Version
,FileId
,OffsetInFile
FROM (SELECT * FROM @ResourceKeys WHERE Version IS NOT NULL) A
INNER LOOP JOIN dbo.ResourceIdIntMap B WITH (INDEX = U_ResourceIdIntMap_ResourceId_ResourceTypeId) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId
INNER LOOP JOIN dbo.Resource C ON C.ResourceTypeId = A.ResourceTypeId AND C.ResourceIdInt = B.ResourceIdInt AND C.Version = A.Version
UNION ALL
SELECT B.ResourceTypeId
,B.ResourceId
,ResourceSurrogateId
,B.Version
,C.ResourceSurrogateId
,C.Version
,IsDeleted
,IsHistory
,RawResource
,IsRawResourceMetaSet
,SearchParamHash
FROM (SELECT TOP (@DummyTop) * FROM @ResourceKeys WHERE Version IS NULL) A
JOIN dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId
WHERE IsHistory = 0
,FileId
,OffsetInFile
FROM (SELECT * FROM @ResourceKeys WHERE Version IS NULL) A
INNER LOOP JOIN dbo.ResourceIdIntMap B WITH (INDEX = U_ResourceIdIntMap_ResourceId_ResourceTypeId) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId
INNER LOOP JOIN dbo.CurrentResources C ON C.ResourceTypeId = A.ResourceTypeId AND C.ResourceIdInt = B.ResourceIdInt AND C.IsHistory = 0
LEFT OUTER JOIN dbo.RawResources D ON D.ResourceTypeId = A.ResourceTypeId AND D.ResourceSurrogateId = C.ResourceSurrogateId
) A
OPTION (MAXDOP 1, OPTIMIZE FOR (@DummyTop = 1))
OPTION (MAXDOP 1)
ELSE
SELECT B.ResourceTypeId
,B.ResourceId
,ResourceSurrogateId
,B.Version
,C.ResourceSurrogateId
,C.Version
,IsDeleted
,IsHistory
,RawResource
,IsRawResourceMetaSet
,SearchParamHash
FROM (SELECT TOP (@DummyTop) * FROM @ResourceKeys) A
JOIN dbo.Resource B WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId
WHERE IsHistory = 0
OPTION (MAXDOP 1, OPTIMIZE FOR (@DummyTop = 1))
,FileId
,OffsetInFile
FROM (SELECT * FROM @ResourceKeys) A
INNER LOOP JOIN dbo.ResourceIdIntMap B WITH (INDEX = U_ResourceIdIntMap_ResourceId_ResourceTypeId) ON B.ResourceTypeId = A.ResourceTypeId AND B.ResourceId = A.ResourceId
INNER LOOP JOIN dbo.CurrentResources C ON C.ResourceTypeId = A.ResourceTypeId AND C.ResourceIdInt = B.ResourceIdInt
LEFT OUTER JOIN dbo.RawResources D ON D.ResourceTypeId = A.ResourceTypeId AND D.ResourceSurrogateId = C.ResourceSurrogateId
OPTION (MAXDOP 1)

EXECUTE dbo.LogEvent @Process=@SP,@Mode=@Mode,@Status='End',@Start=@st,@Rows=@@rowcount
END TRY
Expand All @@ -83,5 +94,5 @@ BEGIN CATCH
END CATCH
GO
--DECLARE @ResourceKeys dbo.ResourceKeyList
--INSERT INTO @ResourceKeys SELECT TOP 1 ResourceTypeId, ResourceId, NULL FROM Resource
--INSERT INTO @ResourceKeys SELECT 96, newid(), NULL
--EXECUTE dbo.GetResources @ResourceKeys
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,8 @@ set nocount on
DECLARE @SP varchar(100) = object_name(@@procid)
,@Mode varchar(100) = 'T='+convert(varchar,@TransactionId)+' H='+convert(varchar,@IncludeHistory)
,@st datetime = getUTCdate()
,@DummyTop bigint = 9223372036854775807
,@TypeId smallint

BEGIN TRY
DECLARE @Types TABLE (TypeId smallint PRIMARY KEY, Name varchar(100))
INSERT INTO @Types EXECUTE dbo.GetUsedResourceTypes

DECLARE @Keys TABLE (TypeId smallint, SurrogateId bigint PRIMARY KEY (TypeId, SurrogateId))
WHILE EXISTS (SELECT * FROM @Types)
BEGIN
SET @TypeId = (SELECT TOP 1 TypeId FROM @Types ORDER BY TypeId)

INSERT INTO @Keys SELECT @TypeId, ResourceSurrogateId FROM dbo.Resource WHERE ResourceTypeId = @TypeId AND TransactionId = @TransactionId

DELETE FROM @Types WHERE TypeId = @TypeId
END

IF @ReturnResourceKeysOnly = 0
SELECT ResourceTypeId
,ResourceId
Expand All @@ -34,20 +19,20 @@ BEGIN TRY
,IsRawResourceMetaSet
,SearchParamHash
,RequestMethod
FROM (SELECT TOP (@DummyTop) * FROM @Keys) A
JOIN dbo.Resource B ON ResourceTypeId = TypeId AND ResourceSurrogateId = SurrogateId
WHERE IsHistory = 0 OR @IncludeHistory = 1
OPTION (MAXDOP 1, OPTIMIZE FOR (@DummyTop = 1))
,FileId
,OffsetInFile
FROM dbo.Resource
WHERE TransactionId = @TransactionId AND (IsHistory = 0 OR @IncludeHistory = 1)
OPTION (MAXDOP 1)
ELSE
SELECT ResourceTypeId
,ResourceId
,ResourceSurrogateId
,Version
,IsDeleted
FROM (SELECT TOP (@DummyTop) * FROM @Keys) A
JOIN dbo.Resource B ON ResourceTypeId = TypeId AND ResourceSurrogateId = SurrogateId
WHERE IsHistory = 0 OR @IncludeHistory = 1
OPTION (MAXDOP 1, OPTIMIZE FOR (@DummyTop = 1))
FROM dbo.Resource
WHERE TransactionId = @TransactionId AND (IsHistory = 0 OR @IncludeHistory = 1)
OPTION (MAXDOP 1)

EXECUTE dbo.LogEvent @Process=@SP,@Mode=@Mode,@Status='End',@Start=@st,@Rows=@@rowcount
END TRY
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
--DROP PROCEDURE dbo.GetResourcesByTypeAndSurrogateIdRange
GO
CREATE PROCEDURE dbo.GetResourcesByTypeAndSurrogateIdRange @ResourceTypeId smallint, @StartId bigint, @EndId bigint, @GlobalEndId bigint = NULL, @IncludeHistory bit = 0, @IncludeDeleted bit = 0
CREATE PROCEDURE dbo.GetResourcesByTypeAndSurrogateIdRange @ResourceTypeId smallint, @StartId bigint, @EndId bigint, @GlobalEndId bigint = NULL, @IncludeHistory bit = 1, @IncludeDeleted bit = 1
AS
set nocount on
DECLARE @SP varchar(100) = 'GetResourcesByTypeAndSurrogateIdRange'
Expand All @@ -9,18 +9,19 @@ DECLARE @SP varchar(100) = 'GetResourcesByTypeAndSurrogateIdRange'
+' E='+isnull(convert(varchar,@EndId),'NULL')
+' GE='+isnull(convert(varchar,@GlobalEndId),'NULL')
+' HI='+isnull(convert(varchar,@IncludeHistory),'NULL')
+' DE'+isnull(convert(varchar,@IncludeDeleted),'NULL')
+' DE='+isnull(convert(varchar,@IncludeDeleted),'NULL')
,@st datetime = getUTCdate()
,@DummyTop bigint = 9223372036854775807
,@Rows int

BEGIN TRY
DECLARE @ResourceIds TABLE (ResourceId varchar(64) COLLATE Latin1_General_100_CS_AS PRIMARY KEY)
DECLARE @ResourceIdInts TABLE (ResourceIdInt bigint PRIMARY KEY)
DECLARE @SurrogateIds TABLE (MaxSurrogateId bigint PRIMARY KEY)

IF @GlobalEndId IS NOT NULL AND @IncludeHistory = 0 -- snapshot view
BEGIN
INSERT INTO @ResourceIds
SELECT DISTINCT ResourceId
INSERT INTO @ResourceIdInts
SELECT DISTINCT ResourceIdInt
FROM dbo.Resource
WHERE ResourceTypeId = @ResourceTypeId
AND ResourceSurrogateId BETWEEN @StartId AND @EndId
Expand All @@ -31,30 +32,44 @@ BEGIN TRY
IF @@rowcount > 0
INSERT INTO @SurrogateIds
SELECT ResourceSurrogateId
FROM (SELECT ResourceId, ResourceSurrogateId, RowId = row_number() OVER (PARTITION BY ResourceId ORDER BY ResourceSurrogateId DESC)
FROM dbo.Resource WITH (INDEX = IX_Resource_ResourceTypeId_ResourceId_Version) -- w/o hint access to Resource table is inefficient when many versions are present. Hint is ignored if Resource is a view.
FROM (SELECT ResourceIdInt, ResourceSurrogateId, RowId = row_number() OVER (PARTITION BY ResourceIdInt ORDER BY ResourceSurrogateId DESC)
FROM dbo.Resource
WHERE ResourceTypeId = @ResourceTypeId
AND ResourceId IN (SELECT TOP (@DummyTop) ResourceId FROM @ResourceIds)
AND ResourceIdInt IN (SELECT TOP (@DummyTop) ResourceIdInt FROM @ResourceIdInts)
AND ResourceSurrogateId BETWEEN @StartId AND @GlobalEndId
) A
WHERE RowId = 1
AND ResourceSurrogateId BETWEEN @StartId AND @EndId
OPTION (MAXDOP 1, OPTIMIZE FOR (@DummyTop = 1))
END

SELECT ResourceTypeId, ResourceId, Version, IsDeleted, ResourceSurrogateId, RequestMethod, IsMatch = convert(bit,1), IsPartial = convert(bit,0), IsRawResourceMetaSet, SearchParamHash, RawResource
FROM dbo.Resource
WHERE ResourceTypeId = @ResourceTypeId
AND ResourceSurrogateId BETWEEN @StartId AND @EndId
AND (IsHistory = 0 OR @IncludeHistory = 1)
AND (IsDeleted = 0 OR @IncludeDeleted = 1)
UNION ALL
SELECT ResourceTypeId, ResourceId, Version, IsDeleted, ResourceSurrogateId, RequestMethod, IsMatch = convert(bit,1), IsPartial = convert(bit,0), IsRawResourceMetaSet, SearchParamHash, RawResource
FROM @SurrogateIds
JOIN dbo.Resource ON ResourceTypeId = @ResourceTypeId AND ResourceSurrogateId = MaxSurrogateId
WHERE IsHistory = 1
AND (IsDeleted = 0 OR @IncludeDeleted = 1)
OPTION (MAXDOP 1)
IF @IncludeHistory = 0
SELECT ResourceTypeId, ResourceId, Version, IsDeleted, ResourceSurrogateId, RequestMethod, IsMatch = convert(bit,1), IsPartial = convert(bit,0), IsRawResourceMetaSet, SearchParamHash, RawResource, FileId, OffsetInFile
FROM dbo.Resource
WHERE ResourceTypeId = @ResourceTypeId
AND ResourceSurrogateId BETWEEN @StartId AND @EndId
AND IsHistory = 0
AND (IsDeleted = 0 OR @IncludeDeleted = 1)
UNION ALL
SELECT ResourceTypeId, ResourceId, Version, IsDeleted, ResourceSurrogateId, RequestMethod, IsMatch = convert(bit,1), IsPartial = convert(bit,0), IsRawResourceMetaSet, SearchParamHash, RawResource, FileId, OffsetInFile
FROM @SurrogateIds
JOIN dbo.Resource ON ResourceTypeId = @ResourceTypeId AND ResourceSurrogateId = MaxSurrogateId
WHERE IsHistory = 1
AND (IsDeleted = 0 OR @IncludeDeleted = 1)
OPTION (MAXDOP 1, LOOP JOIN)
ELSE -- @IncludeHistory = 1
SELECT ResourceTypeId, ResourceId, Version, IsDeleted, ResourceSurrogateId, RequestMethod, IsMatch = convert(bit,1), IsPartial = convert(bit,0), IsRawResourceMetaSet, SearchParamHash, RawResource, FileId, OffsetInFile
FROM dbo.Resource
WHERE ResourceTypeId = @ResourceTypeId
AND ResourceSurrogateId BETWEEN @StartId AND @EndId
AND (IsDeleted = 0 OR @IncludeDeleted = 1)
UNION ALL
SELECT ResourceTypeId, ResourceId, Version, IsDeleted, ResourceSurrogateId, RequestMethod, IsMatch = convert(bit,1), IsPartial = convert(bit,0), IsRawResourceMetaSet, SearchParamHash, RawResource, FileId, OffsetInFile
FROM @SurrogateIds
JOIN dbo.Resource ON ResourceTypeId = @ResourceTypeId AND ResourceSurrogateId = MaxSurrogateId
WHERE IsHistory = 1
AND (IsDeleted = 0 OR @IncludeDeleted = 1)
OPTION (MAXDOP 1, LOOP JOIN)

EXECUTE dbo.LogEvent @Process=@SP,@Mode=@Mode,@Status='End',@Start=@st,@Rows=@@rowcount
END TRY
Expand Down
Loading
Loading