Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add aws xray context propagation #3898

Open
wants to merge 59 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
2d39509
fix aws tests
wconti27 Dec 11, 2023
a3f31e7
Merge branch 'master' into conti/fix-aws-sdk-tests
wconti27 Dec 11, 2023
8f7f569
fix kinesis timeouts
wconti27 Dec 11, 2023
bbc5897
fix error
wconti27 Dec 11, 2023
51fe166
increase timeout
wconti27 Dec 11, 2023
107e02b
try again
wconti27 Dec 11, 2023
af16096
no timeout test
wconti27 Dec 11, 2023
aa84b7e
add dsm for sns sqs and kinesis
wconti27 Dec 11, 2023
1923d87
fix failing tests
wconti27 Dec 11, 2023
b3b35b6
add debugging statemtn
wconti27 Dec 12, 2023
c118fd3
fix tests
wconti27 Dec 12, 2023
afcefb1
fix sns tests
wconti27 Dec 12, 2023
f014b2c
fix again
wconti27 Dec 12, 2023
eb13676
try env variable to enable
wconti27 Dec 12, 2023
b0cb24b
fix lint
wconti27 Dec 12, 2023
6449e96
Merge branch 'master' into conti/add-dsm-for-aws-services
wconti27 Dec 12, 2023
62aad55
add message payload size tests for sns / kinesis
wconti27 Dec 12, 2023
43bb3f1
add more payloadSize tests
wconti27 Dec 13, 2023
3f38244
fix kinesis payload size
wconti27 Dec 13, 2023
d8e7bff
fix kinesis tests
wconti27 Dec 13, 2023
2b88ce5
Merge branch 'master' into conti/add-dsm-for-aws-services
wconti27 Dec 13, 2023
564bff3
add context propagation to kinesis
wconti27 Dec 13, 2023
3a692aa
more tests
wconti27 Dec 13, 2023
0e9010d
increase timeout
wconti27 Dec 13, 2023
c4a7d54
rewrite SQS to set DSM checkpoint for each message
wconti27 Dec 14, 2023
8fc0843
change payloadSize calculations
wconti27 Dec 15, 2023
8c5f653
fix payload size error
wconti27 Dec 15, 2023
73519f3
use stream name for checkpoint if arn unavailable
wconti27 Dec 15, 2023
5a11c04
Merge branch 'conti/add-dsm-for-aws-services' into conti/add-context-…
wconti27 Dec 15, 2023
6e0d695
add xray propagation
wconti27 Dec 20, 2023
8cb98ef
remove test that i don't remember adding
wconti27 Dec 20, 2023
b9c8bf3
fix failing tests
wconti27 Dec 20, 2023
bea6673
clean up code
wconti27 Apr 4, 2024
c210127
Merge branch 'master' into conti/add-aws-xray-header-context-extraction
wconti27 Apr 4, 2024
8528904
Revert all changes except propagation class
wconti27 Apr 4, 2024
2b58872
add baggage tags
wconti27 Apr 5, 2024
abf28ca
change encoding / decoding of hashes to match other tracers
wconti27 Apr 5, 2024
7d99016
Revert "change encoding / decoding of hashes to match other tracers"
wconti27 Apr 5, 2024
03008e6
Merge branch 'master' into conti/add-aws-xray-header-context-extraction
wconti27 Dec 13, 2024
f7a542f
add inject method
wconti27 Dec 13, 2024
33ee7de
clean up code
wconti27 Dec 13, 2024
991d69b
add testing
wconti27 Dec 16, 2024
4e306b2
add more test cases
wconti27 Dec 16, 2024
c4de314
fix lint
wconti27 Dec 17, 2024
f6ff0b5
small changes
wconti27 Dec 19, 2024
f427e9b
remove check for aws signed requests
wconti27 Dec 19, 2024
2f702c7
add test for baggage too large
wconti27 Dec 19, 2024
70cbfd8
add more tests to ensure new context propagation works with existing …
wconti27 Dec 26, 2024
4f7fe84
Merge branch 'master' into conti/add-aws-xray-header-context-extraction
wconti27 Jan 3, 2025
504b584
fix lint and other errors
wconti27 Jan 3, 2025
7b07cce
remove aws signature tests
wconti27 Jan 3, 2025
fc1e874
fix signature tests
wconti27 Jan 3, 2025
246994e
fix function call
wconti27 Jan 3, 2025
397984c
another fix
wconti27 Jan 3, 2025
ba57fba
another fixx
wconti27 Jan 3, 2025
f2be5c3
remove test
wconti27 Jan 3, 2025
6acb364
be more careful
wconti27 Jan 6, 2025
42ba094
fix logging
wconti27 Jan 6, 2025
57d68a7
fix error
wconti27 Jan 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 161 additions & 0 deletions packages/dd-trace/src/opentracing/propagation/text_map.js
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,16 @@ const tracestateTagKeyFilter = /[^\x21-\x2b\x2d-\x3c\x3e-\x7e]/g
const tracestateTagValueFilter = /[^\x20-\x2b\x2d-\x3a\x3c-\x7d]/g
const invalidSegment = /^0+$/
const zeroTraceId = '0000000000000000'
// AWS X-Ray specific constants
const xrayHeaderKey = 'x-amzn-trace-id'
const xrayRootKey = 'root'
const xrayRootPrefix = '1-'
const xrayParentKey = 'parent'
const xraySampledKey = 'sampled'
const xrayE2EStartTimeKey = 't0'
const xraySelfKey = 'self'
const xrayOriginKey = '_dd.origin'
const xrayMaxAdditionalBytes = 256

class TextMapPropagator {
constructor (config) {
Expand All @@ -60,6 +70,7 @@ class TextMapPropagator {
this._injectB3MultipleHeaders(spanContext, carrier)
this._injectB3SingleHeader(spanContext, carrier)
this._injectTraceparent(spanContext, carrier)
this._injectAwsXrayContext(spanContext, carrier)
wconti27 marked this conversation as resolved.
Show resolved Hide resolved

if (injectCh.hasSubscribers) {
injectCh.publish({ spanContext, carrier })
Expand Down Expand Up @@ -252,6 +263,66 @@ class TextMapPropagator {
carrier.tracestate = ts.toString()
}

_injectAwsXrayContext (spanContext, carrier) {
// injects AWS Trace Header (X-Amzn-Trace-Id) to carrier
// ex: 'Root=1-00000000-00000000fffffffffffffffe;Parent=ffffffffffffffff;Sampled=1;_dd.origin=fakeOrigin;

// based off: https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader
if (!this._hasPropagationStyle('inject', 'xray')) return

// TODO: Do we have access to a start time? Java does
const e2eStart = this._getEndToEndStartTime(spanContext.start_ms)

let str = (
xrayRootKey + '=' +
xrayRootPrefix +
e2eStart + '-' +
spanContext._traceId.toString().padStart(24, '0') +
';' + xrayParentKey + '=' +
spanContext._spanId.toString().padStart(16, '0')
)

if (spanContext?._sampling?.priority) {
str += ';' + xraySampledKey + '=' + (spanContext._sampling.priority > 0 ? '1' : '0')
}

const maxAdditionalCapacity = xrayMaxAdditionalBytes - str.length

const origin = spanContext._trace.origin
if (origin) {
this._addXrayBaggage(str, xrayOriginKey, origin, maxAdditionalCapacity)
}
if (e2eStart !== '00000000') {
this._addXrayBaggage(str, xrayE2EStartTimeKey, e2eStart.toString(), maxAdditionalCapacity)
}

for (const [key, value] of Object.entries(spanContext._baggageItems)) {
if (!this._isReservedXrayKey(key)) {
str = this._addXrayBaggage(str, key, value, maxAdditionalCapacity)
}
}

carrier[xrayHeaderKey] = str
}

_getEndToEndStartTime (start) {
if (!start) return '00000000'
wconti27 marked this conversation as resolved.
Show resolved Hide resolved

const e2eStart = start > 0 ? start : Date.now() / 1000
return e2eStart.toString().padStart(8, '0')
}

_isReservedXrayKey (key) {
return [xrayRootKey, xrayParentKey, xrayOriginKey, xrayE2EStartTimeKey, xraySampledKey].includes(key.toLowerCase())
}

_addXrayBaggage (str, key, value, maxCapacity) {
if (str.length + key.length + value.toString().length + 2 <= maxCapacity) {
str += `;${key}=${value}`
}
return str
}

_hasPropagationStyle (mode, name) {
return this._config.tracePropagationStyle[mode].includes(name)
}
Expand Down Expand Up @@ -303,6 +374,9 @@ class TextMapPropagator {
case 'b3 single header': // TODO: delete in major after singular "b3"
extractedContext = this._extractB3SingleContext(carrier)
break
case 'xray':
extractedContext = this._extractAwsXrayContext(carrier)
break
case 'b3':
if (this._config.tracePropagationStyle.otelPropagators) {
// TODO: should match "b3 single header" in next major
Expand Down Expand Up @@ -692,6 +766,93 @@ class TextMapPropagator {
return spanContext._traceId.toString(16)
}

_extractAwsXrayContext (carrier) {
if (carrier[xrayHeaderKey]) {
const parsedHeader = this._parseAWSTraceHeader(carrier[xrayHeaderKey])

let traceId
let spanId
let samplingPriority
let ddOrigin
const baggage = {}

if (!(
xrayRootKey in parsedHeader &&
// Regex check to ensure received header is in the same format as expected
// Format:
// 'Root=1-'
// 8 hexadecimal characters representing start time
// '-'
// 24 hexadecimal characters representing trace id
// ';'
// 'Parent='
// 16 hexadecimal characters representing parent span id
/^(?=.*Root=1-[0-9a-f]{8}-[0-9a-f]{24})(?=.*Parent=[0-9a-f]{16}).*$/i.test(carrier[xrayHeaderKey])
tlhunter marked this conversation as resolved.
Show resolved Hide resolved
)) {
// header doesn't match formatting
return null
}

for (const key in parsedHeader) {
const value = parsedHeader[key]
if (key === xrayRootKey) {
const awsTraceIdSegments = value.split('-')
for (let i = 0; i < awsTraceIdSegments.length; i++) {
if (awsTraceIdSegments[i] === '1') {
continue
} else if (i === 1 && awsTraceIdSegments.length === 3) {
continue
} else if (i === 2 && awsTraceIdSegments.length === 3) {
// trace id padded by: "00000000"
traceId = awsTraceIdSegments[i].substring(8)
}
}
} else if (key === xrayParentKey) {
spanId = value
} else if (key === xraySampledKey) {
samplingPriority = parseInt(value)
} else if (key === xrayOriginKey) {
ddOrigin = String(value)
} else if (key === xraySelfKey) {
// self is added by load balancers and should be ignored
continue
} else if (key === xrayE2EStartTimeKey) {
baggage.startMs = parseInt(value)
} else {
baggage[key] = value
}
}

if (traceId && spanId) {
const spanContext = new DatadogSpanContext({
traceId: id(traceId, 16),
spanId: id(spanId, 16),
sampling: { samplingPriority },
baggageItems: baggage
})
if (ddOrigin) {
spanContext._trace.origin = ddOrigin
}
return spanContext
}
return null
} else {
return null
}
}

_parseAWSTraceHeader (header) {
// parses AWSTraceHeader string to object
// ex: 'Root=1-00000000-00000000fffffffffffffffe;Parent=ffffffffffffffff;Sampled=1;_dd.origin=fakeOrigin;
const obj = {}
const keyValuePairs = header.split(';')
keyValuePairs.forEach(pair => {
const [key, value] = pair.split('=')
obj[key.toLowerCase()] = value.toLowerCase()
})
return obj
}

static _convertOtelContextToDatadog (traceId, spanId, traceFlag, ts, meta = {}) {
const origin = null
let samplingPriority = traceFlag
Expand Down
133 changes: 133 additions & 0 deletions packages/dd-trace/test/opentracing/propagation/text_map.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,45 @@ describe('TextMapPropagator', () => {
injectCh.unsubscribe(onSpanInject)
}
})

it('should skip injection of aws xray header without the feature flag', () => {
const carrier = {}
const spanContext = createContext({
traceId: id('0000000000000123'),
spanId: id('0000000000000456')
})

config.tracePropagationStyle.inject = []

propagator.inject(spanContext, carrier)

expect(carrier).to.not.have.property('x-amzn-trace-id')
})

it('should inject AWSTraceHeader X-Amzn-Trace-Id when configured', () => {
const baggageItems = {
bool: true,
a: 'b'
}
const spanContext = createContext({
baggageItems,
sampling: {
priority: 1
}
})

config.tracePropagationStyle.inject = ['xray']

const traceId = spanContext._traceId.toString().padStart(24, '0')
const spanId = spanContext._spanId.toString().padStart(16, '0')
const expectedHeader = `root=1-00000000-${traceId};parent=${spanId};sampled=1`
const additionalParts = ';bool=true;a=b'
const carrier = {}

propagator.inject(spanContext, carrier)

expect(carrier['x-amzn-trace-id']).to.equal(expectedHeader + additionalParts)
})
})

describe('extract', () => {
Expand Down Expand Up @@ -662,6 +701,100 @@ describe('TextMapPropagator', () => {
expect(spanContext._tracestate).to.be.undefined
})

it('should not extract AWSTraceHeader X-Amzn-Trace-Id when not configured', () => {
const traceId = '4ef684dbd03d632e'
const spanId = '7e8d56262375628a'
const sampled = 1
const unparsedHeader = `Root=1-6583199d-00000000${traceId};Parent=${spanId};Sampled=${sampled}`
const carrier = {
'x-amzn-trace-id': unparsedHeader
}

config.tracePropagationStyle.extract = []

const spanContext = propagator.extract(carrier)

expect(spanContext).to.be.null
})

it('should extract AWSTraceHeader X-Amzn-Trace-Id when configured', () => {
const traceId = '4ef684dbd03d632e'
const spanId = '7e8d56262375628a'
const sampled = 1
const unparsedHeader = `Root=1-6583199d-00000000${traceId};Parent=${spanId};Sampled=${sampled}`
const carrier = {
'x-amzn-trace-id': unparsedHeader
}

config.tracePropagationStyle.extract = ['xray']

const spanContext = propagator.extract(carrier)

expect(spanContext.toTraceId()).to.equal(id(traceId, 16).toString(10))
expect(spanContext.toSpanId()).to.equal(id(spanId, 16).toString(10))
expect(spanContext._sampling.samplingPriority).to.equal(sampled)
})

it('should not extract AWSTraceHeader X-Amzn-Trace-Id if in an unexpected format or missing data', () => {
const traceId = '4ef684dbd03d632e'
const spanId = '7e8d56262375628a'
const sampled = 1

// All necessary fields included but trace id root segment is not formatted properly
let unparsedHeader = `Root=${traceId};Parent=${spanId};Sampled=${sampled}`
const carrier = {
'x-amzn-trace-id': unparsedHeader
}

config.tracePropagationStyle.extract = ['xray']

let spanContext = propagator.extract(carrier)

expect(spanContext).to.be.null

// Missing necessary parent field
unparsedHeader = `Root=1-6583199d-00000000${traceId};Sampled=${sampled}`

config.tracePropagationStyle.extract = ['xray']

spanContext = propagator.extract(carrier)

expect(spanContext).to.be.null

// Missing necessary root field
unparsedHeader = `Parent=${spanId};Sampled=${sampled}`

config.tracePropagationStyle.extract = ['xray']

spanContext = propagator.extract(carrier)

expect(spanContext).to.be.null
})

it('should extract AWSTraceHeader X-Amzn-Trace-Id with origin and baggage', () => {
tlhunter marked this conversation as resolved.
Show resolved Hide resolved
const traceId = '4ef684dbd03d632e'
const spanId = '7e8d56262375628a'
const sampled = 1
const unparsedHeader = `Root=1-6583199d-00000000${traceId};Parent=${spanId};Sampled=${sampled}`
const additionalParts = ';_dd.origin=localhost;baggage_key=baggage_value;foo=bar'
const carrier = {
'x-amzn-trace-id': unparsedHeader + additionalParts
}

config.tracePropagationStyle.extract = ['xray']

const spanContext = propagator.extract(carrier)

expect(spanContext.toTraceId()).to.equal(id(traceId, 16).toString(10))
expect(spanContext.toSpanId()).to.equal(id(spanId, 16).toString(10))
expect(spanContext._sampling.samplingPriority).to.equal(sampled)

expect(spanContext._baggageItems).to.deep.equal({
baggage_key: 'baggage_value', foo: 'bar'
})
expect(spanContext._trace.origin).to.equal('localhost')
})

it('extracts span_id from tracecontext headers and stores datadog parent-id in trace_distributed_tags', () => {
textMap['x-datadog-trace-id'] = '61185'
textMap['x-datadog-parent-id'] = '15'
Expand Down
Loading