Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TH2-3857 #15

Open
wants to merge 51 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
e69c873
Self-closing tag and th2-codec error events (#13)
eugene-zheltov May 27, 2022
717b38d
[TH2-3778] Set codec version from snapshot with fixes
eugene-zheltov Jun 7, 2022
6ab2f0b
[TH2-3778] Set codec version from snapshot with fixes
eugene-zheltov Jun 8, 2022
75773ee
[TH2-3778] Change verison of codec core
eugene-zheltov Jun 8, 2022
d785d78
[TH2-3778] Change verison of codec core
eugene-zheltov Jun 8, 2022
542bde8
[TH2-3778] Change version of codec core
eugene-zheltov Jun 9, 2022
a47c0b1
[TH2-3778] Change version of codec core
eugene-zheltov Jun 10, 2022
09e44f9
[TH2-3778] Change version of codec core to 4.7.1-TH2-3778-tmp-2495038…
eugene-zheltov Jun 14, 2022
1e3527b
[TH2-3857] tmp
eugene-zheltov Jun 28, 2022
4764809
[TH2-3857] tmp-2
eugene-zheltov Jul 4, 2022
4f61923
[TH2-3857] tmp-3
eugene-zheltov Jul 4, 2022
8d785c8
[TH2-3857] tmp-4
eugene-zheltov Jul 7, 2022
74c2c5a
[TH2-3857] tmp-4
eugene-zheltov Jul 11, 2022
68f0a0c
[TH2-3857] Successful parsing; to be refactored
eugene-zheltov Jul 12, 2022
cec90fe
[TH2-3857] Refactor XMLSchemaCore
eugene-zheltov Jul 14, 2022
16f85d0
[TH2-3857] tmp-5
eugene-zheltov Jul 18, 2022
07d3a28
TH2-3778 (#14)
eugene-zheltov Jul 19, 2022
8cfce4c
[TH2-3857] tmp-6
eugene-zheltov Jul 20, 2022
8cf2d64
[TH2-3857] Fix the absence of some list values
eugene-zheltov Jul 25, 2022
62c3030
[TH2-3857] Remove prints and change xsd processing a bit
eugene-zheltov Jul 26, 2022
24d6d97
[TH2-3857] Disable tests
eugene-zheltov Jul 26, 2022
06e3d3e
[TH2-3857] Minor changes
eugene-zheltov Jul 26, 2022
725b78e
Merge branch 'master' into TH2-3857
eugene-zheltov Jul 26, 2022
0c6a616
[TH2-3857] Move StreamReaderDelegateDecorator
eugene-zheltov Jul 27, 2022
7697106
[TH2-3857] Chagne xsdProperties loading
eugene-zheltov Jul 27, 2022
468a849
[TH2-3857] Add th2-id to the log when th2-codec-xml-via-xsd cannot de…
eugene-zheltov Jul 28, 2022
6e996da
[TH2-3857] Minor changes
eugene-zheltov Jul 29, 2022
90a4ce7
Merge remote-tracking branch 'origin/TH2-3857' into TH2-3857
eugene-zheltov Jul 29, 2022
652cf79
[TH2-3857] tmp
eugene-zheltov Aug 1, 2022
c8474de
[TH2-3857] tmp 2
eugene-zheltov Aug 2, 2022
14be420
[TH2-3857] tmp 3
eugene-zheltov Aug 2, 2022
71ff9c2
[TH2-3857] Implement new decoding
eugene-zheltov Aug 3, 2022
17507d0
[TH2-3857] Remove comments
eugene-zheltov Aug 3, 2022
207e5c1
[TH2-3857] Fix repeated elements of lists
eugene-zheltov Aug 3, 2022
cba5310
[TH2-3857] Fix missing empty tags
eugene-zheltov Aug 3, 2022
3b6e346
[TH2-3857] Remove usage of XmlPipelineCodec for now
eugene-zheltov Aug 3, 2022
fe488b8
[TH2-3857] Remove garbage
eugene-zheltov Aug 4, 2022
8d5a873
[TH2-3857] Fix truncated values
eugene-zheltov Aug 5, 2022
cea2111
[TH2-3857] Add missing attributes
eugene-zheltov Aug 5, 2022
bbe67f7
[TH2-3857] Disable test
eugene-zheltov Aug 5, 2022
0fb452b
[TH2-3857] Add prefixes to node names
eugene-zheltov Aug 5, 2022
7791388
[TH2-3857] Remove redundant ":" from node names
eugene-zheltov Aug 5, 2022
06dc868
[TH2-3857] Migrated to StAX
Nikita-Smirnov-Exactpro Aug 9, 2022
23f0005
[TH2-3857] Fixed problem related to release sub-nodes
Nikita-Smirnov-Exactpro Aug 10, 2022
a6aab96
[TH2-3857] Updated common libraries
Nikita-Smirnov-Exactpro Aug 10, 2022
7a66236
[TH2-3857] Consider empty xmlns property
eugene-zheltov Oct 6, 2022
a127615
[TH2-3857] Add a new parameter encodeValidation
eugene-zheltov Feb 16, 2023
3f9843f
[TH2-3857] Update dev-docker-publish.yml and docker-publish.yml
eugene-zheltov Feb 20, 2023
1d327a4
[TH2-3857] Update common version
eugene-zheltov Feb 20, 2023
bb17598
[TH2-3857] Update codec-core version
eugene-zheltov Feb 21, 2023
0510415
[TH2-3857] Update slf4j dependencies
eugene-zheltov Feb 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions src/main/kotlin/com/exactpro/th2/codec/xml/NodeContent.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
package com.exactpro.th2.codec.xml

import com.exactpro.th2.common.grpc.ListValue
import com.exactpro.th2.common.grpc.Message
import com.exactpro.th2.common.grpc.Value
import com.exactpro.th2.common.grpc.Value.KindCase.SIMPLE_VALUE
import com.exactpro.th2.common.grpc.Value.KindCase.MESSAGE_VALUE
import com.exactpro.th2.common.grpc.Value.KindCase.LIST_VALUE
import com.exactpro.th2.common.message.addField
import com.exactpro.th2.common.message.message
import com.exactpro.th2.common.value.add
import com.exactpro.th2.common.value.listValue
import javax.xml.namespace.QName

class NodeContent(val nodeName: QName) {
val attributes: MutableMap<String, String> = mutableMapOf()
val childNodes: MutableMap<QName, MutableList<NodeContent>> = mutableMapOf()
Nikita-Smirnov-Exactpro marked this conversation as resolved.
Show resolved Hide resolved

var text: String? = null
var type: Value.KindCase = SIMPLE_VALUE

fun StreamReaderDelegateDecorator.addAttributes() {
if (attributeCount > 0) {
for (i in 0 until attributeCount) {
attributes[getAttributeName(i).localPart] = getAttributeValue(i)
}
}
}

fun setMessageType() {
Nikita-Smirnov-Exactpro marked this conversation as resolved.
Show resolved Hide resolved
if (this.type == SIMPLE_VALUE) {
this.type = MESSAGE_VALUE
}
}

fun release(messageBuilder: Message.Builder) {

// TODO: don't forget about attributes

childNodes.forEach {
messageBuilder.addNode(it.key, it.value)
}

messageBuilder.addField(nodeName.localPart, this)
}

private fun Message.Builder.addNode(nodeName: QName, nodeList: MutableList<NodeContent>) {
println("Message addNode $nodeName")
nodeList.forEach { node ->
when (node.type) {
MESSAGE_VALUE -> {
val count = nodeList.count()
if (count > 1) {
val list = listValue()

nodeList.forEach { nodeContent ->
nodeContent.childNodes.forEach {
list.addNode(it.key, it.value)
}
}

addField(nodeName.localPart, list)
} else if (count == 1) {
val message = message()

node.childNodes.forEach {
message.addNode(it.key, it.value)
}

addField(nodeName.localPart, message)
}
}

// TODO: mb I it's possible to have a list of simple values too
SIMPLE_VALUE -> addField(node.nodeName.localPart, node.text)

else -> throw IllegalArgumentException("Node $node can be either MESSAGE_VALUE or SIMPLE_VALUE")
}
}
}

private fun ListValue.Builder.addNode(nodeName: QName, nodeList: MutableList<NodeContent>) {
println("List addNode $nodeName, nodeList $nodeList")
nodeList.forEach { node ->
when (node.type) {
MESSAGE_VALUE -> {
val count = nodeList.count()
if (count > 1) {
val list = listValue()

node.childNodes.forEach {
list.addNode(it.key, it.value)
}

add(list)
} else if (count == 1) {
val message = message()

node.childNodes.forEach {
message.addNode(it.key, it.value)
}

add(message)
}
}

// TODO: mb I it's possible to have a list of simple values too
SIMPLE_VALUE -> add(node.text)

else -> throw IllegalArgumentException("Node $node can be either MESSAGE_VALUE or SIMPLE_VALUE")
}
}
}

override fun toString(): String {
return "NodeContent(attributes=$attributes, childNodes=$childNodes, text=$text, type=$type)"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,22 +12,21 @@ import javax.xml.namespace.QName
import javax.xml.stream.XMLStreamException
import javax.xml.stream.XMLStreamReader
import javax.xml.stream.util.StreamReaderDelegate
import com.exactpro.th2.common.grpc.Value.KindCase.SIMPLE_VALUE
import com.exactpro.th2.common.grpc.Value.KindCase.MESSAGE_VALUE
import com.exactpro.th2.common.grpc.Value.KindCase.LIST_VALUE
import com.exactpro.th2.common.grpc.Value.KindCase.KIND_NOT_SET
import com.exactpro.th2.common.grpc.Value.KindCase.MESSAGE_VALUE
import com.exactpro.th2.common.grpc.Value.KindCase.SIMPLE_VALUE
import com.exactpro.th2.common.message.addField
import com.exactpro.th2.common.message.message
import com.exactpro.th2.common.message.set
import com.exactpro.th2.common.value.add
import com.exactpro.th2.common.value.listValue
import com.exactpro.th2.common.value.toValue
import java.util.*
import kotlin.collections.ArrayList
import kotlin.collections.HashMap

class StreamReaderDelegateDecorator(reader: XMLStreamReader,
private val rawMessage: RawMessage,
private val xmlSchemaCore: XMLSchemaCore, ) : StreamReaderDelegate(reader) {
private val xmlSchemaCore: XMLSchemaCore) : StreamReaderDelegate(reader) {
private val elementStack = ArrayList<QName>()
Nikita-Smirnov-Exactpro marked this conversation as resolved.
Show resolved Hide resolved
private val elementTypeStack = Stack<Value.KindCase>()

Expand All @@ -49,6 +48,8 @@ class StreamReaderDelegateDecorator(reader: XMLStreamReader,
private val allElements = mutableMapOf<QName, Value.KindCase>()
private val xsdElements = mutableMapOf<QName, List<XmlElementWrapper>>()

private val elements = Stack<NodeContent>()

@Throws(XMLStreamException::class)
override fun next(): Int {
val n: Int = super.next()
Expand All @@ -68,63 +69,21 @@ class StreamReaderDelegateDecorator(reader: XMLStreamReader,
val attributeName = getAttributeName(i).localPart
val attributeValue = getAttributeValue(i)

if (attributeName == "schemaLocation") {
if (attributeName == SCHEMA_LOCATION) {
cacheXsdFromAttribute(attributeValue)
}
}

val elementType = if (allElements[qName] != KIND_NOT_SET) {
allElements[qName]
} else {
checkNotNull(xsdElements[elementStack[elementStack.lastIndex - 1]]?.find { it.qName == qName }).elementType
}

when(elementType) {
SIMPLE_VALUE -> {
elementTypeStack.push(SIMPLE_VALUE)
}
MESSAGE_VALUE -> {
elementTypeStack.push(MESSAGE_VALUE)

val builder = message()
val nodeContent = NodeContent(qName)

if (attributeCount > 0) {
writeAttributes(builder)
}

messageValueStack.push(builder)
}
LIST_VALUE -> {
elementTypeStack.push(LIST_VALUE)

val parentName = checkNotNull(xsdElements[elementStack[elementStack.lastIndex - 1]]?.find { it.qName == qName }).qName

val msgBuilderWrapper: MessageBuilderWrapper = if (msgBuilderWrapperMap.contains(parentName)) {
checkNotNull(msgBuilderWrapperMap[parentName])
} else {
MessageBuilderWrapper(qName).also {
msgBuilderWrapperMap[qName] = it
}
}

val listBuilder = if (msgBuilderWrapper.contains(qName)) {
checkNotNull(msgBuilderWrapper[qName])
} else {
val list = listValue()
msgBuilderWrapper[qName] = list
list
}

if (attributeCount > 0) {
writeAttributes(listBuilder)
}

listValueStack.push(listBuilder)
}
null -> { throw IllegalArgumentException("There's no element for $qName") }
else -> { throw IllegalArgumentException("Element $qName is not a simpleValue, messageValue or listValue") }
if (elements.isNotEmpty()) {
val parent = elements.peek()
parent.setMessageType()
parent.childNodes[qName] = mutableListOf(nodeContent)
}

elements.push(nodeContent)

// TODO: also use pointer
if (!foundMsgType) {
metadataBuilder.messageType = localName
Expand All @@ -134,76 +93,35 @@ class StreamReaderDelegateDecorator(reader: XMLStreamReader,
CHARACTERS -> {
if (text.isNotBlank()) {
val qName = elementStack[elementStack.lastIndex]
val localElementName = qName.localPart

val elementType = if (allElements[qName] != KIND_NOT_SET) {
allElements[qName]
} else {
checkNotNull(xsdElements[elementStack[elementStack.lastIndex - 1]]?.find { it.qName == qName }).elementType
}

when(elementType) {
SIMPLE_VALUE -> {
simpleValueStack.add(Value.newBuilder().setSimpleValue(text))
}
MESSAGE_VALUE -> {
val builder = messageValueStack.peek()
builder[localElementName] = text.toValue()
}
else -> { throw IllegalArgumentException("Element is not a simpleValue, messageValue or listValue") }
}
elements.peek().text = text
}
}
END_ELEMENT -> {
elementTypeStack.pop()
// elementTypeStack.pop()

val qName = QName(namespaceURI, localName, namespaceContext.getPrefix(namespaceURI))

val element = elements.pop()

if (elements.isNotEmpty()) {
val parentElement = elements.peek()

checkNotNull(parentElement.childNodes[qName]).add(element)
} else {
// println("Element: ${element.childNodes}")
val builder = message()
element.release(builder)
// println("Final message: ${builder.build()}")
}

if (elementTypeStack.isNotEmpty()) {
val qName = elementStack.removeLast()

val parentType = elementTypeStack.peek()

val elementType = if (allElements[qName] != KIND_NOT_SET) {
allElements[qName]
} else {
checkNotNull(xsdElements[elementStack[elementStack.lastIndex]]?.find { it.qName == qName }).elementType
}

when(elementType) {
SIMPLE_VALUE -> {
if (parentType == MESSAGE_VALUE) {
messageValueStack.peek()[qName.localPart] = simpleValueStack.pop()
} else {
listValueStack.peek().add(simpleValueStack.pop())
}
}
MESSAGE_VALUE -> {
if (parentType == MESSAGE_VALUE) {
val message = messageValueStack.pop()
messageValueStack.peek()[qName.localPart] = message

message.addLists(qName)
} else {
val message = messageValueStack.pop()
listValueStack.peek().add(message)

message.addLists(qName)
}
}
LIST_VALUE -> {
if (parentType == MESSAGE_VALUE) {
val list = listValueStack.pop()
messageValueStack.peek()[qName.localPart] = list
} else {
val list = listValueStack.pop()
listValueStack.peek().add(list)

list.addLists(qName)
}
}
else -> { throw IllegalArgumentException("Element $qName is not a simpleValue, messageValue or listValue") }
}
} else {
messageBuilder = messageValueStack.pop()
// messageBuilder = messageValueStack.pop()
}
}
}
Expand All @@ -222,17 +140,18 @@ class StreamReaderDelegateDecorator(reader: XMLStreamReader,
putAllProperties(metadata.propertiesMap)
}

messageBuilder.metadata = metadataBuilder.build()

if (rawMessage.hasParentEventId()) {
messageBuilder.parentEventId = rawMessage.parentEventId
}

val message = messageBuilder.build()
messageBuilder.clear()
msgBuilderWrapperMap.clear()

return message
// messageBuilder.metadata = metadataBuilder.build()
//
// if (rawMessage.hasParentEventId()) {
// messageBuilder.parentEventId = rawMessage.parentEventId
// }
//
// val message = messageBuilder.build()
// messageBuilder.clear()
// msgBuilderWrapperMap.clear()

// return
return message().build()
}

fun clearElements() { elementStack.clear() }
Expand Down Expand Up @@ -304,6 +223,7 @@ class StreamReaderDelegateDecorator(reader: XMLStreamReader,
}

companion object {
private val SCHEMA_LOCATION = "schemaLocation"
private val LOGGER = KotlinLogging.logger { }
}

Expand All @@ -318,4 +238,5 @@ class StreamReaderDelegateDecorator(reader: XMLStreamReader,

operator fun get(key: QName): ListValue.Builder? = listBuilderMap[key]
}

}