Benchmarking Performance-Critical Components in a Native XML Database System [chapter]

Karsten Schmidt, Sebastian Bächle, Theo Härder
2009 Lecture Notes in Computer Science  
The rapidly increasing number of XML-related applications indicates a growing need for efficient, dynamic, and native XML support in database management systems (XDBMS). So far, both industry and academia primarily focus on benchmarking of high-level performance figures for a variety of applications, queries, or documents -frequently executed in artificial workload scenarios -and, therefore, may analyze and compare only specific or incidental behavior of the underlying systems. To cover the
more » ... XDBMS support, it is mandatory to benchmark performance-critical components bottom-up, thereby removing bottlenecks and optimizing component behavior. In this way, wrong conclusions are avoided when new techniques such as tailored XML operators, index types, or storage mappings with unfamiliar performance characteristics are used. As an experience report, we present what we have learned from benchmarking a native XDBMS and recommend certain setups to do it in a systematic and meaningful way. Related Work Selecting an appropriate benchmark for the targeted application scenario can be challenging and has become a favorite research topic especially for XML in the recent years. In particular, the definitions of XPath [36] as an XML query language for path expressions and of the Turing-complete XQuery [37] language, which is actually based on XPath, caused the emergence of several XML benchmarks. One of the first XML benchmarks is XMach-1 [5] developed to analyze web applications using XML documents of varying structure and content. Besides simple search queries, coarse operations to delete entire documents or to insert new documents are defined. Thus, XMach-1 does not address, among other issues, efficiency checking of concurrency control when evaluating fine-grained modifications in XML documents. Another very popular project is XMark [33], providing artificial auction data within a single, but scalable document and 20 pre-defined queries. Actually such a benchmark is useful to evaluate search functions at the query layer, whereas, again, multi-user access is not addressed at all. A more recent alternative proposed by IBM can be used to evaluate transactional processing over XML -TPoX [29], which utilizes the XQuery Update Facility [38] to include XML modification operations. By referring to a real-world XML schema (FixML), the benchmark defines three types of documents having rather small sizes (≤ 26 KB) and 11 initial queries for a financial application scenario. The dataset size can be scaled from several GB to one PB and the performance behavior of multi-user read/write transactions is a main objective of that project. Nevertheless, all benchmarking is done at the XQuery layer and no further system internals are to be inspected in detail. However, other benchmarks explicitly investigate the XQuery engine in a stand-alone mode [1, 23] . Such "black box" benchmarks seem to be reasonable for scenarios operating on XML-enabled databases having relational cores, e.g., [4, 10, 11, 35] , because their internals are optimized during the last 40 years. Furthermore, it is interesting to take a look at the list of performance-critical aspects suggested by [31] and [32]: bulk loading, document reconstruction, path traversal, data type casting, missing elements, ordered access, references, valuebased joins, construction of large result sets, indexing, and full-text search for the containment function of XQuery. Almost all aspects are solely focusing on the application level or query level of XML processing, or they address relational systems, which have to cope with missing elements (NULL values). Indeed, the development towards native XML databases, e.g., [15, 18, 28, 30, 34] was necessary to overcome all shortcomings of "Shredding" [35] and lead to new benchmark requirements and opportunities [27] . Other comparative work in [1] and [26] either compared existing XML benchmarks or discussed the cornerstones of good XML benchmarks (e.g, [21, 32] ) defining queries and datasets. The only work that tried to extend existing approaches is presented in [31] . Although the authors forgot to consider TPoX, which has a focal point on updates, they extended other existing benchmarks with update queries to overcome that drawback.
doi:10.1007/978-3-642-04205-8_7 fatcat:kpq7jw5tazd65ovwmaovxmiydm