Methodology

The dataset is built by extracting information from the German parliamentary archives' content overviews, combining it with the DIP, and enriching it with further sources.

Parliamentary archives documents

The relevant data were extracted from the archives' content overviews. This was possible because the overviews all adhere to a similar structure and include the same type of information (although they vary in layout over time and are inconsistent in their spacing). Each overview contains:

• a title page with meta-information on the legislative process (e.g. the title or the Bundestag/Bundesrat identification numbers);
• a list of the committees involved and their role within the process;
• a list of the legislative material including date, respective actors, and both the content and result (grouped by general phases and sorted by working steps within each grouping); and
• additional material such as press statements.

Legislative and additional material are ordered — by consecutive numbers — along the temporal order of their position in the law-making process. This documentation is organised in an implicit tabular form, whereby the entries have an internal hierarchy, like a tree structure.

From the archives' content overviews

The documents were made machine-readable — from PDF, to CSV, to JSON. The raw data from the PDF documents were extracted and transformed in two automated steps, followed by a manual cleaning process and both a manual and automated check for consistency with the original PDF data.

The data in the PDF documents were extracted and stored in CSV format using a cloud-based service. Even though the extraction generated reasonable results, variations in the tabular representation were still present — for example, due to incorrectly identified columns or rows — and the relationships between entries are not represented in the CSV. A second, significantly more complex step is therefore necessary: the structure of and the relations between all information within the PDF are formalised by a JSON schema that captures the document structure and content types.

The transformation separates:

• the general information of the legislative process,
• the committees involved,
• the law-making materials (sorted by consecutive numbers), and
• the additional material.

It is based on regular expressions indicating the beginning or end of a section and specific rules dealing with the tabular form and contextual relations. The law-making material is the most difficult part of the document: all responsible institutions have to be identified correctly and all consecutive numbers correlated to the right institution. Lines that cannot be addressed are written to an error output for manual monitoring and debugging.

Schema-conform documents can be further refined with two editors: a form-based editor generated from the JSON schema (ensuring syntactically valid documents), and a text-based JSON editor (which warns when the schema is violated).

From the DIP (Bundestag)

The Parlamentsdokumentation provides access to a database of legislative material (text, plenary session records, etc.) from all initiated bills through the Documentation and Information System (DIP) — a joint information system of the Bundestag and Bundesrat. It documents parliamentary events in both chambers, as recorded in printed papers and stenographic reports, and can be retrieved and exported in XML format.

From further sources

To analyse the causes and effects of a legislature's temporality, further information is included in the database:

• federal and state government coalitions,
• election dates.