Table of Contents

Title Page

Copyright

About the Author

Credits

Acknowledgments

Introduction

What This Book Covers

Chapter 1: Introduction and Overview

1.1 Tasks of the Kernel

1.2 Implementation Strategies

1.3 Elements of the Kernel

1.4 Why the Kernel Is Special

1.5 Some Notes on Presentation

1.6 Summary

Chapter 2: Process Management and Scheduling

2.1 Process Priorities

2.2 Process Life Cycle

2.3 Process Representation

2.4 Process Management System Calls

2.5 Implementation of the Scheduler

2.6 The Completely Fair Scheduling Class

2.7 The Real-Time Scheduling Class

2.8 Scheduler Enhancements

2.9 Summary

Chapter 3: Memory Management

3.1 Overview

3.2 Organization in the (N)UMA Model

3.3 Page Tables

3.4 Initialization of Memory Management

3.5 Management of Physical Memory

3.6 The Slab Allocator

3.7 Processor Cache and TLB Control

3.8 Summary

Chapter 4: Virtual Process Memory

4.1 Introduction

4.2 Virtual Process Address Space

4.3 Principle of Memory Mappings

4.4 Data Structures

4.5 Operations on Regions

4.6 Address Spaces

4.7 Memory Mappings

4.8 Reverse Mapping

4.9 Managing the Heap

4.10 Handling of Page Faults

4.11 Correction of Userspace Page Faults

4.12 Kernel Page Faults

4.13 Copying Data between Kernel and Userspace

4.14 Summary

Chapter 5: Locking and Interprocess Communication

5.1 Control Mechanisms

5.2 Kernel Locking Mechanisms

5.3 System V Interprocess Communication

5.4 Other IPC Mechanisms

5.5 Summary

Chapter 6: Device Drivers

6.1 I/O Architecture

6.2 Access to Devices

6.3 Association with the Filesystem

6.4 Character Device Operations

6.5 Block Device Operations

6.6 Resource Reservation

6.7 Bus Systems

6.8 Summary

Chapter 7: Modules

7.1 Overview

7.2 Using Modules

7.3 Inserting and Deleting Modules

7.4 Automation and Hotplugging

7.5 Version Control

7.6 Summary

Chapter 8: The Virtual Filesystem

8.1 Filesystem Types

8.2 The Common File Model

8.3 Structure of the VFS

8.4 Working with VFS Objects

8.5 Standard Functions

8.6 Summary

Chapter 9: The Extended Filesystem Family

9.1 Introduction

9.2 Second Extended Filesystem

9.3 Third Extended Filesystem

9.4 Summary

Chapter 10: Filesystems without Persistent Storage

10.1 The proc Filesystem

10.2 Simple Filesystems

10.3 Sysfs

10.4 Summary

Chapter 11: Extended Attributes and Access Control Lists

11.1 Extended Attributes

11.2 Access Control Lists

11.3 Summary

Chapter 12: Networks

12.1 Linked Computers

12.2 ISO/OSI and TCP/IP Reference Model

12.3 Communication via Sockets

12.4 The Layer Model of Network Implementation

12.5 Networking Namespaces

12.6 Socket Buffers

12.7 Network Access Layer

12.8 Network Layer

12.9 Transport Layer

12.10 Application Layer

12.11 Networking from within the Kernel

12.12 Summary

Chapter 13: System Calls

13.1 Basics of System Programming

13.2 Available System Calls

13.3 Implementation of System Calls

13.4 Summary

Chapter 14: Kernel Activities

14.1 Interrupts

14.2 Software Interrupts

14.3 Tasklets

14.4 Wait Queues and Completions

14.5 Summary

Chapter 15: Time Management

15.1 Overview

15.2 Implementation of Low-Resolution Timers

15.3 Generic Time Subsystem

15.4 High-Resolution Timers

15.5 Dynamic Ticks

15.6 Broadcast Mode

15.7 Implementing Timer-Related System Calls

15.8 Managing Process Times

15.9 Summary

Chapter 16: Page and Buffer Cache

16.1 Structure of the Page Cache

16.2 Structure of the Buffer Cache

16.3 Address Spaces

16.4 Implementation of the Page Cache

16.5 Implementation of the Buffer Cache

16.6 Summary

Chapter 17: Data Synchronization

17.1 Overview

17.2 The pdflush Mechanism

17.3 Starting a New Thread

17.4 Thread Initialization

17.5 Performing Actual Work

17.6 Periodic Flushing

17.7 Associated Data Structures

17.8 Central Control

17.9 Superblock Synchronization

17.10 Inode Synchronization

17.11 Congestion

17.12 Forced Writeback

17.13 Laptop Mode

17.14 System Calls for Synchronization Control

17.15 Full Synchronization

17.16 Summary

Chapter 18: Page Reclaim and Swapping

18.1 Overview

18.2 Page Reclaim and Swapping in the Linux Kernel

18.3 Managing Swap Areas

18.4 The Swap Cache

18.5 Writing Data Back

18.6 Page Reclaim

18.7 The Swap Token

18.8 Handling Swap-Page Faults

18.9 Initiating Memory Reclaim

18.10 Shrinking Other Caches

18.11 Summary

Chapter 19: Auditing

19.1 Overview

19.2 Audit Rules

19.3 Implementation

19.4 Summary

Appendix A: Architecture Specifics

A.1 Overview

A.2 Data Types

A.3 Alignment

A.4 Memory Pages

A.5 System Calls

A.6 String Processing

A.7 Thread Representation

A.8 Bit Operations and Endianness

A.9 Page Tables

A.10 Miscellaneous

A.11 Summary

Appendix B: Working with the Source Code

B.1 Organization of the Kernel Sources

B.2 Configuration with Kconfig

B.3 Compiling the Kernel with Kbuild

B.4 Useful Tools

B.5 Debugging and Analyzing the Kernel

B.6 User-Mode Linux

B.7 Summary

Appendix C: Notes on C

C.1 How the GNU C Compiler Works

C.2 Standard Data Structures and Techniques of the Kernel

C.3 Summary

Appendix D: System Startup

D.1 Architecture-Specific Setup on IA-32 Systems

D.2 High-Level Initialization

D.3 Summary

Appendix E: The ELF Binary Format

E.1 Layout and Structure

E.2 Data Structures in the Kernel

E.3 Summary

Appendix F: The Kernel Development Process

F.1 Introduction

F.2 Kernel Trees and the Structure of Development

F.3 The Structure of Patches

F.4 Linux and Academia

F.5 Summary

References

Advertisement

Professional Linux®Kernel Architecture

Published by

Wiley Publishing, Inc.

10475 Crosspoint Boulevard

Indianapolis, IN 46256

www.wiley.com

Copyright©2008 by Wolfgang Mauerer

Published by Wiley Publishing, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN: 978-0-470-34343-2

Library of Congress Cataloging-in-Publication Data:

Mauerer, Wolfgang, 1978-

Professional Linux kernel architecture / Wolfgang Mauerer.

p. cm.

Includes index.

ISBN 978-0-470-34343-2 (pbk.)

1. Linux. 2. Computer architecture. 3. Application software. I. Title.

QA76.9.A73M38 2008

005.4’32--dc22

2008028067

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4355, or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Website is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Website may provide or recommendations it may make. Further, readers should be aware that Internet Websites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Wrox Programmer to Programmer, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. All other trademarks are the property of their respective owners. Wiley Publishing, Inc., is not associated with any product or vendor mentioned in this book.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

About the Author

Wolfgang Mauerer is a quantum physicist whose professional interests are centered around quantum cryptography, quantum electrodynamics, and compilers for—you guessed it—quantum architectures. With the confirmed capacity of being the worst experimentalist in the known universe, he sticks to the theoretical side of his profession, which is especially reassuring considering his constant fear of accidentally destroying the universe. Outside his research work, he is fascinated by operating systems, and for more than a decade—starting with an article series about the kernel in 1997—he has found great pleasure in documenting and explaining Linux kernel internals. He is also the author of a book about typesetting with LaTeX and has written numerous articles that have been translated into seven languages in total.

When he's not submerged in vast Hilbert spaces or large quantities of source code, he tries to take the opposite direction, namely, upward—be this with model planes, a paraglider, or on foot with an ice axe in his hands: Mountains especially have the power to outrival even the Linux kernel. Consequently, he considers planning and accomplishing a first-ascent expedition to the vast arctic glaciers of east Greenland to be the really unique achievement in his life.

Being interested in everything that is fundamental, he is also the author of the first compiler for Plankalkül, the world's earliest high-level language devised in 1942–1946 by Konrad Zuse, the father of the computer. As an avid reader, he is proud that despite the two-digit number of computers present in his living room, the volume required for books still occupies a larger share.

Credits

Executive Editor

Carol Long

Senior Development Editor

Tom Dinse

Production Editor

Debra Banninger

Copy Editors

Cate Caffrey

Kathryn Duggan

Editorial Manager

Mary Beth Wakefield

Production Manager

Tim Tate

Vice President and Executive Group Publisher

Richard Swadley

Vice President and Executive Publisher

Joseph B. Wikert

Project Coordinator, Cover

Lynsey Stanford

Proofreader

Publication Services, Inc.

Indexer

Jack Lewis

Acknowledgments

First and foremost, I have to thank the thousands of programmers who have created the Linux kernel over the years—most of them commercially based, but some also just for their own private or academic joy. Without them, there would be no kernel, and I would have had nothing to write about. Please accept my apologies that I cannot list all several hundred names here, but in true UNIX style, you can easily generate the list by:

for file in $ALL_FILES_COVERED_IN_THIS_BOOK; do 

         git log --pretty=“format:%an” $file; done |

sort -u -k 2,2

It goes without saying that I admire your work very much—you are all the true heroes in this story!

What you are reading right now is the result of an evolution over more than seven years: After two years of writing, the first edition was published in German by Carl Hanser Verlag in 2003. It then described kernel 2.6.0. The text was used as a basis for the low-level design documentation for the EAL4+ security evaluation of Red Hat Enterprise Linux 5, requiring to update it to kernel 2.6.18 (if the EAL acronym does not mean anything to you, then Wikipedia is once more your friend). Hewlett-Packard sponsored the translation into English and has, thankfully, granted the rights to publish the result. Updates to kernel 2.6.24 were then performed specifically for this book.

Several people were involved in this evolution, and my appreciation goes to all of them: Leslie Mackay-Poulton, with support from David Jacobs, did a tremendous job at translating a huge pile of text into English. I'm also indebted to Sal La Pietra of atsec information security for pulling the strings to get the translation project rolling, and especially to Stephan Müller for close cooperation during the evaluation. My cordial thanks also go to all other HP and Red Hat people involved in this evaluation, and also to Claudio Kopper and Hans Löhr for our very enjoyable cooperation during this project. Many thanks also go to the people at Wiley—both visible and invisible to me—who helped to shape the book into its current form.

The German edition was well received by readers and reviewers, but nevertheless comments about inaccuracies and suggestions for improvements were provided. I'm glad for all of them, and would also like to mention the instructors who answered the publisher's survey for the original edition. Some of their suggestions were very valuable for improving the current publication. The same goes for the referees for this edition, especially to Dr. Xiaodong Zhang for providing numerous suggestions for Appendix F.4.

Furthermore, I express my gratitude to Dr. Christine Silberhorn for granting me the opportunity to suspend my regular research work at the Max Planck Research Group for four weeks to work on this project. I hope you enjoyed the peace during this time when nobody was trying to install Linux on your MacBook!

As with every book, I owe my deepest gratitude to my family for supporting me in every aspect of life—I more than appreciate this indispensable aid. Finally, I have to thank Hariet Fabritius for infinite patience with an author whose work cycle not only perfectly matched the most alarming forms of sleep dyssomnias, but who was always right on the brink of confusing his native tongue with “C,” and whom she consequently had to rescue from numerous situations where he seemingly had lost his mind (see below…). Now that I have more free time again, I'm not only looking forward to our well-deserved holiday, but can finally embark upon the project of giving your laptop all joys of a proper operating system! (Writing these acknowledgments, I all of a sudden realize why people always hasten to lock away their laptops when they see me approaching….)

Introduction

UNIX is simple and coherent, but it takes a genius (or at any rate a programmer) to understand and appreciate the simplicity.

—Dennis Ritchie

Note from the authors: Yes, we have lost our minds. Be forewarned: You will lose yours too.

—Benny Goodheart & James Cox

UNIX is distinguished by a simple, coherent, and elegant design—truly remarkable features that have enabled the system to influence the world for more than a quarter of a century. And especially thanks to the growing presence of Linux, the idea is still picking up momentum, with no end of the growth in sight.

UNIX and Linux carry a certain fascination, and the two quotations above hopefully capture the spirit of this attraction. Consider Dennis Ritchie's quote: Is the coinventor of UNIX at Bell Labs completely right in saying that only a genius can appreciate the simplicity of UNIX? Luckily not, because he puts himself into perspective immediately by adding that programmers also qualify to value the essence of UNIX.

Understanding the meagerly documented, demanding, and complex sources of UNIX as well as of Linux is not always an easy task. But once one has started to experience the rich insights that can be gained from the kernel sources, it is hard to escape the fascination of Linux. It seems fair to warn you that it's easy to get addicted to the joy of the operating system kernel once starting to dive into it. This was already noted by Benny Goodheart and James Cox, whose preface to their book The Magic Garden Explained (second quotation above) explained the internals of UNIX System V. And Linux is definitely also capable of helping you to lose your mind!

This book acts as a guide and companion that takes you through the kernel sources and sharpens your awareness of the beauty, elegance, and—last but not least—esthetics of their concepts. There are, however, some prerequisites to foster an understanding of the kernel. C should not just be a letter; neither should it be a foreign language. Operating systems are supposed to be more than just a “Start” button, and a small amount of algorithmics can also do no harm. Finally, it is preferable if computer architecture is not just about how to build the most fancy case. From an academic point of view, this comes closest to the lectures “Systems Programming,” “Algorithmics,” and “Fundamentals of Operating Systems.” The previous edition of this book has been used to teach the fundamentals of Linux to advanced undergraduate students in several universities, and I hope that the current edition will serve the same purpose.

Discussing all aforementioned topics in detail is outside the scope of this book, and when you consider the mass of paper you are holding in your hands right now (or maybe you are not holding it, for this very reason), you'll surely agree that this would not be a good idea. When a topic not directly related to the kernel, but required to understand what the kernel does, is encountered in this book, I will briefly introduce you to it. To gain a more thorough understanding, however, consult the books on computing fundamentals that I recommend. Naturally, there is a large selection of texts, but some books that I found particularly insightful and illuminating include C Programming Language, by Brian W. Kernighan and Denis M. Ritchie [KR88]; Modern Operating Systems, by Andrew S. Tanenbaum [Tan07] on the basics of operating systems in general; Operating Systems: Design and Implementation, by Andrew S. Tanenbaum and Albert S. Woodhull [TW06] on UNIX (Minix) in particular; Advanced Programming in the Unix Environment, by W. Richard Stevens and Stephen A. Rago [SR05] on userspace programming; and the two volumes Computer Architecture and Computer Organization and Design, on the foundations of computer architecture by John L. Hennessy and David A. Patterson [HP06, PH07]. All have established themselves as classics in the literature.

Additionally, Appendix C contains some information about extensions of the GNU C compiler that are used by the kernel, but do not necessarily find widespread use in general programming.

When the first edition of this book was written, a schedule for kernel releases was more or less nonexistent. This has changed drastically during the development of kernel 2.6, and as I discuss in Appendix F, kernel developers have become pretty good at issuing new releases at periodic, predictable intervals. I have focused on kernel 2.6.24, but have also included some references to 2.6.25 and 2.6.26, which were released after this book was written but before all technical publishing steps had been completed. Since a number of comprehensive changes to the whole kernel have been merged into 2.6.24, picking this release as the target seems a good choice. While a detail here or there will have changed in more recent kernel versions as compared to the code discussed in this book, the big picture will remain the same for quite some time.

In the discussion of the various components and subsystems of the kernel, I have tried to avoid overloading the text with unimportant details. Likewise, I have tried not to lose track of the connection with source code. It is a very fortunate situation that, thanks to Linux, we are able to inspect the source of a real, working, production operating system, and it would be sad to neglect this essential aspect of the kernel. To keep the book's volume below the space of a whole bookshelf, I have selected only the most crucial parts of the sources. Appendix F introduces some techniques that ease reading of and working with the real source, an indispensable step toward understanding the structure and implementation of the Linux kernel.

One particularly interesting observation about Linux (and UNIX in general) is that it is well suited to evoke emotions. Flame wars on the Internet and heated technical debates about operating systems may be one thing, but for which other operating system does there exist a handbook (The Unix-Haters Handbook, edited by Simson Garfinkel et al. [GWS94]) on how best to hate it? When I wrote the preface to the first edition, I noted that it is not a bad sign for the future that a certain international software company responds to Linux with a mixture of abstruse accusations and polemics. Five years later, the situation has improved, and the aforementioned vendor has more or less officially accepted the fact that Linux has become a serious competitor in the operating system world. And things are certainly going to improve even more during the next five years….

Naturally (and not astonishingly), I admit that I am definitely fascinated by Linux (and, sometimes, am also sure that I have lost my mind because of this), and if this book helps to carry this excitement to the reader, the long hours (and especially nights) spent writing it were worth every minute!

Suggestions for improvements and constrictive critique can be passed to wm@linux-kernel.net, or via www.wrox.com. Naturally, I'm also happy if you tell me that you liked the book!

What This Book Covers

This book discusses the concepts, structure, and implementation of the Linux kernel. In particular, the individual chapters cover the following topics:

Chapter 1 provides an overview of the Linux kernel and describes the big picture that is investigated more closely in the following chapters.

Chapter 2 talks about the basics of multitasking, scheduling, and process management, and investigates how these fundamental techniques and abstractions are implemented.

Chapter 3 discusses how physical memory is managed. Both the interaction with hardware and the in-kernel distribution of RAM via the buddy system and the slab allocator are covered.

Chapter 4 proceeds to describe how userland processes experience virtual memory, and the comprehensive data structures and actions required from the kernel to implement this view.

Chapter 5 introduces the mechanisms required to ensure proper operation of the kernel on multiprocessor systems. Additionally, it covers the related question of how processes can communicate with each other.

Chapter 6 walks you through the means for writing device drivers that are required to add support for new hardware to the kernel.

Chapter 7 explains how modules allow for dynamically adding new functionality to the kernel.

Chapter 8 discusses the virtual filesystem, a generic layer of the kernel that allows for supporting a wide range of different filesystems, both physical and virtual.

Chapter 9 describes the extended filesystem family, that is, the Ext2 and Ext3 filesystems that are the standard workhorses of many Linux installations.

Chapter 10 goes on to discuss procfs and sysfs, two filesystems that are not designed to store information, but to present meta-information about the kernel to userland. Additionally, a number of means to ease writing filesystems are presented.

Chapter 11 shows how extended attributes and access control lists that can help to improve system security are implemented.

Chapter 12 discusses the networking implementation of the kernel, with a specific focus on IPv4, TCP, UDP, and netfilter.

Chapter 13 introduces how systems calls that are the standard way to request a kernel action from userland are implemented.

Chapter 14 analyzes how kernel activities are triggered with interrupts, and presents means of deferring work to a later point in time.

Chapter 15 shows how the kernel handles all time-related requirements, both with low and high resolution.

Chapter 16 talks about speeding up kernel operations with the help of the page and buffer caches.

Chapter 17 discusses how cached data in memory are synchronized with their sources on persistent storage devices.

Chapter 18 introduces how page reclaim and swapping work.

Chapter 19 gives an introduction to the audit implementation, which allows for observing in detail what the kernel is doing.

Appendix A discusses peculiarities of various architectures supported by the kernel.

Appendix B walks through various tools and means of working efficiently with the kernel sources.

Appendix C provides some technical notes about the programming language C, and also discusses how the GNU C compiler is structured.

Appendix D describes how the kernel is booted.

Appendix E gives an introduction to the ELF binary format.

Appendix F discusses numerous social aspects of kernel development and the Linux kernel community.

Chapter 1

Introduction and Overview

Operating systems are not only regarded as a fascinating part of information technology, but are also the subject of controversial discussion among a wide public.1 Linux has played a major role in this development. Whereas just 10 years ago a strict distinction was made between relatively simple academic systems available in source code and commercial variants with varying performance capabilities whose sources were a well-guarded secret, nowadays anybody can download the sources of Linux (or of any other free systems) from the Internet in order to study them.

Linux is now installed on millions of systems and is used by home users and professionals alike for a wide range of tasks. From miniature embedded systems in wristwatches to massively parallel mainframes, there are countless ways of exploiting Linux productively. And this makes the sources so interesting. A sound, well-established concept (UNIX) melded with powerful innovations and a strong penchant for dealing with problems that do not arise in academic teaching systems—this is what makes Linux so fascinating.

This book describes the central functions of the kernel, explains its underlying structures, and examines its implementation. Because complex subjects are discussed, I assume that the reader already has some experience in operating systems and systems programming in C (it goes without saying that I assume some familiarity with using Linux systems). I touch briefly on several general concepts relevant to common operating system problems, but my prime focus is on the implementation of the Linux kernel. Readers unfamiliar with a particular topic will find explanations on relevant basics in one of the many general texts on operating systems; for example, in Tanenbaum's outstanding introductions ([TW06] and [Tan07]). A solid foundation of C programming is required. Because the kernel makes use of many advanced techniques of C and, above all, of many special features of the GNU C compiler, Appendix C discusses the finer points of C with which even good programmers may not be familiar. A basic knowledge of computer structures will be useful as Linux necessarily interacts very directly with system hardware—particularly with the CPU. There are also a large number of introductory works dealing with this subject; some are listed in the reference section. When I deal with CPUs in greater depth (in most cases I take the IA-32 or AMD64 architecture as an example because Linux is used predominantly on these system architectures), I explain the relevant hardware details. When I discuss mechanisms that are not ubiquitous in daily live, I will explain the general concept behind them, but expect that readers will also consult the quoted manual pages for more advice on how a particular feature is used from userspace.

The present chapter is designed to provide an overview of the various areas of the kernel and to illustrate their fundamental relationships before moving on to lengthier descriptions of the subsystems in the following chapters.

Since the kernel evolves quickly, one question that naturally comes to mind is which version is covered in this book. I have chosen kernel (2.6.24), which was released at the end of January 2008. The dynamic nature of kernel development implies that a new kernel version will be available by the time you read this, and naturally, some details will have changed—this is unavoidable. If it were not the case, Linux would be a dead and boring system, and chances are that you would not want to read the book. While some of the details will have changed, concepts will not have varied essentially. This is particularly true because 2.6.24 has seen some very fundamental changes as compared to earlier versions. Developers do not rip out such things overnight, naturally.

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!