Table of Contents

Preface

PART 1. MATHEMATICAL STATISTICS

Chapter 1. Introduction to Mathematical Statistics

1.1. Generalities

1.2. Examples of statistics problems

Chapter 2. Principles of Decision Theory

2.1. Generalities

2.2. The problem of choosing a decision function11

2.3. Principles of Bayesian statistics

2.4. Complete classes

2.5. Criticism of decision theory – the asymptotic point of view

2.6. Exercises

Chapter 3. Conditional Expectation

3.1. Definition

3.2. Properties and extension

3.3. Conditional probabilities and conditional distributions

3.4. Exercises

Chapter 4. Statistics and Sufficiency

4.1. Samples and empirical distributions

4.2. Sufficiency

4.3. Examples of sufficient statistics – an exponential model

4.4. Use of a sufficient statistic

4.5. Exercises

Chapter 5. Point Estimation

5.1. Generalities

5.2. Sufficiency and completeness

5.3. The maximum-likelihood method

5.4. Optimal unbiased estimators

5.5. Efficiency of an estimator

5.6. The linear regression model

5.7. Exercises

Chapter 6. Hypothesis Testing and Confidence Regions

6.1. Generalities

6.2. The Neyman–Pearson (NP) lemma

6.3. Multiple hypothesis tests (general methods)

6.4. Case where the ratio of the likelihoods is monotonic

6.5. Tests relating to the normal distribution

6.6. Application to estimation: confidence regions

6.7. Exercises

Chapter 7. Asymptotic Statistics

7.1. Generalities

7.2. Consistency of the maximum likelihood estimator

7.3. The limiting distribution of the maximum likelihood estimator

7.4. The likelihood ratio test

7.5. Exercises

Chapter 8. Non-Parametric Methods and Robustness

8.1. Generalities

8.2. Non-parametric estimation

8.3. Non-parametric tests

8.4. Robustness

8.5. Exercises

PART 2. STATISTICS FOR STOCHASTIC PROCESSES

Chapter 9. Introduction to Statistics for Stochastic Processes

9.1. Modeling a family of observations

9.2. Processes

9.3. Statistics for stochastic processes

9.4. Exercises

Chapter 10. Weakly Stationary Discrete-Time Processes

10.1. Autocovariance and spectral density

10.2. Linear prediction and Wold decomposition

10.3. Linear processes and the ARMA model

10.4. Estimating the mean of a weakly stationary process

10.5. Estimating the autocovariance

10.6. Estimating the spectral density

10.7. Exercises

Chapter 11. Poisson Processes – A Probabilistic and Statistical Study

11.1. Introduction

11.2. The axioms of Poisson processes

11.3. Interarrival time

11.4. Properties of the Poisson process

11.5. Notions on generalized Poisson processes

11.6. Statistics of Poisson processes

11.7. Exercises

Chapter 12. Square-Integrable Continuous-Time Processes

12.1. Definitions

12.2. Mean-square continuity

12.3. Mean-square integration

12.4. Mean-square differentiation

12.5. The Karhunen–Loeve theorem

12.6. Wiener processes

12.7. Notions on weakly stationary continuous-time processes

12.8. Exercises

Chapter 13. Stochastic Integration and Diffusion Processes

13.1. Itô integral

13.2. Diffusion processes

13.3. Processes defined by stochastic differential equations and stochastic integrals

13.4. Notions on statistics for diffusion processes

13.5. Exercises

Chapter 14. ARMA Processes

14.1. Autoregressive processes

14.2. Moving average processes

14.3. General ARMA processes

14.4. Non-stationary models

14.5. Statistics of ARMA processes

14.6. Multidimensional processes

14.7. Exercises

Chapter 15. Prediction

15.1. Generalities

15.2. Empirical methods of prediction

15.3. Prediction in the ARIMA model

15.4. Prediction in continuous time

15.5. Exercises

PART 3. SUPPLEMENT

Chapter 16. Elements of Probability Theory

16.1. Measure spaces: probability spaces

16.2. Measurable functions: real random variables

16.3. Integrating real random variables

16.4. Random vectors

16.5. Independence

16.6. Gaussian vectors

16.7. Stochastic convergence

16.8. Limit theorems

Appendix. Statistical Tables

A1.1. Random numbers

A1.2. Distribution function of the standard normal distribution

A1.3. Density of the standard normal distribution

A1.4. Percentiles (tp) of Student’s distribution

A1.5. Ninety-fifth percentiles of Fisher–Snedecor distributions

A1.6. Ninety-ninth percentiles of Fisher–Snedecor distributions

A1.7. Percentiles images of the χ2 distribution with n degrees of freedom

A1.8. Individual probabilities of the Poisson distribution

A1.9. Cumulative probabilities of the Poisson distribution

A1.10. Binomial coefficients images for n ≤ 30 and 0 ≤ k ≤ 7

A1.11. Binomial coefficients images for n ≤ 30 and 8 ≤ k ≤ 15

Bibliography

Index

First published 2012 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:

ISTE Ltd | John Wiley & Sons, Inc.
27-37 St George’s Road | 111 River Street
London SW19 4EU | Hoboken, NJ 07030
UK | USA
www.iste.co.uk | www.wiley.com

© ISTE Ltd 2012

The rights of Denis Bosq to be identified as the author of this work have been asserted by him in accordance with the Copyright, Designs and Patents Act 1988.

Library of Congress Cataloging-in-Publication Data

Bosq, Denis, 1939-

Mathematical statistics and stochastic processes / Denis Bosq.

p. cm.

Includes bibliographical references and index.

ISBN 978-1-84821-361-6

1. Mathematical statistics. 2. Stochastic processes. I. Title.

QA276.8.B672 2012

519.5--dc23

2012006812

British Library Cataloguing-in-Publication Data

A CIP record for this book is available from the British Library

ISBN: 978-1-84821-361-6

Preface

This book is dedicated to the mathematical modeling of statistical phenomena. It attempts to fill a gap: most textbooks on mathematical statistics only treat the case of sampling, and yet, in applications, the observed variables are very often correlated – examples are numerous in physics, chemistry, biology, demography, economics and finance. It is for this reason that we have given an important place to process statistics.

This book is divided into three parts: Mathematical Statistics, Statistics for Stochastic Processes, and a supplement on probability. The first part begins with decision theory, to develop the classical theory of estimation and tests. It then adopts an asymptotic viewpoint, in particular, in a non-parametric framework. In the second part, we first study the statistics of discrete-time stationary processes, and then the statistics of Poisson processes. The third part is dedicated to continuous-time processes: we define the Itô integral and give some results for the statistics of second-order process and for diffusion. Statistical prediction is addressed in the last chapter. Finally, the principles of probability are described, based on measure theory.

The mathematical level is that of a master’s degree in applied mathematics: I have based the material on some lectures given at UPMC (Paris 6) in France, particularly [BOS 78] and [BOS 93].

A certain number of exercises, some original, illustrate the contents of this textbook. I would like to thank the teachers who contributed in putting them together, notably Gérard Biau, Delphine Blanke, Jérôme Dedecker, and Florence Merlevède.

I would also like to thank François-Xavier Lejeune for his excellent write-up of the manuscript, and for his contribution to the development of this book.

Finally, I would like to thank Maximilian Lock for all his work on this book.

PART 1

Mathematical Statistics

Chapter 1

Introduction to Mathematical Statistics

1.1. Generalities

We may define statistics as the set of methods that allow, from the observation of a random phenomenon, the obtainment of information about the probability associated with this phenomenon.

It is important to note that the random character that we attribute to the considered phenomenon is often merely a way to translate our ignorance of the laws that govern it. Also, a preliminary study, taking into account only the observations made, proves interesting; this is the aim of data analysis.

Data analysis is the set of techniques of statistical description whose purpose is the treatment of data without any probabilistic hypotheses. It aims, in particular, to reveal the dominant parameters among those upon which the observation depends.

Descriptive statistics also treats observations without formulating any prior hypotheses, but we may consider these hypotheses to be underlying, since it essentially consists of studying the empirical probabilistic characteristics of observations (histograms, empirical moments, etc.).

However, we may include data analysis in descriptive statistics by defining it as the set of methods that allow the ordering of data and their presentation in a usable form.

Furthermore, simulation is an additional technique often used in statistics: it consists of carrying out fictitious experiments in such a way as to make visible the expressions of chance in the evolution of a phenomenon. A simple, important simulation problem is that of the generation of random numbers, that is, the generation of a sequence of numbers x1, …, xn, which may be considered as the realizations of random variables X1, …, Xn, the distribution of (X1, …, Xn) being given.

The techniques indicated above fall essentially within applied statistics and we will only discuss them sporadically in this book. We will, in general, focus on the framework of mathematical statistics, that is, theoretical statistics based on measure theory and, in part, on decision theory.

1.2. Examples of statistics problems

We will first give some examples of statistics problems, emphasizing the construction of the associated probability model.

1.2.1. Quality control

A manufacturer receives a batch of objects containing an unknown proportion of defective objects. Supposing the number of objects in the batch to be large, an audit may be carried out only by taking a sample of objects from the batch. Given the number of defective objects in the sample, the manufacturer will accept or reject the batch. We may associate several probability models with this problem:

1) Let E be the batch of objects, Ω the set of subsets of E with r elements given with a uniform distribution, and X the random variable “number of defective objects among the r objects drawn”. We know that X follows a hypergeometric distribution with parameters n, n1, and r images where n = cardE and n1 is the number of defective objects:

2) If n and n1 are large relative to r, we may use the binomial approximation and suppose that X follows the distribution images. This comes from the fact that when n → ∞, n1/n → p > 0,

3) If r is large compared to p, we may suppose that X follows the Poisson distribution images where λ = rp. This is because

when r → ∞, rp → λ > 0.

Since n1 is unknown, so too are the parameters of the above distributions. We must therefore consider the triplet images where images designates the set of hypergeometric distributions with parameters (n, n1, r), where n and r are fixed, and images.

We pose n1/n = p and set a proportion p0 of defective objects beyond which the batch will be refused. It is therefore necessary to determine, in light of the r objects drawn, if p > p0 or if p ≤ p0, which will allow us to accept or reject the batch. This is a testing problem (we “test” the quality of the batch).

The choice of a decision criterion is then based on the fact that we may commit two types of error: accept a bad batch or reject a good one. We therefore aim to minimize these measurement errors as much as possible.

1.2.2. Measurement errors

A physicist measures a real value a certain number of times. The values found are not exact due to measurement errors. The problem is then which value to accept for the measured quantity.

To construct the associated probability model, we generally make the following hypothesis: the measurement errors have extremely varied causes (lack of precision or unreliable instruments, misreading by the experimenter, etc.); we may admit as a first approximation that these “causes” are independent of each other: the error is therefore the sum of a large number of small, independent errors.

The central limit theorem allows us to assert that this error follows (approximately) a normal distribution. Moreover, we may often, for reasons of symmetry, suppose that the measurements carried out have the same expectation (mean) value as the quantity considered.

We may therefore associate n independent observations of this quantity with the triplet images where I and J are intervals of images and images, respectively. We must therefore determine m in as precise a way as possible: this is a problem of estimation.

1.2.3. Filtering

An economist observes the evolution of the price of a certain product in the time interval [t1, t2]; he seeks to predict the price of this product at the time t3 (>t2).

This random phenomenon may be modeled in the following way: we consider a family (ξt, t ≥ t1) of real random variables where ξt represents the price of the product at time t. It is therefore a question of predicting, in light of the realizations of the random variables ξt, t1 ≤ t ≤ t2, the best possible value of ξt₃.

If the distributions of the random variables ξt, or the correlations that exist between them, are not fully known, this problem of prediction falls within the domain of statistics.

The problem of interpolation is of an analogous nature: it concerns the determination of the best possible ξt₀ given ξt, where t ∈ [t1, t2] ∪ [t3, t4] with t2 < t0 < t3.

Prediction and interpolation are two particular cases of general filtering problems, or in other words problems of the estimation of an unobserved random variable Y from an observed random variable X 1.

1.2.4. Confidence intervals

We consider a random experiment with two outcomes (written 0 and 1) that we independently repeat n times. We seek to estimate the distribution of this experiment (i.e. the associated probability Pp in {0, 1}); for this, it is sufficient to estimate p = Pp ({1}).

The associated model is written ({0, 1}n, images, where p ∈]0, 1[), and a natural way to estimate p is to use Nn (ω)/n where Nn (ω) designates the number of 1s contained in ω = (i1, …, in), where ij equals 0 or 1.

To determine the precision of this estimate, we may evaluate

and we then say that with a confidence of 1 − α, Nn/n is an estimator of p to within ε, or equally that p belongs to the confidence interval images with a confidence level 1 − α.

For the calculation of α, we may, when n is large, use the normal approximation of images for which we write

an approximation valid for p ∈]0, 1[.

1.2.5. Homogeneity testing

A doctor wants to test a medication; for this, he chooses a first group of patients to whom the medication is administered, and a second group consists of patients who receive a placebo.

Let Xi be a random variable associated with the ith patient of the first group, which conveys the obtained result: cure, improvement, aggravation, stationary state, etc. We define the variable Yi associated with the jth patient of the second group in a similar way.

The testing problem may then be formulated in the following way: let P1 be the distribution of the Xi and P2 be the distribution of the Yj; do we have P1 = P2? This is a homogeneity test.

1 A general solution to filtering problems in the case where the distribution of the pair (X, Y) is known is given in Chapter 3.

Chapter 2

Principles of Decision Theory

2.1. Generalities

The examples given in Chapter 1 show that a statistical problem may be characterized by the following elements:

a probability distribution that is not entirely known;

observations;

a decision to be made.

Formalization: Wald’s decision theory provided a common framework for statistics problems. We take:

1) A triplet images where P is a family of probabilities on the measurable space images is called a statistical model. We often set P = {Pθ, θ Θ} where we suppose that images is injective, and Θ is called the parameter space.

2) A measurable space images is called a decision (or action) space.

3) A set images of measurable mappings images is called a set of decision functions (d.f.) (or decision rules).

Description: From an observation that follows an unknown distribution P P, a statistician chooses an element a D using an element d from images.

Preference relation: To guide his/her choice, the statistician takes a preorder (i.e. a binary relation that is reflexive and transitive) on images. One such preorder is called a preference relation. We will write it as images so that images reads “d1 is preferable to d2”. We say that d1 is strictly preferable to d2 if images and images.

The statistician is therefore concerned with the choice of a “good” decision as defined by the preference relation considered.

Risk function: One convenient way of defining a preference relation on images is the following:

1) Θ being provided with a σ-algebra images, we take a measurable map L from images to images called a loss function.

2) We set:

where R is called the risk function associated with L; it is often written in the following form:

where X denotes a random variable with distribution Pθ1 and Eθ the expected value associated with Pθ. We note that R takes values in images.

3) We say that d1 is preferable to d2 if:

[2.1] images

We will say that d1 is strictly preferable to d2 if [2.1] holds, and if there exists θ Θ such that:

INTERPRETATION 2.1. L(θ, a) represents the loss incurred by the decision a when the probability of X is Pθ (or when the parameter is equal to θ). R(θ, d) then represents the average loss associated with the decision function d. The best possible choice in light of the preference relation defined by [2.1] becomes the decision that minimizes the average loss, whatever the value of the parameter.

EXAMPLE 2.1.

1) In the example of quality control (), the decision space contains two elements: (accept the batch) and (reject the batch). A decision rule is therefore a map from {1, …, } to {, }.

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!