Abstract. Let M be a real analytic manifold, let X be a complexification of M and denote by T

OF MICRODIFFERENTIAL SYSTEMS

Andrea D’Agnolo

Abstract. Let M be a real analytic manifold, let X be a complexification of M and denote by T

X the cotangent bundle to X. Let M be a system of microdifferential operators defined on an open subset U ⊂ T

X and denote by V its characteristic variety.

In this paper we make the following assumptions: M has simple characteristics along V , the intersection of V and the conormal bundle T

X to M in X is clean and regular, the generalized Levi form of V with respect to T

X has a constant number s

of negative eigenvalues.

With these hypotheses it is known (cf [K-S 1]) that the complex RHom

(M, C

) of microfunction solutions of the system M is concentrated in degree s

. Denote by V ∩ T ^

X the union of the characteristic leaves of V issued from V ∩ T

X. Using a result of [S-T], we prove here that the sheaf Ext

X

(M, C

) is “flabby transver- sally to the leaves of V ” as soon as there is no germ of complex curve in V ∩ T ^

X contained in such a leaf.

In particular, when the generalized Levi form of V with respect to T

X is non degenerate, we recover the conical flabbiness of the sheaf Ext

(M, C

) which was proven in [S-K-K].

1. Notations.

1.1. Let X be a complex analytic manifold and let M ⊂ X be a real analytic submanifold. One denotes by π : T ^∗ X → X the cotangent bundle to X, by T _M ^∗ X the conormal bundle to M in X and by α the canonical one-form on T ^∗ X.

One denotes by s ^± (M, p) the numbers of positive and negative eigenvalues for the Levi form of M at p ∈ T _M ^∗ X.

To a morphism f : X → Y one associates the natural mappings

T ^∗ X

t

f

← X × _Y T ^∗ Y → T ^f

^∗ Y.

1.2. Let Λ be a real analytic homogeneous symplectic manifold of dimension 2n and let E be a complexification of Λ (endowed with the induced homogeneous complex symplectic structure). Let V be a regular smooth involutive complex submanifold of E of codimension l and let ϕ ₁ = · · · = ϕ _l = 0 be a system of local equations for V at p ∈ V with {ϕ _i , ϕ _j } ≡ 0 (where {·, ·} denotes the Poisson bracket);

Appeared in: Ann. Scuola Norm. Sup. Pisa Cl. Sci. (4) 19 (1992), no. 2, 305–311.

1

Definition 1.1. (cf [S-K-K]) The generalized Levi form of V with respect to Λ at p ∈ V ∩ Λ, denoted L _Λ (V, p), is the Hermitian form of matrix

{ϕ _i , ϕ ^c _j }(p)

1≤i,j≤l

where ϕ ^c _j denotes the holomorphic conjugate of ϕ _j with respect to Λ. (Notice that this definition does not depend on the choice of the system of equations for V .)

Choose a system of local homogeneous symplectic coordinates (z; ζ) in E at p ∈ V such that ζ 1 = ϕ 1 , . . . , ζ l = ϕ l and set X = {ζ = 0}, Y = {ζ = z 1 = · · · = z l = 0}.

Let f : X → Y be the projection f (z) = (z _l+1 , . . . , z _n ). Consider the natural mapping f π : V → T ^∗ Y given by the identification V ∼ = X × Y T ^∗ Y ⊂ T ^∗ X and note that f _π ⁻¹ f _π (p) is the leaf of V issued from p.

Definition 1.2. Let F be a sheaf on V and W ⊂ V a locally closed subset. We will say that F is isotropically flabby on W if for every p ∈ W there exists a neighborhood Ω of p in V such that F | _Ω ∼ = (f _π ^{− 1} G)| _Ω for a sheaf G on T ^∗ Y which is conically flabby on f π (Ω). (Notice that this definition does not depend on the choice of coordinates at p.)

1.3. We will also recall some notations from the microlocal theory of sheaves as in [K-S 1].

One denotes by D ^b (X) the derived category of the category of bounded complexes of sheaves of C-vector spaces on a real analytic manifold X.

For F an object of D ^b (X), one denotes by SS(F ) its micro-support, a closed conic involutive subset of T ^∗ X.

If M is a real analytic submanifold of X, one denotes by µ M (F ) the Sato’s microlocalization of F along M , an object of D ^b (T _M ^∗ X). Recall the identification µ M (F ) ∼ = µhom(C M , F ), where µhom denotes the microlocalization bifunctor of [K-S 1] and, for a locally closed subset A ⊂ X, C _A denotes the sheaf which is 0 on X \ A and the constant sheaf with fiber C on A.

For Ω ⊂ T ^∗ X, one denotes by D ^b (X; Ω) the localization of D ^b (X) at Ω. Recall that the objects of D ^b (X; Ω) are the same as those of D ^b (X) and that a morphism h : F → G in D ^b (X) becomes an isomorphism in D ^b (X; Ω) if SS(H) ∩ Ω = ∅, where H is the third term of a distinguished triangle F → G → H ^{h +1} →.

Recall that µhom induces a functor from D ^b (X; Ω) ^◦ × D ^b (X; Ω) to D ^b (Ω).

1.4. Let X be a complex analytic manifold. One denotes by O _X the sheaf of germs of holomorphic functions on X and by E _X the sheaf of finite order microdifferential operators on T ^∗ X.

If X is a complexification of a real analytic manifold M , recall that the sheaf C _M of Sato’s microfunctions is defined by C _M = µ _M (O _X ) ⊗ or _M/X [dim ^R M ], where or _M/X denotes the relative orientation sheaf.

2. Statement of the results.

2.1. Let M be a real analytic manifold and X be a complexification of M . Let M be a left coherent E _X -module defined on an open subset U ⊂ T ^∗ X. Denote by V ⊂ T ^∗ X its characteristic variety and set l = codim ^C V . We will make the following hypotheses:

(2.1)

 



 

M has simple characteristics along V, V ∩ T _M ^∗ X is clean,

α| _{V ∩T}

_X 6= 0.

Note that (2.1) implies in particular that V is a smooth regular involutive subman- ifold of T ^∗ X.

For p ∈ V ∩ T _M ^∗ X we will denote by s ^± (M, V, p) the numbers of positive and negative eigenvalues for L _T

_X (V, p).

Let us recall the following classical result (cf also [D’A-Z] for further develop- ments):

Theorem 2.1. (cf [K-S 1, Theorem 3.1]) Let M be a left coherent E _X -module satisfying (2.1) and assume that for an open subset W ⊂ V ∩ T _M ^∗ X

(i) s ⁻ (M, V, p) is constant, say s ⁻ (M, V ), on W .

Then the complex RHom E

(M, C _M ) of microfunction solutions of the system M is concentrated in degree s ⁻ (M, V ) on W .

As we will recall in the next section, in the proof of Theorem 2.1 given in [K-S 1], the complex RHom E

(M, C M ) is “reduced” to a complex µ N (O Y )[1 − d] of mi- crofunctions along a real analytic hypersurface N of a complex analytic manifold Y (for a shift 1−d which is calculated). Concerning such a complex of microfunctions, we will need the following result.

Theorem 2.2. (cf [S-T, Theorem 3.2]) Let ω be an open subset of ˙ T _N ^∗ Y and assume:

(i) s ⁻ (N, q) is constant, say s ⁻ (N ), on ω, (ii) there is no germ of complex curve in ω.

Then:

(a) the complex µ N (O Y ) is concentrated in degree s ⁻ (N ), (b) the sheaf H ^s

^{(N )} (µ _N (O _Y )) is conically flabby on ω.

2.2. Set V ∩ T ^ _M ^∗ X = ∪ _{p∈V ∩T}

_X Σ _p , where Σ _p denotes the leaf of V issued from p.

Combining the previous results we can now state our main theorem.

Theorem 2.3. Let M be a left coherent E _X -module satisfying (2.1) and assume that for an open subset W ⊂ V ∩ T _M ^∗ X:

(i) s ⁻ (M, V, p) is constant, say s ⁻ (M, V ), on W ,

(ii) for any germ γ of complex curve at p ∈ W contained in V ∩ T ^ _M ^∗ X, one has γ ⊂ Σ p .

Then the sheaf Ext ^s _E

^{(M,V )} (M, C M ) is isotropically flabby on W . We will also recover the following result.

Corollary 2.4. (cf [S-K-K, Chapter III, Theorem 2.3.6]) Let M be a left coherent E _X -module satisfying (2.1) and assume that for an open subset W ⊂ V ∩ T _M ^∗ X:

(i) L _T

M

X (V )(p) is non degenerate.

Then the sheaf Ext ^s _E

^{(M,V )} (M, C _M ) is conically flabby on W . 3. Proof of the results.

We keep the same notations as in §2.

3.1. We first need some preparation to the proof of Theorem 2.3. The first part of our proof will follow the argument used by Kashiwara and Schapira in [K-S 1]

in order to prove Theorem 2.1. Their method consists in “reducing” the germ of

RHom E

(M, C M ) at p ∈ W to a germ of the complex µ N (O Y )[1 − d] of microfunc- tions along a real analytic hypersurface N of a complex analytic manifold Y (for a shift 1 − d which is calculated).

We need to refine here their procedure since, due to the definition of isotropic flabbiness, one has to deal with the restriction of RHom E

(M, C M ) to an open neighborhood Ω of p and not only with its germ at p.

3.2. It follows from the proof of [K-S 1, Theorem 3.1] that one has:

Proposition 3.1. For each p ∈ W there exist an open neighborhood Ω ⊂ T ^∗ X of p, a contact transformation χ : T ^∗ X → T ^∗ X defined on Ω and a smooth map f : X → Y of complex analytic manifolds such that:

 



 

χ(V ) = X × _Y T ^∗ Y,

χ(T _M ^∗ X) = T _S ^∗ X, S being the real analytic boundary of a strictly pseudo-convex set, f _π ((X × _Y T ^∗ Y ) ∩ T _S ^∗ X) = T _N ^∗ Y, N being a real analytic hypersurface of Y.

Moreover, a quantization of χ gives an isomorphism:

χ ∗ (RHom E

(M, C M )) ∼ = µhom(C S , f ⁻¹ O _Y )[1] in D ^b (χ(Ω)).

Let p ∈ χ(W ). If ψ(x) = 0 is a local equation for S at x ◦ with dψ(x ◦ ) = p, we set S _p = {x ∈ X; ψ(x) ≥ 0}. We similarly define N _q for q = f _π (p).

Let us use the following notations:

λ ₀ (p) = T _p π ⁻¹ π(p), λ _S (p) = T _p T _S ^∗ X,

ρ(p) = (T _p (χ(V )) ^⊥ ,

λ e _S (p) = (λ _S (p) ∩ ρ(p) ^⊥ ) + ρ(p), where (·) ^⊥ denotes the symplectic orthogonal. Moreover we set

d = 1/2[τ (λ ₀ (p), e λ _S (p), λ _S (p)) + 2l − dim ^R (λ _S (p) ∩ ρ(p))],

where l = codim ^C V = dim ^C X − dim ^C Y and τ (·, ·, ·) denotes the Maslov index of three Lagrangian planes in T _p T ^∗ X.

Lemma 3.2. (cf [K-S 1, Proposition 7.4.2]) There exists a fundamental system of open neighborhoods U ⊂ X of x ◦ such that the natural morphism:

(3.1) Rf ! C _S

_∩U −→ C _N

[l − d], is an isomorphism in D ^b (f (U )).

3.3. Given p ∈ W let us choose an open neighborhood Ω ⊂ T ^∗ X of p and an open neighborhood U ⊂ X of f π (χ(p)) such that:

(i) Ω satisfies the requirements of Proposition 3.1, (ii) U satisfies the requirements of Lemma 3.2, (iii) χ(Ω) ⊂ π ⁻¹ (U ),

(iv) χ(Ω) ∩ SS(S _p ) ^a = ∅,

(v) C _S

_∩U ∼ = C S in D ^b (X; χ(Ω)),

(vi) C N

∼ = C N in D ^b (Y ; f π (χ(Ω))),

where (·) ^a denotes the antipodal map.

Proposition 3.3. The natural morphism in D ^b (χ(Ω)):

(3.2) f _π ^{− 1} µhom(C _N , O _Y )[d] → ^t f ^{0− 1} µhom(C _S , f ^{− 1} O _Y ⊗ ω _X/Y )[l], induced by (3.1), is an isomorphism.

Let us first explain how (3.2) is induced by (3.1). The natural morphism R ^t f _! ⁰ f _π ^{− 1} µhom(Rf _! C _S

_∩U , O _Y ) → µhom(C _S

_∩U , f ^{− 1} O _Y ⊗ ω _X/Y ) induces by adjunction

(3.3) f _π ⁻¹ µhom(Rf ! C _S

_∩U , O Y ) → ^t f ⁰⁻¹ µhom(C S

∩U , f ⁻¹ O _Y ⊗ ω _X/Y ).

From (3.1) and (3.3) we get the morphisms

f _π ^{− 1} µhom(C _N

, O _Y )[d] → f _π ^{− 1} µhom(Rf _! C _S

∩ U , O _Y )[l]

→ ^t f ⁰⁻¹ µhom(C _S

∩ U , f ⁻¹ O _Y ⊗ ω _X/Y )[l]

which give (3.2) owing to (v) and (vi) above.

Proof of Proposition 3.3. For each p ⁰ ∈ χ(Ω) and j ∈ N we have to prove the isomorphism

H ^j (f _π ^{− 1} µhom(C _N , O _Y )[d]) _p

∼ = H ^j (µhom(C _S , f ^{− 1} O _Y ⊗ ω _X/Y )[l]) _p

. If p ⁰ ∈ T / _S ^∗ X ∩ (X × _Y T ^∗ Y ) both cohomology groups vanish.

If p ⁰ ∈ T _S ^∗ X ∩ (X × _Y T ^∗ Y ) ∩ χ(Ω) one has for q ⁰ = f _π (p ⁰ ):

H ^j (µhom(C N , O Y )[d]) q

∼ = H ^j (µhom(C N

, O Y )[d]) q

∼ = Hom _D

_{(Y ;q}

₎ (Rf _! C _S

∩ U , O _Y )[l + j]

∼ = Hom D

(X;p

) (C S

∩U , f ^! O _Y )[l + j]

∼ = H ^j (µhom(C _S

∩ U , f ^! O _Y )[l]) _p

∼ = H ^j (µhom(C S , f ⁻¹ O _Y ⊗ ω _X/Y )[l]) p

. q.e.d.

3.4. Let ω = f _π (χ(W )). For each p ∈ W let q = f _π (χ(p)).

Proof of Theorem 2.3. One has (cf [K-S 1]):

(3.4) s ⁻ (N, q) = s ⁻ (M, V, p) − d.

The hypothesis (i) of Theorem 2.3 and the equality (3.4) imply that s ⁻ (N, q) is constant, say s ⁻ (N ), for q ∈ ω. Combining Propositions 3.1 and 3.3 we get:

(3.5) χ ∗ (Ext ^s _E

^{(M,V )} (M, C _M )| _Ω∩V ) ∼ = f _π ^{− 1} (H ^s

^{(N )} µ _N (O _Y ))| _{χ(Ω∩V )} .

In order to prove Theorem 2.3 we are then reduced to show the flabbiness of H ^s

^{(N )} µ _N (O _Y ) on ω.

The hypothesis (ii) of Theorem 2.3 is a symplectic invariant and, after application of χ, it reads:

(ii) _χ If γ is a germ of complex curve in f _π ⁻¹ (T _N ^∗ Y ) at ˜ p ∈ χ(W ), then γ ⊂

f _π ^{− 1} f _π (˜ p)).

It follows that there is no germ of complex curve in ω and hence the conical flab- biness of H ^s

^{(N )} µ _N (O _Y ) is obtained by applying Theorem 2.2. q.e.d.

Proof of Corollary 2.4. The totally isotropic space of L _T

M

X (V, p) has real dimension equal to dim ^R (ρ(p) ∩ λ _M (p)) + dim ^C (λ _N (q) ∩ iλ _N (q)) (cf [K-S 1, Lemma 3.4]).

Since L _T

_X (V, p) is non degenerate, one has λ N (q) ∩ iλ N (q) = {0}, (3.6)

ρ(p) ∩ λ _M (p) = {0}.

(3.7)

From (3.6) one deduces that the Levi form of N ⊂ Y at q is non degenerate. It follows that there is no germ of complex curve on ω, that s ⁻ (N, q) is constant (say s ⁻ (N )) on ω and that, due to (3.4), s ⁻ (M, V, p) is constant (say s ⁻ (M, V )) on W . The isotropical flabbiness of Ext ^s _E

^{(M,V )}

X

(M, C _M ) is then deduced as in the proof of Theorem 2.3.

One concludes noticing that (3.7) implies the injectivity of f _π | _{χ(W )} and hence isotropically flabby means in fact conically flabby. q.e.d.

Acknowledgments

The author likes to express here his gratitude to Pierre Schapira who proposed him this problem.

References

[D’A-Z] A. D’Agnolo, G. Zampieri, Micro-hypoellipticity at the boundary for systems with simple characteristics, to appear.

[K-S 1] M. Kashiwara, P. Schapira, A vanishing theorem for a class of systems with simple characteristics, Invent. Math. 82 (1985), 579–592.

[K-S 2] M. Kashiwara, P. Schapira, Sheaves on manifolds, Springer-Verlag 292 (1990).

[S-K-K] M. Sato, T. Kawai, M. Kashiwara, Hyperfunctions and pseudo-differential equations, Lecture Notes in Math., Springer-Verlag 287 (1973), 265–529.

[S-T] P. Schapira, J.-M. Tr´ epreau, Microlocal pseudoconvexity and “edge of the wedge” theorem, Duke Math. J. 61, 1 (1990), 105–118.

Dept. de Math´ ematiques, Univ. Paris Nord, F-93430 Villetaneuse, France

Dip. di Matematica, Univ. di Padova, via Belzoni 7, 35131 Padova, Italy