Stability of Hodge bundles and a numerical characterization of Shimura varieties

Let U be a connected quasi-projective manifold and f : A --> U a family of abelian varieties of dimension g. Suppose that the induced map U --> A_g is generically finite and there is a compactification Y with complement S = Y \ U a normal crossing divisor such that Ω_Y¹ (log S) is nef and ω_Y(S) is ample with respect to U.
We characterize whether U is a Shimura variety by numerical data attached to the variation of Hodge structures, rather than by properties of the map U --> A_g or by the existence of CM points. More precisely, we show that f : A --> U is a Kuga fibre space, if and only if two conditions hold. First, each irreducible local subsystem V of R¹f_*C_A either unitary or satisfies the Arakelov equality. Second, for each factor M in the universal cover of U whose tangent bundle behaves like the one of a complex ball, an iterated Kodaira-Spencer map associated with V has minimal possible length in the direction of M. If in addition f : A --> U is rigid, it will be a connected Shimura subvariety of A_g of Hodge type.