W3C XML Schemas
W3C XML Schemas were developed in a long and intensive process as a reaction to perceived limitations in DTDs: for better datatypes, integration with namespaces, element syntax, support for more powerful grammars, and the ability to model various important languages that are difficult for DTDs, such as RDF.
The result has been applauded in some circles, particularly those who want to use a schema to automatically generate interfaces to DBMS and middleware. However it has also been criticized for its size, complexity, and on many individual issues: it does not support attribute values subclassing the element, and it does not support localized datatypes, for example.
Schematron and RELAX NG were developed by members of the W3C XML Schemas Working Group specifically as an alternative and complement to W3C XML Schemas. In many cases, where W3C XML Schemas has adopted one decision, Schematron has adopted the opposite. For example, W3C XML Schemas limits itself to information present in a streaming implementation, while Schematron allows random access to the whole document (and external documents!) during validation.
Use W3C XML Schemas for their datatyping ability.
For the specification of W3C XML Schemas see the XML Schemas: Structures, Datatypes and Primer at W3C.
ISO RELAX NG
RELAX NG is a small, lightweight language for defining document grammars. It is much simpler than W3C XML Schemas, and provides more powerful modeling. It is especially good for publishing requirements.
For material on RELAX NG, see the OASIS website.
You can embed Schematron patterns in W3C XML schemas and RELAX NG schemas. See the article by Eddie Robertsson Combining Schematron with other XML Schema languages.