What the difference between -Xss and -XX:ThreadStackSize ?

There's basically no difference between -Xss and -XX:ThreadStackSize, except
that the former dates before HotSpot became the default JVM in Sun's JDK,
where as the latter is an internal VM flag of HotSpot. In effect, the former
is support in a lot of other JVMs, but the latter is specific to HotSpot.

As an historical aside, the command line arguments for Sun JDK 1.0.x and
1.1.x had these:
-ss<number>       set the C stack size of a process
-oss<number>      set the JAVA stack size of a process

These arguments got deprecated by -Xss and -Xoss in later versions of JDK.

Non-internal flags either get processd by the java launcher (such as
-server, -client, -verbosegc, etc.) or get converted into VM internal flags
by HotSpot. See hotspot/src/share/vm/runtime/arguments.cpp for details.

For example, see this version:
the launcher part:
http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/f097ca2434b1/src/share/bin/java.c
lines 1052 - 1058,
and the VM part:
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/81d815b05abb/src/share/vm/runtime/arguments.cpp
lines 2227 - 2240.

As you can see, -Xss gets converted into ThreadStackSize before the VM gets
to use it.
If the user did specify -ss or -Xss in command line arguments, that'll get
picked up directly by the launcher to run the main Java thread. Otherwise,
the launcher asks the VM for a preferred stack size, and HotSpot will return
ThreadStackSize, see here:
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/81d815b05abb/src/share/vm/prims/jni.cpp,
lines 3286 - 3295.

That should explain how they've been implemented to be basically equivalent
in HotSpot.

One thing, though: because the -Xss/-XX:ThreadStackSize argument is handled
by the VM, *after* the primordial thread has been created, so they won't
cover the stack size of the primordial thread. In order to control stack
size correctly, Oracle/Sun's JDK doesn't run Java code in the primordial
thread. See this bug: http://bugs.sun.com/view_bug.do?bug_id=6316197.
If you ever felt the function "ContinueInNewThread" in the launcher was
weird, that's the fix for this issue, so that Java code doesn't get run in
the primordial thread in HotSpot.

resource from@ http://mail.openjdk.java.net/pipermail/hotspot-dev/2011-June/004272.html