If you use pushd and popd to maintain your directory stack, then ~N (where ~0 is the top of the stack) will expand to the appropriate directory in the stack. For example:

cygnus:~$ pushd usr
~/usr ~
cygnus:~/usr$ pushd /etc
/etc ~/usr ~
cygnus:/etc$ pushd /usr/lib
/usr/lib /etc ~/usr ~
cygnus:/usr/lib$ cd ~2
cygnus:~/usr$

Furthermore, you can make cd use pushd and popd functionality implicitly:

~> DIRSTACKSIZE=8
~> setopt autopushd pushdminus pushdsilent pushdtohome
~> alias dh='dirs -v'
~> cd /tmp
/tmp> cd /usr
/usr> cd bin
/usr/bin> cd ../pub
/usr/pub> dh
0       /usr/pub
1       /usr/bin
2       /usr
3       /tmp
4       ~
/usr/pub> cd -3
/tmp> dh
0       /tmp
1       /usr/pub
2       /usr/bin
3       /usr
4       ~
/tmp> ls =2/df
/usr/bin/df
/tmp> cd -4
~>

(Note: The text above is taken essentially verbatim from http://www.b2pi.com/zsh/Intro/intro_6.html.)