The delay isn't coming from the program, but from the downloading of actor thumbs from whatever their source.
I created the appearence of a speedup by displaying the page then getting the actor thumb after this was all done, you will see the actor appear a second or so after the page has loaded. It still causes delay when someone trys to change the TV Show before this is finished downloading.
Episodes are faster because the TVDB actor information for episodes doesn't include thumbnails. If IMDB actors are used then the first actor in the list is always the same and therefore cached, which speeds things up.
Ultimately the only solution is to devise a system that hosts all actor thumbs on the local machine, much the same as XBMC, this is in my list of things to do. My main problem with this is creating a method that isn't needlessly hosting the same image multiple times. This is easy with IMDB since the actor ID could be used (nm0185819.jpg for Daniel Craig). It becomes a little bit more tricky with TVDB since they are of the oppinion that each show should have a thumb of the actor as they appear in that particular show.