Scraping taking forever

I installed 3.175 and tried to scrape the 5 new movies I had stored up. It's been working on movie #1 for about 1 hour now and I can see that XBMC is doing tons of reads from my server which contains the movies.

How do I tell what's going on and whether this is a bug or not?

[Quick Update] It finished scraping the first movie after about 1 hour. I'm adding the other 4 movies to see if that takes the same amount of time.

Mark

I just scraped a movie and it took less then 30secs. Perhaps imdb was messing around with site. Has the scraping time improved?

No, I'm currently on movie 3 out of 4. Been running for over 2 hours on the 4 movies and doing a ton of reads from the server before generating the appropriate files. Any idea what it's reading?

Sorry i got no idea about servers, hopefully someone else can help.

Perhaps as a test you could put the movies onto your PC and scrape from there to see if the problem is just with the server?

I also just scraped a new movie in the usual 30-odd sec with 3.175

How long does it take you to open IMDB to a movie through a browser?
Do you have XBMC on XBOX? If so, How long does that take to scrape a movie?

If the scan run your doing is still running I'd suggest killing it and start again.

Hello,
I am having a similar issue. This has happened to me for the past 3 or 4 versions (at least). I don’t have any of the version numbers with me as I am at work, but I am currently using the build from last week and last night it took at least 2-3 hours to scrape 5 movies.

All of my movies are also stored on a server, and I have not yet tried to scrape them locally. I will try that when I get home tonight. I have been using this program for quite a while and never had an issue with scraping from my server until recently. In fact, if I go back to one of my older versions I can scrape them at normal speed, but since that version includes the old IMDB scraper it returns results that are blank. I can then go into the new version and manually change the movie… That works, but as you can imagine is not the way I want to use this program .

I’ll post again this evening after I test out a local scrape instead of from my server.

Thanks!

The only thing I have noticed as of 3.175 is that on rare occasions (perhaps twice out of 20) MC has been very slow to come up with the initial screen......not sure whats happening. I close it down & restart & its as quick as ever. Maybe its something to do with the mini-posters?

ok, I had a little time to do some testing. Here are my results:

Version 3.148 takes less than a minute to scrape a movie from my server
Version 3.175 takes about 2 minutes to scrape a movie from my desktop
Version 3.175 took somewhere between 15-20 minutes to scrape a movie from my server (I was away from my desk, but checking every 5 minutes)

Every version I have tried since 3.148 has this slow scraping speed. While I am willing to concede that something could be wrong getting a movie from the server, I am not fully convinced the problem is all on my end because of the following reasons:

• My speed was slower that that reported by others in this thread when scraping from a local folder
  
• Version 3.148 scraped quickly as every prior version did
  
• In my day to day computing, I notice no performance issues with my network. This includes streaming movies to multiple Xbox's running XBMC concurrently (which I was not doing during my tests above).

If there is anything else I can do to help isolate the problem, please let me know. This software has saved me tons of headaches, and I would love to continue using it.

Thanks!

The time consuming part is the internet access, not the local file access - although you do indicate a difference in the two.....

Its the internet that we wait for & "scrape" the data from. When you load up IMDB & search for that same movie, MC waits for that to fully load & then gets the data it needs from the html code sent to display that page.

Do you have any delays in seeing the movie details on IMDB?

Once the details have been read, MC creates a text file (the nfo file) with that information. This is very quick ofcourse....

Posters are also downloaded, perhaps this is where the delay is.....
I'm not sure but perhaps if you monitor your movie folder, you could see the time it takes to create each file & then judge which one is taking the longest.

Another thought I had is that MC for each session checks Google for the IMDB number for a particular movie for the first 50 movies, after that its asks IMDB which is apparently much slower.....if Google have banned you, you may have some delay because of that. Usually the ban is only temporary......

The only other thing I can think of is to try another computer on your network or another computer accessing the internet different to yours....

PS can you give us the name of a movie you know is causing trouble, including the format of tha movie. e.g. avi or mkv.

Also do you have "Create HD Tags when scraping" turn on in the movie scraper preferences.....if so try it with it off.

Cheers

There are a couple of things that can slow things down.

In your instance I suspect you are using dvd image files, these need to be opened by mediainfo to get the HD tags and it can take a very long time, depending on the system. Try disabling it.

Billy, you are right. All of my movies are ISO's. I'm not sure what you are asking me to disable though, I assume the HD tags as Stormy mentioned?

Stormy, the first thing I checked was my speed trying to open IMDB through a web browser and there were no issues. The entire page loaded rather quickly. The movie I was using last night was "12 Rounds", but it seems to happen with all of my movies.

I do have the HD tags turned on, I will try tonight with it turned off and post my results.

It might also be worth noting that I do not experience these slow speeds when scraping TV shows that are located on the same server as my movies.

Thanks for your help!

Hey guys,
Just wanted to give you an update. Turning off the HD tags solved the issue for me. I can now scrape in under a minute from my server as well.

Thanks!

Good to know we narrowed it down....

Unfortunately the the bit of the program that gets the tags is not written by Billy, however if you type in mediainfo into google, you will see the sourceforge site where it comes from.....

It would be intersting to see if you download the GUI version if it takes that long to analyse the IFO's....


Cheers

This is a problem that is pretty much beyond my control,

ISO files are not media, they are generic disk images, and as such need to be unpacked before they can be read. Over a network this can take a long time, especially since mediainfo seems to need to unpack the whole thing, maybe a future build will fix this, but maybe it won't.

The only 100% work-around I can think of would be to extract the ISO to a folder, DVD files are supported by XBMC and MC and all the file details would be scraped quickly and without issue.

I also have to wonder if other container formats have similar issues, eg IMG, RAR, etc. Maybe someone could do an experiment and compare the speed mediainfo works at.


For everybodys information the scraping speed can be increased dramatically by disabling the options to include thumb urls within the nfo file, especially TMdb. IMDB is by far the fastest source, and by limiting a scrape to that source you should find that scraping speed is dramatically improved. If you are someone that tends to manually select the poster and backdrop after scraping then disable these aswell. The time it takes to scrape a movie from only IMDB is negligible.