Media Companion

Discussion and development board for the program Media Companion.
 
HomeHome  SearchSearch  RegisterRegister  Log inLog in  

 

 v 3.244 - Keeps scraping w/ new movies

Go down 
AuthorMessage
JPD
New User
New User


Posts : 2
Join date : 2010-01-17

v 3.244 - Keeps scraping w/ new movies Empty
PostSubject: v 3.244 - Keeps scraping w/ new movies   v 3.244 - Keeps scraping w/ new movies Icon_minitimeSun Mar 28, 2010 4:11 pm

When adding new movies to my set, MC stalls, as it keeps scraping for some movie information, never finishing.

In my network monitor I can see that it tries to download something (continually) as it keeps a
connection open and uses bandwidth, I don't know what though.

To have MC continue scraping other movies, I reset my network connection (so it times out on the open connection),
and then it continues nicely on retrieving the rest of the movie-info & goes on w/ another movie, where it stall again...

Using fiddler I noticed that it always stalls AFTER requesting the imdb trailers page.

So requests I see are in order:
Result Protocol Host URL
200 HTTP www.google.co.uk /search?hl=en&q=%3CBEVERLY+HILLS+COP+1984%3E+site%3Aimdb.com&meta=
301 HTTP www.imdb.com /title/tt0086960
200 HTTP www.imdb.com /title/tt0086960/
200 HTTP www.imdb.com /title/tt0086960/plotsummary
200 HTTP www.imdb.com /title/tt0086960/fullcredits
200 HTTP www.imdb.com /title/tt0086960/trailers
---
--- HERE IT STALLS, AND I HAVE TO RESET THE CONNECTION. IT THEN CONTINUES WITH:
---
200 HTTP www.google.com /custom?hl=en&client=pub-6811780361519631&cof=FORID%3A1%3BGL%3A1%3BLBGC%3A000000%3BBGC%3A%23000000%3BT%3A%23cccccc%3BLC%3A%2333cc33%3BVLC%3A%2333ff33%3BGALT%3A%2333CC33%3BGFNT%3A%23ffffff%3BGIMP%3A%23ffffff%3B&domains=www.impawards.com&ie=ISO-8859-1&oe=ISO-8859-1&q=beverly+hills+cop+1984&sitesearch=www.impawards.com
200 HTTP www.impawards.com /1984/beverly_hills_cop.html
200 HTTP www.impawards.com /1984/posters/beverly_hills_cop.jpg
200 HTTP api.themoviedb.org /2.0/Movie.imdbLookup?imdb_id=tt0086960&api_key=3f026194412846e530a208cf8a39e9cb
200 HTTP images.themoviedb.org /backdrops/11978/Beverly_Hills_Cop_v2_1920x1080.jpg


This is fully reproducable by my, also after reboot(s), and happens with ever new scrape.
It didn't before Smile

Disabling the 'include IMDB trailer URL in NFO' doesn't make a difference (only order of request differs,
it will then request trailers before fullcredits)

Note that the imdb trailers page does not give me anything back (i'm in the Netherlands, a not supported country, so just a "no trailers for you" message)
Back to top Go down
billyad2000
Admin
billyad2000

Posts : 1326
Join date : 2008-09-20

v 3.244 - Keeps scraping w/ new movies Empty
PostSubject: Re: v 3.244 - Keeps scraping w/ new movies   v 3.244 - Keeps scraping w/ new movies Icon_minitimeSun Mar 28, 2010 8:05 pm

If trailers are unavailable to you then the easiest option is to disable the feature 'Include IMDB Movie Trailer url within the nfo file' in the movie preferences.

This should stop MC from even attempting to obtain a trailer url.

I'll look into why this does not work.

_________________
Do you have a problem with Media Companion, before posting a bug then please check you have the very latest version of Media Companion http://billyad2000.darkbb.com/announcements-f1/
Back to top Go down
http://billyad2000.co.uk
billyad2000
Admin
billyad2000

Posts : 1326
Join date : 2008-09-20

v 3.244 - Keeps scraping w/ new movies Empty
PostSubject: Re: v 3.244 - Keeps scraping w/ new movies   v 3.244 - Keeps scraping w/ new movies Icon_minitimeSun Mar 28, 2010 8:26 pm

OK - I've checked the code here and the only call to get trailer url is made after the full credits, and only if the option to get trailer URL is enabled.

I also installed fiddler to have a look at the results with the option disabled and no trailer url is scraped.

http://www.google.co.uk/search?hl=en&q=%3CIron+Man+2008%3E+site%3Aimdb.com&meta=
http://www.imdb.com/title/tt0371746
http://www.imdb.com/title/tt0371746/
http://www.imdb.com/title/tt0371746/plotsummary
http://www.imdb.com/title/tt0371746/fullcredits
http://www.google.com/custom?hl=en&client=pub-6811780361519631&cof=FORID%3A1%3BGL%3A1%3BLBGC%3A000000%3BBGC%3A%23000000%3BT%3A%23cccccc%3BLC%3A%2333cc33%3BVLC%3A%2333ff33%3BGALT%3A%2333CC33%3BGFNT%3A%23ffffff%3BGIMP%3A%23ffffff%3B&domains=www.impawards.com&ie=ISO-8859-1&oe=ISO-8859-1&q=iron+man+2008&sitesearch=www.impawards.com
http://www.impawards.com/2008/iron_man_ver3.html

As you can see, the ID is gained from Google, then just the IMDB main page, plot summary, and credits are scraped. The others are looking for posters which it finds at the first attempt. The call for trailers only occurs once, and only if the option is selected. The call is always made before the credits.

_________________
Do you have a problem with Media Companion, before posting a bug then please check you have the very latest version of Media Companion http://billyad2000.darkbb.com/announcements-f1/
Back to top Go down
http://billyad2000.co.uk
billyad2000
Admin
billyad2000

Posts : 1326
Join date : 2008-09-20

v 3.244 - Keeps scraping w/ new movies Empty
PostSubject: Re: v 3.244 - Keeps scraping w/ new movies   v 3.244 - Keeps scraping w/ new movies Icon_minitimeSun Mar 28, 2010 8:28 pm

p.s.

My tests were performed with my development build, but I don't think that anything here has changed since version 3.244

I'll be uploading another version to the announcements forum shortly which maybe worth trying.

_________________
Do you have a problem with Media Companion, before posting a bug then please check you have the very latest version of Media Companion http://billyad2000.darkbb.com/announcements-f1/
Back to top Go down
http://billyad2000.co.uk
JPD
New User
New User


Posts : 2
Join date : 2010-01-17

v 3.244 - Keeps scraping w/ new movies Empty
PostSubject: Re: v 3.244 - Keeps scraping w/ new movies   v 3.244 - Keeps scraping w/ new movies Icon_minitimeTue Mar 30, 2010 9:05 am

I tried it w/ the newest 251 build, but it still happens.

Itg might not have to be in the /trailers/ page request (as fiddler tells me that it was a successful request),
but in a request immediatly after that (maybe downloading fanart or the likes?). It only happens w/ movies,
tvshows go fine.

I'll try to reinstall clean and rescrape tonight, will let you know.
Back to top Go down
billyad2000
Admin
billyad2000

Posts : 1326
Join date : 2008-09-20

v 3.244 - Keeps scraping w/ new movies Empty
PostSubject: Re: v 3.244 - Keeps scraping w/ new movies   v 3.244 - Keeps scraping w/ new movies Icon_minitimeTue Mar 30, 2010 11:24 am

The requests you are talking about are related to posters and art.

There are a couple of things you can try,
In the movie preferences / advanced tab, there is a section titled 'nfo poster options', make sure all of these are unchecked, they create a list of all available posters from selected sources to add to the nfo, but it does slow things down.

TMDB can be quite slow, how fast can you browse fanart and posters from there when you are selecting manually.

If it is still to slow then try disabling scraping fanart and posters, just to see where the problem is.

_________________
Do you have a problem with Media Companion, before posting a bug then please check you have the very latest version of Media Companion http://billyad2000.darkbb.com/announcements-f1/
Back to top Go down
http://billyad2000.co.uk
Sponsored content




v 3.244 - Keeps scraping w/ new movies Empty
PostSubject: Re: v 3.244 - Keeps scraping w/ new movies   v 3.244 - Keeps scraping w/ new movies Icon_minitime

Back to top Go down
 
v 3.244 - Keeps scraping w/ new movies
Back to top 
Page 1 of 1

Permissions in this forum:You cannot reply to topics in this forum
Media Companion :: Media Companion :: Bug Discussion-
Jump to: