-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
switch instagram to scraping due to new permission policy :( #603
Comments
the new set of oauth scopes aka permissions is on https://www.instagram.com/developer/authorization/ :
|
i started on the review process, but stopped when i saw it requires a screencast. ugh. i'll do that eventually. here's the rest of what i have written up so far: https://www.instagram.com/developer/clients/580be8883446443d8216ebdf0462f3b8/review/
|
made the screencast: https://youtu.be/eGMNItivBdY |
...and submitted to instagram for approval. fingers crossed! https://www.instagram.com/developer/clients/580be8883446443d8216ebdf0462f3b8/edit/#permissions |
they denied us. :(
well. that's a problem. they also denied commenting and liking, which is a bit less surprising, and due to a technicality: we didn't describe our use case well. meh.
|
next step: apply for oauth-dropins and see if i can get it approved. not holding my breath, but i'd like to find at least one app i can get approved, just to see how the process works all the way through. |
done. fingers crossed! |
oauth-dropins got rejected too. :/
|
i'm running out of ideas. i may have to start scraping. :/ |
i took a brief look at what it would take to switch to scraping. the good news is, it's doable. instagram profile and photo pages happily serve without being logged in, and the data is easily available in JSON that we already have code to extract and parse. the bad news is, profile pages only include counts of comments and likes for each photo, not the actual data about them. we'd have to fetch the individual photo pages to get the data. annoying, but not too bad. we already do this for twitter and google+. the more worrisome part is that comments and likes are paged, so fetching the photo only gets us the first 10 of each. hrmph. if it's the most recent 10, we'll be able to backfeed at least 10 comments and likes per photo per poll period (20m right now)...but i expect some people peak above that sometimes. hrmph. |
Iiiiiii'd give some serious thought to whether it's worth the effort. Because of aaronpk/OwnYourGram#16 PESOS doesn't work for many people any more anyway, and it's very likely OYG will be cut off altogether (even if he rebrands it). I'm curious what the situation with IFTTT/Zapier/etc. integration is... whether their channels will be shut off too. |
hrmph, true. point taken. i still posse to IG manually, so i may still do it if only for myself. we'll see. |
well, if you do do it, I'll certainly continue to use it :P |
ok, this is implemented, naively. it has to do an HTTP fetch per picture, in serial, to get comments and likes. ideally, those would be parallelized, and also cache and check the counts like G+ now (and i think twitter) so it only does the fetches when there are new comments or likes. |
current plan for deleting legacy API accounts is that we'll indieauth into their first web site in
|
ran this in for i in Instagram.query(Instagram.features == 'publish'):
i.features.remove('publish')
i.put() |
flipped the switch! all instagram accounts are now on scraping and using indieauth for login/delete. fingers crossed! |
looking good so far. tentatively closing. woo! |
@snarfed May I ask how you passed the "Still in Development: Your app is still in development. Please resubmit only when your app is ready to go live and no longer in development." part? I am trying to submit my app now and I get this error back. And I can't for the life of my understand what it means. Sorry to hijack your thread but you seem to be the only one that has faced this issue. |
@Johnathangalliano sounds like your app is still in sandbox mode? https://www.instagram.com/developer/sandbox/ i didn't actually get approved, so i don't have more specific advice, sorry. i switched to scraping their html instead. :/ |
i was also doing scrapping, and it was all going very well, but suddenly my all accounts started getting limit exceeded. even when i sign in. do u have a fix for this.. and did you monitor the rate limit on different end points? |
@rummykhan if you're getting 429s, then yeah, instagram rate limits HTTP requests by IP address or subnet. i hit that at one point too. lots of details in #665 and https://groups.google.com/d/msg/google-appengine/rpendSIxJMo/_u4G6uXiBQAJ . |
thanks @snarfed and yea i was getting response code 429, today i did some testing and what i found is here.. Instagram Scrapping WORKAROUND Tests:
Key Observation
Solution
|
If I may ask, about how many requests were you guys making per hour before hitting that limit? |
sure! details on request volume above and in https://groups.google.com/d/msg/google-appengine/rpendSIxJMo/_u4G6uXiBQAJ :
|
Thanks @snarfed . I'm making ~.7qps non-stop for the whole day and haven't gotten any 429s (its only been 3-4 days). Do you think I should be concerned about getting them in the future at that rate? |
based on this data, maybe yes, within weeks. good luck though! |
Do I have to go through app approval process even for fetching photos from my Instagram account? |
@harshdamaniahd to write your own Instagram app? yes, and good luck even then. |
Instagram is locking down their API and requiring all apps to go through a review process similar to facebook's. details in snarfed/granary#65.
they're mainly locking down
/users/self/feed
and/media/popular
and sending photos outside of instagram, neither of which bridgy does, so i think we'll be ok, but no guarantees.TODO for switching to scraping:
get_comment
,get_like
, etc.)publish
feature from all existing accts so that when they deletelisten
, their acct disappears correctly.The text was updated successfully, but these errors were encountered: