Create instagram_crawler.py by yogeshwaran01 · Pull Request #2508 · TheAlgorithms/Python

yogeshwaran01 · 2020-09-29T09:46:43Z

An algorithm crawls the Instagram page of the user and Scarpe the data.

Describe your change:

Add an algorithm?
Fix a bug or typo in an existing algorithm?
Documentation change?

Checklist:

It crawls the Instagram page of the user and Scarpe the data.

cclauss

Cool submission!!

cclauss · 2020-09-29T10:02:13Z

+from bs4 import BeautifulSoup
+import json
+
+headers = \


No backslashes please in Python code. See PEP8. The problem with backslashes is that whitespace to the right of the backslash breaks the script on a change that is invisible to the reader.

You can run your code through psf/black to autofix that issue as discussed in CONTRIBUTING.md

cclauss · 2020-09-29T10:04:34Z

+
+    def __init__(self, username):
+        self.username = username
+        self.url = 'https://www.instagram.com/{}/'.format(username)


Suggested change

self.url = 'https://www.instagram.com/{}/'.format(username)

self.url = f'https://www.instagram.com/{username}/'

As discussed in CONTRIBUTING.md, please use f-strings where they make sense.

Do we really want to query the backend (Instagram) for every one of the fields below? Most of these values are very static and are unlikely to change minute-to-minute. Perhaps it would be better to have a self.user_data dict that contained the results of self.get_json() and the other methods could use that data.

cclauss · 2020-09-29T10:06:39Z

+            info = html_1(soup)
+            return info
+        except:
+            info = html_2(soup)
+            return info


Suggested change

info = html_1(soup)

return info

except:

info = html_2(soup)

return info

return html_1(soup)

except: # <-- This repo does not accept bare excepts

return html_2(soup)

Bare excepts are discussed in PEP8 and https://realpython.com/the-most-diabolical-python-antipattern/

cclauss · 2020-09-29T10:22:06Z

+            info = html_2(soup)
+            return info
+
+    def get_followers(self):


Let's make this (and similar methods below) into a @property so that we can use the syntax instagram_user.number_of_followers (without the ()).

Suggested change

def get_followers(self):

@property

def number_of_followers(self) -> int:

Also, let's streamline to a one-line implementation like:

return self.get_json()['edge_followed_by']['count'] # or... return self.data['edge_followed_by']['count']

In general, do not create a variable that you get rid of on the very next line unless the variable name really helps the reader understand something nonobvious.

cclauss · 2020-09-29T10:22:49Z

+        followers = info['edge_followed_by']['count']
+        return followers
+
+    def get_followings(self):


Suggested change

def get_followings(self):

def get_number_of_followings(self) -> int:

cclauss · 2020-09-29T10:23:27Z

+        following = info['edge_follow']['count']
+        return following
+
+    def get_posts(self):


Suggested change

def get_posts(self):

def get_number_of_posts(self) -> int:

cclauss · 2020-09-29T10:23:50Z

+        posts = info['edge_owner_to_timeline_media']['count']
+        return posts
+
+    def get_biography(self):


Suggested change

def get_biography(self):

def get_biography(self) -> str:

cclauss · 2020-09-29T10:28:08Z

+user = Instagram('github')
+print(user.is_verified())
+print(user.get_biography())


Code that is at global scope will be run by our Travis CI / pytest process as discussed in CONTRIBUTING.md.

Suggested change

user = Instagram('github')

print(user.is_verified())

print(user.get_biography())

if __name__ == '__main__':

user = Instagram('github')

print(f"{user.is_verified() = })

print(f"{user.get_biography() = })

Co-authored-by: Christian Clauss <cclauss@me.com>

yogeshwaran01

According to @cclauss some changes are done

cclauss · 2020-09-29T11:54:11Z

-    {
-        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
+headers = {
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"


Suggested change

"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) "

"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

TravisBuddy · 2020-09-29T12:13:32Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 303e78f0-024d-11eb-aba2-872ffb2742c8

TravisBuddy · 2020-09-29T12:13:32Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 303bb9d0-024d-11eb-aba2-872ffb2742c8

TravisBuddy · 2020-09-29T12:13:33Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 303f1530-024d-11eb-aba2-872ffb2742c8

TravisBuddy · 2020-09-29T12:13:33Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 30424980-024d-11eb-aba2-872ffb2742c8

cclauss · 2020-09-29T12:22:20Z

Closing in favor of #2509

Create instagram_crawler.py

6428bf5

It crawls the Instagram page of the user and Scarpe the data.

cclauss requested changes Sep 29, 2020

View reviewed changes

yogeshwaran01 and others added 3 commits September 29, 2020 16:38

Update instagram_crawler.py

7e359cf

changed class name to InstagramUser

8bc9104

Co-authored-by: Christian Clauss <cclauss@me.com>

some chamges had done

bafda76

yogeshwaran01 commented Sep 29, 2020

View reviewed changes

cclauss reviewed Sep 29, 2020

View reviewed changes

cclauss closed this Sep 29, 2020

	self.url = 'https://www.instagram.com/{}/'.format(username)
	self.url = f'https://www.instagram.com/{username}/'

	def get_followers(self):
	@property
	def number_of_followers(self) -> int:

	def get_followings(self):
	def get_number_of_followings(self) -> int:

	"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
	"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) "
	"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

Uh oh!

Conversation

yogeshwaran01 commented Sep 29, 2020

Describe your change:

Checklist:

Uh oh!

cclauss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cclauss Sep 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yogeshwaran01 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 303e78f0-024d-11eb-aba2-872ffb2742c8

Uh oh!

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 303bb9d0-024d-11eb-aba2-872ffb2742c8

Uh oh!

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 303f1530-024d-11eb-aba2-872ffb2742c8

Uh oh!

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 30424980-024d-11eb-aba2-872ffb2742c8

Uh oh!

cclauss commented Sep 29, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cclauss Sep 29, 2020 •

edited

Loading