Scraping dynamic data

I am scraping profiles on ask.fm for a research question. The problem is that only the top most recent questions are viewable and I have to click "view more" to see the next 15.

The source code for clicking view more looks like this:

<input class="submit-button-more submit-button-more-active" name="commit" onclick="return Forms.More.allowSubmit(this)" type="submit" value="View more" />

What is an easy way of calling this 4 times before scraping it. I want the most recent 60 posts on the site. Python is preferable.

You could probably use selenium to browse to the website and click on the button/link a few times. You can get that here:

    https://pypi.python.org/pypi/selenium

Or you might be able to do it with mechanize:

    http://wwwsearch.sourceforge.net/mechanize/

I have also heard good things about twill, but never used it myself:

    http://twill.idyll.org/



Source: http://stackoverflow.com/questions/19437782/scraping-dynamic-data

Web Scraping data from different sites

I am looking for a few ideas on how can I solve a design problem I'm going to be faced with building a web scraper to scrape multiple sites. Writing the scraper(s) is not the problem, matching the data from different sites (which may have small differences) is.

For the sake of being generic assume that I am scraping something like this from two or more different sites:

    public class Data {
        public int id;
        public String firstname;
        public String surname;
        ....
    }

If i scrape this from two different sites, I will encounter the situation where I could have the following:

Site A: id=100, firstname=William, surname=Doe

Site B: id=1974, firstname=Bill, surname=Doe

Essentially, I would like to consider these two sets of data the same (they are the same person but with their name slightly different on each site). I am looking for possible design solutions that can handle this.

The only idea I've come up with is scraping the data from a third location and using it as a reference list. Then when I scrape site A or B I can, over time, build up a list of failures and store them in a list for each scraper so that it can know (if i find id=100 then i know that the firstname will be William etc). I can't help but feel this is a rubbish idea!

If you need any more info, or if you think my description is a bit naff, let me know!

Thanks,

DMcB


Source: http://stackoverflow.com/questions/23970057/web-scraping-data-from-different-sites

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Collecting Data With Web Scrapers

There is a large amount of data available only through websites. However, as many people have found out, trying to copy data into a usable database or spreadsheet directly out of a website can be a tiring process. Data entry from internet sources can quickly become cost prohibitive as the required hours add up. Clearly, an automated method for collating information from HTML-based sites can offer huge management cost savings.

Web scrapers are programs that are able to aggregate information from the internet. They are capable of navigating the web, assessing the contents of a site, and then pulling data points and placing them into a structured, working database or spreadsheet. Many companies and services will use programs to web scrape, such as comparing prices, performing online research, or tracking changes to online content.

Let's take a look at how web scrapers can aid data collection and management for a variety of purposes.

Improving On Manual Entry Methods

Using a computer's copy and paste function or simply typing text from a site is extremely inefficient and costly. Web scrapers are able to navigate through a series of websites, make decisions on what is important data, and then copy the info into a structured database, spreadsheet, or other program. Software packages include the ability to record macros by having a user perform a routine once and then have the computer remember and automate those actions. Every user can effectively act as their own programmer to expand the capabilities to process websites. These applications can also interface with databases in order to automatically manage information as it is pulled from a website.

Aggregating Information

There are a number of instances where material stored in websites can be manipulated and stored. For example, a clothing company that is looking to bring their line of apparel to retailers can go online for the contact information of retailers in their area and then present that information to sales personnel to generate leads. Many businesses can perform market research on prices and product availability by analyzing online catalogues.

Data Management

Managing figures and numbers is best done through spreadsheets and databases; however, information on a website formatted with HTML is not readily accessible for such purposes. While websites are excellent for displaying facts and figures, they fall short when they need to be analyzed, sorted, or otherwise manipulated. Ultimately, web scrapers are able to take the output that is intended for display to a person and change it to numbers that can be used by a computer. Furthermore, by automating this process with software applications and macros, entry costs are severely reduced.

This type of data management is also effective at merging different information sources. If a company were to purchase research or statistical information, it could be scraped in order to format the information into a database. This is also highly effective at taking a legacy system's contents and incorporating them into today's systems.

Overall, a web scraper is a cost effective user tool for data manipulation and management.

Source:http://ezinearticles.com/?Collecting-Data-With-Web-Scrapers&id=4223877

How Web Data Extraction Services Will Save Your Time and Money by Automatic Data Collection

Data scrape is the process of extracting data from web by using software program from proven website only. Extracted data any one can use for any purposes as per the desires in various industries as the web having every important data of the world. We provide best of the web data extracting software. We have the expertise and one of kind knowledge in web data extraction, image scrapping, screen scrapping, email extract services, data mining, web grabbing.

Who can use Data Scraping Services?

Data scraping and extraction services can be used by any organization, company, or any firm who would like to have a data from particular industry, data of targeted customer, particular company, or anything which is available on net like data of email id, website name, search term or anything which is available on web. Most of time a marketing company like to use data scraping and data extraction services to do marketing for a particular product in certain industry and to reach the targeted customer for example if X company like to contact a restaurant of California city, so our software can extract the data of restaurant of California city and a marketing company can use this data to market their restaurant kind of product. MLM and Network marketing company also use data extraction and data scrapping services to to find a new customer by extracting data of certain prospective customer and can contact customer by telephone, sending a postcard, email marketing, and this way they build their huge network and build large group for their own product and company.

We helped many companies to find particular data as per their need for example.

Web Data Extraction

Web pages are built using text-based mark-up languages (HTML and XHTML), and frequently contain a wealth of useful data in text form. However, most web pages are designed for human end-users and not for ease of automated use. Because of this, tool kits that scrape web content were created. A web scraper is an API to extract data from a web site. We help you to create a kind of API which helps you to scrape data as per your need. We provide quality and affordable web Data Extraction application

Data Collection

Normally, data transfer between programs is accomplished using info structures suited for automated processing by computers, not people. Such interchange formats and protocols are typically rigidly structured, well-documented, easily parsed, and keep ambiguity to a minimum. Very often, these transmissions are not human-readable at all. That's why the key element that distinguishes data scraping from regular parsing is that the output being scraped was intended for display to an end-user.

Email Extractor

A tool which helps you to extract the email ids from any reliable sources automatically that is called a email extractor. It basically services the function of collecting business contacts from various web pages, HTML files, text files or any other format without duplicates email ids.

Screen scrapping

Screen scraping referred to the practice of reading text information from a computer display terminal's screen and collecting visual data from a source, instead of parsing data as in web scraping.

Data Mining Services

Data Mining Services is the process of extracting patterns from information. Datamining is becoming an increasingly important tool to transform the data into information. Any format including MS excels, CSV, HTML and many such formats according to your requirements.

Web spider

A Web spider is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Many sites, in particular search engines, use spidering as a means of providing up-to-date data.

Web Grabber

Web grabber is just a other name of the data scraping or data extraction.

Web Bot

Web Bot is software program that is claimed to be able to predict future events by tracking keywords entered on the Internet. Web bot software is the best program to pull out articles, blog, relevant website content and many such website related data We have worked with many clients for data extracting, data scrapping and data mining they are really happy with our services we provide very quality services and make your work data work very easy and automatic.

Source:http://ezinearticles.com/?How-Web-Data-Extraction-Services-Will-Save-Your-Time-and-Money-by-Automatic-Data-Collection&id=5159023

Restaurant Internet Marketing - It's Not Your Father's Yellow Pages

As a restaurant owner or manager are you taking advantage of all the opportunities offered up by the Internet or are you still counting on word of mouth and a traditional print marketing program to get diners filling your place? Are you even aware of what the Internet marketing can do for you? If you're not you're giving up business to your competition.

Take a moment to Google your type of restaurant (Italian, seafood, deli etc) and the name of your town and see what comes up on the first page. Do you see your name? How many competitors do you see? Now ask yourself "How many potential diners are going to come to my restaurant based on these results"? The answer of course is nada. Can you afford to give that business to your competition?

Location, location, location

Many local small and medium sized businesses have discovered the value of having an internet presence but it wasn't always that way. Most local businesses thought the internet best served big companies. On top of that, most small local businesses simply didn't have a clue how to participate effectively and they were too busy running their businesses to learn how.

The fact of the matter is that 72% of all searches are related to a search for local content. Nearly a year ago the monster search engine Google changed their algorithms so that local results show up on a search inquiry whenever appropriate. That means your shop can be just as competitive as a national restaurant chain when it comes to being found in search.

The difference between search and traditional advertising

So why is search such a big deal? The reason it is so much more effective than traditional marketing is that it responds to a specific need at a time when the potential diner is interested in the information. Unlike a weekly flier that's dropped in the home mailbox every Thursday, search doesn't clutter up a prospective diner's life but rater provides relevant information when the diner wants it not when the USPS delivers it.

When that potential customer is thinking about where to go for lunch he or she is going to search "pizza san pedro, ca" to find what's available in the area. In other words search delivers the information when the searcher is most receptive to it.

But wait there's more

Search by itself is reason to have a presence on the internet but the benefits certainly don't end there.

How would you like to be able to:

- Notify all of your customers what your daily specials are every day automatically.
- Invite your customers to place their order in advance so they don't have to wait for their food and do it without spending any time on the phone.
- Accept and confirm reservations electronically.
- Run promotional campaigns without spending a dime on advertising.
- Build solid customer relationships that will strengthen your word of mouth referrals.

And that's just the beginning. A professional internet presence will cost you significantly less than you are currently paying today for marketing. What's more, professionally managed internet presence keeps on giving even when you stop paying. Can you say that about your Yellow Page listing or print ads?

Online Reputation Management

There can be a potential dark side to this whole internet thing. You live or die on your reputation and it is important that you manage that reputation online. Never has the consumer had such an easy way to post anonymous comments for all the world to see. If you have a customer who has a bad experience don't be surprised if he or she posts it to a blog or a forum or gives you a bad review in the business directories.

Speaking of business directories, you already are on the internet even if you didn't know it. These directories scrape information out of Yellow Pages and other sources and add it to a business profile. You have a right to claim those listings and they are actually a great way to promote the business. But registered visitors also have the right to post reviews and if the only review your store has is from that disgruntled customer...your rep is shot.

There are just too many reasons that you have to get involved with the internet marketing. To not do so is to pass up an incredible opportunity for new business and to cement loyalty with regular customers. It's effective, inexpensive and essential if you want to remain competitive.

WebFrootz is an internet marketing agency, Baltimore website design, and Baltimore web development company MD specializing in comprehensive marketing solutions for small and medium business including branding, Website design and development, graphic design, copywriting, and SEO.

Source: http://ezinearticles.com/?Restaurant-Internet-Marketing---Its-Not-Your-Fathers-Yellow-Pages&id=6498349

Infinity Logo Design Launches Content Writing Services

Infinity Logo Design recently added Content Writing Service to its pool of services, available at a staggering discount of 70%. Following the huge success of its Animated Video Service, Infinity Logo Design has now officially commenced its Content Writing Service as well. With a clientele of nearly a million, Infinity Logo Design is one of the fastest growing design companies of the world. And its recent diversification advocates the fact that the company looks fanatical in achieving its goals and objects with one of them being "to become the number 1 design & development company of the world". Providing a complete range of Content Writing services which includes Web Content Writing, SEO Content Writing, Creative writing, Press Release writing, Article Writing and Brand/Product Name Suggestions, Infinity Logo Design is dedicated to providing a superior experience to the readers.

According to the company's Creative Manager, "We have been offering Content Writing service as a secondary service for quite some time, but following its augmented demand and some great feedback from our clients, we decided to launch it officially. We consider Content Writing as an art that truly fascinates and inspires the readers, and makes them believe in the message that the writer is trying to covey. Our copy writers are handpicked, and have a proven track record. We provide our customers with nothing but the very best!"

Infinity Logo Design believes that Content Writing in an effective tool for customer retention and generating a profound user experience. Infinity Logo Design is offering this service at quite a substantial price, and its exceptional discount of 70% is a bonus for all those who are in pursuit of quality content at an affordable price. Judging from the company's recent endeavors, there's no denying that we'll be seeing some great things from them in the years to come.

About Infinity Logo Design:

Infinity Logo Design is one of the world's leading design companies. The company offers a wide range of services including custom logo design, website design, brochure design, banner design, stationery design and promotional items. ILD provides high-quality designs to small- and medium-sized businesses at the most competitive prices. In addition, the company engages the services of skilled and talented designers who can create your design exactly according to your specifications. The Company also offers 100% satisfaction and 100% money back guarantee, making it one of the most sought-after design companies in the world.

Source: http://www.digitaljournal.com/pr/1985193