Thursday, June 11, 2015

Gulf Stream brings ever warmer water into Arctic Ocean


The image below shows sea surface temperature anomalies in the Arctic as at June 9, 2015.


The image below shows the Arctic from a 180° rotated angle, while also showing the high sea surface temperature anomalies that are so prominent in the North Pacific (note also that the scale of sea surface temperature anomalies differs).



One may wonder why sea surface temperature anomalies below zero are visible in the North Atlantic, given that ocean heat is rising rapidly. As the IPCC said in AR5, more than 60% of the net energy increase in the climate system is stored in the upper ocean (0–700 m) during the relatively well-sampled 40-year period from 1971 to 2010, and about 30% is stored in the ocean below 700 m.

Global heat content at 0-2000 m is rising even faster than at 0-700 m 
The image below further pictures the situation as at June 9, 2015, with large blue and purple areas showing in the North Atlantic where meltwater from the Arctic has spread over time.



Indeed, the accumulation of meltwater over time has created a huge area with relatively cold water that tends to float at the surface, rather than sink, as the meltwater's salt content is very low.

In other words, the ocean underneath the meltwater at the sea surface is much warmer than the temperatures shown on above images. This can be illustrated by the situation near Svalbard. The image below shows the depth of Barents Sea, which is relatively shallow around Svalbard,


As the image shows, cold meltwater with low salt content floats around Svalbard where the water is most shallow. A 'polar front' separates cold and warm water, following the borders of the area where the seafloor is high. Warm, salty water is carried by the Gulf Stream from the (much deeper) Atlantic Ocean into the Arctic Ocean. This warm water collides with cold water east of Svalbard where the seafloor rises steeply, making this warm water come to the surface. 

Warm water from the Atlantic also comes to the surface west of Svalbard, where warm and cold water are similarly separated by the height of the seafloor. 

The image below shows that on June 8, 2015, sea surface temperatures as high as 11.4°C (52.52°F) were recorded to the south-east of Svalbard (a 9.8°C or 17.64°F anomaly), while sea surface temperatures as high as 7.4°C (45.32°F) were recorded to the west of Svalbard (a 3.5°C or 6.3°F anomaly). 

Sea surface temperatures (top) and sea surface temperature anomalies (bottom) on June 8, 2015.
The image below shows the situation on June 21, 2015, when sea surface temperatures as high as 12.5°C (54.5°F) were recorded to the south-east of Svalbard (a 10.2°C or 18.4°F anomaly), while sea surface temperatures as high as 8.5°C (47.3°F) were recorded to the west of Svalbard (a 3.7°C or 6.7°F anomaly) and as high as 7.3°C (45.1°F) further west of Svalbard (a 3.7°C or 6.7°F anomaly).

Sea surface temperatures (top) and sea surface temperature anomalies (bottom) on June 21, 2015.
These spots where warm water comes to the surface give an indication of how high temperatures of the water are below the surface. As more than 90% of the extra heat caused by people's emissions continues to go into oceans, ever warmer water will be carried by the Gulf Stream into the Arctic Ocean, with the danger that this will warm up sediments under the Arctic Ocean seafloor, triggering huge methane eruptions with gigantic warming potential.

The above images picture the situation as at June 8 and June 21, 2015, when summer on the Northern Hemisphere had just started. In other words, temperatures will rise over the next few months. To get an idea of what can be expected, the image below shows the situation as at September 1, 2014, when sea surface temperatures near Svalbard were as high as 17.5°C (or 63.5°F), an anomaly of 11.9°C (or 21.42°F)

Sea surface temperatures (top) and sea surface temperature anomalies (bottom) on September 1, 2014.
On the combination image below, the image on the left shows large areas (red circles) where warmer water is visible through the sea ice, indicating the presence of even warmer water at greater depth in the Arctic Ocean. The image on the right (from an earlier post) roughly shows how ocean heat can be carried by the Gulf Stream from the Atlantic Ocean off the coast of North America into the Arctic Ocean, diving under the sea ice somewhere between Greenland and Svalbard.


Then, there is also the impact of the heat wave in Russia warming up the Arctic Ocean, as indicated by the red circle on the image below.


The image below shows May Northern Hemisphere ocean temperature anomalies with respect to the period 1901-2000, based on NOAA data and with a polynomial trendline added.



ACCELERATED WARMING IN ARCTIC CAUSING MORE CIRRUS CLOUDS

As oceans warm, the atmosphere can be expected to carry more water vapor. This conclusion is supported by studies such as this one. With more water vapor in the atmosphere, storms can be expected to strike with greater intensity. This conclusion is supported by studies such as this one. This situation gets worse as weather gets more extreme.

What makes things even worse is that, as the Gulf Stream keeps bringing ever warmer water into the Arctic Ocean, loss of sea ice in the Arctic Ocean and more open water will be the result. More open water means more opportunity for storms to develop and for water to evaporate into the atmosphere. The combination of more open water, more extreme weather, and more water vapor in the atmosphere leads to ever more severe storms that can come with destructive winds and that can suddenly unleash massive amounts of precipitation.

Studies such as this one warn that plumes above the anvils of severe storms can bring water vapor up into the stratosphere, contributing to the formation of cirrus clouds that block a lot of heat that would otherwise be radiated away, from Earth into space.

More cirrus clouds thus is another self-reinforcing feedback loop of accelerated warming in the Arctic. As the Gulf Stream keeps bringing ever warmer water into the Arctic Ocean, such feedbacks will further speed up warming, as discussed at the feedbacks page.

Are there geoengineering methods to reduce cirrus clouds? Seeding of high altitude clouds with ice may be able to do this, resulting in more longwave radiation escaping into space, as discussed in this study.

The text in this box was also posted at the Geoengineering group at facebook

The situation is dire and calls for comprehensive and effective action, as discussed at the Climate Plan. 


on June 8, 2015, sea surface temperatures as high as 11.4°C (52.52°F) were recorded to the south-east of Svalbard (a 9.8...
Posted by Sam Carana on Thursday, June 11, 2015

Monday, June 8, 2015

Scraping Services - Assuring Scraping Success with Proxy Data Scraping

Have you ever heard of "Data Scraping?" Data Scraping is the process of collecting useful data that has been placed in the public domain of the internet (private areas too if conditions are met) and storing it in databases or spreadsheets for later use in various applications. Data Scraping technology is not new and many a successful businessman has made his fortune by taking advantage of data scraping technology.

Sometimes website owners may not derive much pleasure from automated harvesting of their data. Webmasters have learned to disallow web scrapers access to their websites by using tools or methods that block certain ip addresses from retrieving website content. Data scrapers are left with the choice to either target a different website, or to move the harvesting script from computer to computer using a different IP address each time and extract as much data as possible until all of the scraper's computers are eventually blocked.

Thankfully there is a modern solution to this problem. Proxy Data Scraping technology solves the problem by using proxy IP addresses. Every time your data scraping program executes an extraction from a website, the website thinks it is coming from a different IP address. To the website owner, proxy data scraping simply looks like a short period of increased traffic from all around the world. They have very limited and tedious ways of blocking such a script but more importantly -- most of the time, they simply won't know they are being scraped.

You may now be asking yourself, "Where can I get Proxy Data Scraping Technology for my project?" The "do-it-yourself" solution is, rather unfortunately, not simple at all. Setting up a proxy data scraping network takes a lot of time and requires that you either own a bunch of IP addresses and suitable servers to be used as proxies, not to mention the IT guru you need to get everything configured properly. You could consider renting proxy servers from select hosting providers, but that option tends to be quite pricey but arguably better than the alternative: dangerous and unreliable (but free) public proxy servers.

There are literally thousands of free proxy servers located around the globe that are simple enough to use. The trick however is finding them. Many sites list hundreds of servers, but locating one that is working, open, and supports the type of protocols you need can be a lesson in persistence, trial, and error. However if you do succeed in discovering a pool of working public proxies, there are still inherent dangers of using them. First off, you don't know who the server belongs to or what activities are going on elsewhere on the server. Sending sensitive requests or data through a public proxy is a bad idea. It is fairly easy for a proxy server to capture any information you send through it or that it sends back to you. If you choose the public proxy method, make sure you never send any transaction through that might compromise you or anyone else in case disreputable people are made aware of the data.

A less risky scenario for proxy data scraping is to rent a rotating proxy connection that cycles through a large number of private IP addresses. There are several of these companies available that claim to delete all web traffic logs which allows you to anonymously harvest the web with minimal threat of reprisal. Companies such as offer large scale anonymous proxy solutions, but often carry a fairly hefty setup fee to get you going.

The other advantage is that companies who own such networks can often help you design and implementation of a custom proxy data scraping program instead of trying to work with a generic scraping bot. After performing a simple Google search, I quickly found one company (www.ScrapeGoat.com) that provides anonymous proxy server access for data scraping purposes. Or, according to their website, if you want to make your life even easier, ScrapeGoat can extract the data for you and deliver it in a variety of different formats often before you could even finish configuring your off the shelf data scraping program.

Whichever path you choose for your proxy data scraping needs, don't let a few simple tricks thwart you from accessing all the wonderful information stored on the world wide web!

Source: http://ezinearticles.com/?Assuring-Scraping-Success-with-Proxy-Data-Scraping&id=248993

Friday, June 5, 2015

High Temperatures in the Arctic


The images below illustrate extremely high temperatures forecast to hit Russia on June 6, 2015, as also discussed in the previous post.


A temperature of 29.4°C (84.92°F) is forecast for the location at the green circle for June 6, 2015. The location is close to the Arctic Ocean and to rivers ending in the Arctic Ocean, as also shown on the image below.


The location, at a latitude of 66.48°N, is approximately on the Arctic Circle, which runs 66°33′45.8″ north of the Equator. North of the Arctic Circle, the sun is above the horizon for 24 continuous hours at least once a year.


The many hours of sunshine make that, during the months June and July, insolation in the Arctic is higher than anywhere else on Earth, as shown on above image, by Pidwirny (2006).
Insolation, with contour labels (green) in units of W m−2

The size of the June snow and ice cover is so vitally important as insolation in the Arctic is at its highest at the June Solstice.

The Wikipedia image on the right calculates the theoretical daily-average insolation at the top of the atmosphere, where θ is the polar angle of the Earth's orbit, and θ = 0 at the vernal equinox, and θ = 90° at the summer solstice; φ is the latitude of the Earth.

The calculation assumed conditions appropriate for 2000 A.D.: a solar constant of S0 = 1367 W m−2, obliquity of ε = 23.4398°, longitude of perihelion of ϖ = 282.895°, eccentricity e = 0.016704.

Snow and ice cover on land can take up a large area, even larger than sea ice. In May 2015, the area of snow extent on the Northern Hemisphere was 17 million square km, while sea ice extent in May 2015 was below 13.5 million square km. 

Northern Hemisphere snow, May 2015. Credit: Rutgers University Global Snow Lab
The chart below shows the decline of snow cover on land on the Northern Hemisphere in Spring over the years. 

Credit: Rutgers University Global Snow Lab
High temperatures over the Arctic Ocean are heating up the snow cover on land and the sea ice from above. High temperatures also set the scene for wildfires that can emit huge amounts of pollutants, including dust and black carbon that, when settling on the sea ice, can cause its reflectivity to fall. Rivers furthermore feed warm water into the Arctic Ocean, further heating up the sea ice from below. 

The image below shows Arctic sea ice extent at June 3, 2015, when Arctic sea ice extent was merely 11.624 million square kilometers, a record low for the time of the year since satellite started measurements in 1979. 



Sea ice melting occurs due to heat from above, i.e. absorbed sunlight. Once the sea ice is gone, energy from sunlight that previously went into melting and transforming ice into water, will instead go into warming up the Arctic Ocean and the sediments under the seafloor.

In addition, sea ice is also melting due to heat from below. Much of this heat is carried by the Gulf Stream and by rivers into the Arctic Ocean. Once the sea ice is gone, all this heat will go into warming up the Arctic Ocean and the sediments under the seafloor.

The sea ice acts as a heat buffer by absorbing energy in the process of melting. In other words, as long as there is sea ice, it will absorb heat and this will prevent this heat from raising the temperature of the water in the Arctic. Once the sea ice is gone, this latent heat must go elsewhere.

As the sea ice heats up, 2.06 J/g of heat goes into every degree Celsius that the temperature of the ice rises. While the ice is melting, all energy (at 334J/g) goes into changing ice into water and the temperature remains at 0°C (273.15K, 32°F).

Once all ice has turned into water, all subsequent heat goes into heating up the water, at 4.18 J/g for every degree Celsius that the temperature of water rises.

The amount of energy absorbed by melting ice is as much as it takes to heat an equivalent mass of water from zero to 80°C. The energy required to melt a volume of ice can raise the temperature of the same volume of rock by 150º C.
Decline of Arctic sea ice means that a lot more heat will be absorbed by the Arctic Ocean.



Thick sea ice covered with snow can reflect as much as 90% of the incoming solar radiation. After the snow begins to melt, and because shallow melt ponds have an albedo (or reflectivity) of approximately 0.2 to 0.4, the surface albedo drops to about 0.75. As melt ponds grow and deepen, the surface albedo can drop to 0.15, while the ocean reflects only 6% of the incoming solar radiation and absorbs the rest.

As Professor Peter Wadhams, University of Cambridge, once calculated, a collapse of the sea ice would go hand on hand with dramatic loss of snow and ice cover on land in the Arctic. The albedo change resulting from the snowline retreat on land is similarly large as the retreat of sea ice, so the combined impact could be well over 2 W/sq m. To put this in context, albedo changes in the Arctic alone could more than double the net radiative forcing resulting from the emissions caused by all people of the world, estimated by the IPCC to be 1.6 W/sq m in 2007 and 2.29 W/sq m in 2013.

Professor Peter Wadhams on albedo changes in the Arctic

Update June 8, 2015: The website at earth.nullschool.net shows that over the past few days temperatures over 30°C (86°F) were reached at several locations over rivers ending up in the Arctic Ocean.

The animation below, by ClimateReanalyzer, shows the heat wave and the storm that hit the Arctic recently.

This animation shows the current GFS model 8-day forecast for the Arctic for six meteorological parameters (precip/cloudcover; wind, pressure, precipitable water, temperature, temperature anomaly). The forecast begins with an impressive storm twirling around the North Pole with 10-meter winds peaking around 55 km/h (~35 mi/h), which fades as the low pressure breaks down. The storm is coupled to an early season heat wave that hit Siberia this week with the development of a high amplitude ridge in the jet stream.In mid August 2012, a comparable storm churned up the sea ice and contributed to the record minimum ice extent that emerged in September. Arctic sea ice is more resilient to wind in early June when it is still relatively thick and compacted than it is in mid August towards the end of the melt season. This current storm is therefore unlikely to have the same impact as the Aug 2012 storm. But the event is worth mentioning nonetheless.

Posted by Climate Reanalyzer on Sunday, June 7, 2015

Arctic sea ice extent at June 3, 2015, was merely 11.624 million square kilometers, a record low for the time of the...

Posted by Sam Carana on Friday, June 5, 2015

Tuesday, June 2, 2015

Twitter Scraper Python Library

I wanted to save the tweets from Transparency Camp. This prompted me to turn Anna‘s basic Twitter scraper into a library. Here’s how you use it.

Import it. (It only works on ScraperWiki, unfortunately.)

from scraperwiki import swimport

search = swimport('twitter_search').search

Then search for terms.

search(['picnic #tcamp12', 'from:TCampDC', '@TCampDC', '#tcamp12', '#viphack'])

A separate search will be run on each of these phrases. That’s it.

A more complete search

Searching for #tcamp12 and #viphack didn’t get me all of the tweets because I waited like a week to do this. In order to get a more complete list of the tweets, I looked at the tweets returned from that first search; I searched for tweets referencing the users who had tweeted those tweets.

from scraperwiki.sqlite import save, select

from time import sleep

# Search by user to get some more

users = [row['from_user'] + ' tcamp12' for row in \

select('distinct from_user from swdata where from_user where user > "%s"' \

% get_var('previous_from_user', ''))]

for user in users:

    search([user], num_pages = 2)

    save_var('previous_from_user', user)

    sleep(2)

By default, the search function retrieves 15 pages of results, which is the maximum. In order to save some time, I limited this second phase of searching to two pages, or 200 results; I doubted that there would be more than 200 relevant results mentioning a particular user.

The full script also counts how many tweets were made by each user.

Library

Remember, this is a library, so you can easily reuse it in your own scripts, like Max Richman did.

Source: https://scraperwiki.wordpress.com/2012/07/04/twitter-scraper-python-library/

Monday, June 1, 2015

Heat Wave Forecast For Russia Early June 2015


Following heat waves in Alaska and the north of Canada, the Arctic looks set to be hit by heat waves along the north coast of Russia in early June, 2015. The image below shows temperature anomalies at the top end of the scale for a large area of Russia forecast for June 6, 2015.


Meanwhile, the heat wave in India continues. It killed more than 2,100 people, reports Reuters, adding that the heat wave also killed more than 17 million chickens in May. The number of people killed by the heat wave is now approaching the 2,541 people killed by the 1998 heat wave in India, which is listed as the record number of deaths due to extreme temperatures in India by the Emergency Events Database.

Further records listed by the database are the well over 70,000 people killed by the 2003 heat wave in Europe and 55,736 people killed by the 2010 heat wave in Russia alone.

On above temperature forecast (left image, top right), temperatures over a large area of India will be approaching the top end of the scale, i.e. 50°C or 120°F. While such temperatures are not unusual in India around this time of year, the length of the heat wave is extraordinary. The heat wave that is about to hit Russia comes with even higher temperature anomalies. Even though temperatures in Russia are unlikely to reach the peaks that hit India, the anomalies are at the top end of the scale, i.e. 20°C or 36°F.

The image below shows a forecast for June 6, 2015, with high temperatures highlighted at four locations (green circles).


Below is a forecast for the jet stream as at June 7, 2015.

The animation below runs the time of the top image (June 6, 2015, 0900 UTC) to the above image (June 7, 2015, 1200 UTC), showing forecasts of the jet stream moving over the Arctic Ocean, with its meandering shape holding warm air that extends from Russia deep into the Arctic Ocean.


Below is another view of the situation.
Jet stream on June 6, 2015, 0900 UTC, i.e. the date and time that corresponds with the top image.
Clicking on this link will bring you to an animated version that also shows the wind direction, highlighting the speed (I clocked winds of up to 148 km/h, or 92 mph) of the jet stream as it moves warm air from Russia into the Arctic Ocean, sped up by cyclonic wind around Svalbard.

This is the 'open doors' feedback at work, i.e. feedback #4 on the feedbacks page, where accelerated warming in the Arctic causes the jet stream to meander more, which allows warm air to enter the Arctic more easily, in a self-reinforcing spiral that further accelerates warming in the Arctic.

The implications of temperatures that are so much higher than they used to be are huge for the Arctic. These high temperatures are heating up the sea ice from above, while rivers further feed warm water into the Arctic Ocean, heating up the sea ice from below.

Furthermore, such high temperatures set the scene for wildfires that can emit huge amounts of pollutants, among which dust and black carbon that, when settling on the sea ice, can cause large albedo falls.

The image below shows Russian rivers that end up in the Arctic Ocean, while the image also shows sea surface temperature anomalies as high as 8.2°C or 14.76°F (at the green circle, near Svalbard).



The big danger is that the combined impact of these feedbacks will accelerate warming in the Arctic to a point where huge amounts of methane will erupt abruptly from the seafloor of the Arctic Ocean.
The image below shows that methane levels as high as 2,566 ppb were recorded on May 31, 2015, while high methane levels are visible over the East Siberian Arctic Shelf.


Below is part of a comment on the situation by Albert Kallio:
As the soils warm up the bacteria in them and the insulating capacities of snow themselves tend to lead snow cover melting faster the warmer the soil it rests on becomes. (Thus the falling snow melts very rapidly on British soil surface if compared to Finland or Siberia where the underlying ground is much colder, even if occasionally the summers have similar or even higher temperatures).

The large snow cover over the mid latitude land masses is a strong negative feedback for the heat intake from the sun if the season 2015 is compared with the season 2012, but the massive sea ice and polar air mass out-transportation equally strongly weakens formation of new sea ice around the North Pole (and along the edges of the Arctic Ocean) as the air above the Arctic Ocean remains warm. The pile up of thin coastal ice also increases vertical upturning of sea water and this could have detrimental effects for the frozen seabed that is storing methane clathrates. The sunlight intake of the sea areas where sea ice has already disappeared corresponds largely with the 2012 season.

The inevitable snow melting around the Arctic Ocean will also transport record volumes of warmed melt water from the south to the Arctic Ocean. The available heat in the Arctic may also be later enhanced by the high sea water temperatures that prevail along the eastern and western coasts of North America, as well as El Nino event increasing temporarily air and sea surface temperatures. This leads to more depressions around Japan and Korea from where the warm air, storms and rains migrate towards Alaska and pull cold air away from Arctic over Russia, while pushing warm air through the Baring Strait area and Alaska to the Arctic Ocean region.

Forecasting seasonal out comes is likely to be increasingly difficult to make due to increasing number of variables in the seasonal melting processes and the resulting lack of historic precedents when the oceans and Arctic has been as warm as today. Thus the interplay of the opposing forces makes increasingly chaotic outcomes, in which the overall trend will always be for less ice and snow at the end of the season. Because of these reasons - including many others not explicitly mentioned here - the overall outcome for the blue ocean, or the ice-free Arctic Ocean, will be inevitable.

Whether the loss of sea ice happens this summer, or next, or one after that, the problem isn't going to go away and more needs to be done to geoengineer to save Arctic ice and wildlife dependent on summer sea ice.
John Davies responds:
Albert Kallio is absolutely right in saying that warmer temperatures are leading to a blue ocean event though the problem remains in which year this will happen. Additionally Methane is being released from the bottom of the ocean leading to increased Methane concentrations and all that means for a destabilising global climate. Frustratingly, the higher temperatures and increasing Methane concentrations are not yet quite sufficient for us to persuade the scientific community and the public that Armageddon is on the way. Hence it is not yet possible to be in a position to persuade the world community of the urgent need for Geo-engineering to save the Arctic and Global climate. However we may reach this situation in the near future and that will be the only time when it might be possible to save the global climate and prevent Armageddon.

The situation is dire and calls for comprehensive and effective action, as discussed at the Climate Plan page.



This image shows Russian rivers that end up in the Arctic Ocean, while it also shows sea surface temperature anomalies...
Posted by Sam Carana on Monday, June 1, 2015

Sunday, May 31, 2015

Arctic Methane Skyrocketing

The map below shows observatories in the Arctic.


    'Arctic methane skyrocketing' is the title of a video by Paul Beckwith discussing recent rises in methane levels in the Arctic.


    Paul's description: "I discuss how ground level flask measurements of methane have been spiking upwards over the last few years. I analyze the implications to the breakdown of climate stability, causing jet stream fracturing and weather regime change. I believe that this behaviour will rapidly worsen as Arctic temperature amplification continues, leading our planet to a much warmer and unrecognizable climate over the next 5 to 10 years."

    Below are some NOAA images showing methane levels (surface flasks) recorded at Arctic observatories.




    Below is an image showing hourly average in situ measurements at Barrow, Alaska, including one very high reading, from an earlier post.


    The image below shows sea surface temperature anomalies in the Arctic on May 30, 2015.


    The situation is dire and calls for comprehensive and effective action, as discussed at the Climate Plan page.


    Sea surface temperature anomalies in the Arctic on May 30, 2015. From the post: 'Arctic Methane Skyrocketing' http://arctic-news.blogspot.com/2015/05/arctic-methane-skyrocketing.html

    Posted by Sam Carana on Sunday, May 31, 2015

    Thursday, May 28, 2015

    Data Scraping Services - Things to take care while doing Web Scraping!!!

    In the present day and age, web scraping word becomes most popular in data science. Basically web scraping is extracting the information from the websites using pre-written programs and web scraping scripts. Many organizations have successfully used web site scraping to build relevant and useful database that they use on a daily basis to enhance their business interests. This is the age of the Big Data and web scraping is one of the trending techniques in the data science.

    Throughout my journey of learning web scraping and implementing many successful scraping projects, I have come across some great experiences we can learn from.  In this post, I’m going to discuss some of the approaches to take and approaches to avoid while executing web scraping.

    User Proxies: Anonymously scraping data from websites

    One should not scrape website with a single IP Address. Because when you repeatedly request the web page for web scraping, there is a chance that the remote web server might block your IP address preventing further request to the web page. To overcome this situation, one should scrape websites with the help of proxy servers (anonymous scraping). This will minimize the risk of getting trapped and blacklisted by a website. Use of Proxies to hide your identity (network details) to remote web servers while scraping data. You may also use a VPN instead of proxies to anonymously scrape websites.

    Take maximum data and store it.

    Do not follow “process the web page as it comes from the remote server”. Instead take all the information and store it to disk. This approach will be useful when your scraping algorithm breaks in the middle. In this case you don’t have to start scraping again. Never download the same content more than once as you are just wasting bandwidth. Try and download all content to disk in one go and then do the processing.

    Follow strict rules in parsing:

    Check various rules while parsing the information from the web site. For example if you expect a value to be a date then check that it’s really a date. This may greatly improve the quality of information. When you get unexpected data, then the algorithm need to be changed accordingly.

    Respect Robots.txt

    Robots.txt specifies the set of rules that should be followed by web crawlers and robots. I strongly advise you to consider and adjust your crawler to fully respect robots.txt. Robots.txt contains instructions on the exact pages that you are allowed to crawl, user-agent, and the requisite intervals between page requests. Following to these instructions minimizes the chance of getting blacklisted and banned from website owner.

    Use XPath Smartly

    XPath is a nice option to select elements of the HTML document more flexibly than CSS Selectors.  Be careful about HTML structure change through page to page so one xpath you made may be failed to extract data on another page due to changes in HTML structure.

    Obey Website TOC:

    Some websites make it absolutely apparent in their terms and conditions that they are particularly against to web scraping activities on their content. This can make you vulnerable against possible ethical and legal implications.

    Test sample scrape and verify the data with actual scrape

    Once you are done with web scraping project set up, you need to test it for sometimes. Check the extracted data. If something is not good, find out the cause and make changes accordingly and finally come to a perfect web scraping project.

    Source: http://webdata-scraping.com/things-take-care-web-scraping/