Thursday, April 30, 2015

Web Data Extraction Solutions for business Automation

Your business today is driven purely and solely by information. This vital component is carved out from data scraped from relevant sources, cleansed and compiled to form the crux of your enterprise analytics plan. Therefore, following a manual practice for Data Extraction often makes it prone to errors which may be detrimental to the health of your business. Various Web Data Extraction Solutions therefore are made available which help to not only automate Website Data Scraping, but in the process also help in the automation of several business processes.  Let us take a look at some of these business processes:

Execute Data Mining

One of the most obvious usages of the data extractor tools is the function of Web Data Mining, which, if done through manual processes is neither cost-effective nor accurate. Therefore, data extraction solutions provide simple and convenient point and click data extraction interfaces. Moreover, these do not require any additional programming knowledge.

Validate Data accuracy

Web Site Data scraping tools use advanced technology to not only extract data, but also validate their accuracy. This is particularly beneficial for businesses which are involved in background screening and credit reporting activities. Automating the process of validating records helps in increasing turnaround time of the business and ensures accuracy, two of the most crucial criteria for achieving success in their line of business.

Be Price Wise

Organizations are always involved in studying the price dynamics in order to better understand the challenges and areas of opportunities. Your awareness on these helps you to provide more competitive pricing for your products and services. Data extraction solutions are equipped with pricing intelligence technology that helps to collect data on the price that your customer’s expect, their feedback on your products and services and helps you develop insights on products being launched by competitors, their price and availability. This automated process therefore ensures the following things for your business:

•    Increased market – share

•    Enhanced product strategy

•    Informed decisions

Be Compliance Ready

If yours is a financial services firm, you must have often found it to be a major challenge to keep yourself abreast of the compliance and risk factors. Tracking the myriad watch lists, sanction lists and federal and state regulations, available on the web is not only time taking but also an expensive endeavor. The automated Web Data Mining tools ensure that you are now able to do this without impacting the time and money aspect of your business. You are now ensured of compliance with all regulations and sanction lists that are updated on a regular basis. Consequently, your business can also breathe easy with a reduced exposure to financial fraud and identity theft.

Easy Access to Customer Feedback

Your customers are in a way the real owners of your business. They define the way you need to design your product strategy. It is therefore crucial that you need to listen to what they have to say.  The automated Web extraction solutions with their ability to tap into several relevant sources help you to get access to customer sentiments and feedback on your products and services. This is a vital aspect in your organization's growth as it helps you to tackle three important factors:

•    Assimilate positive or negative vibes on your newly launched product or service which might require you to revisit your product strategy

•    Provide immediate attention to issues, if any, with any particular product

Create a trend of aggregated customer sentiments, for analysis

Source: http://scraping-solutions.blogspot.in/2014_07_01_archive.html

Tuesday, April 28, 2015

Scraping a website from a windows service

Question

Hi there.  I have a windows forms application that scrapes a website to retrieve some data.  I would like to implement the same functionality as a windows service.  The reason for this is to allow the program to run 24/7 without  having a user signed in. 

To that end, my current version of the program uses a web browser control (system.windows.forms.webbrowser) to navigate the pages, click the buttons, allow scripts to do their thing, etc.  I cannot figure out a way to do the same without the web browser control, but the web browser control cannot be instantiated in a windows service (because there is no user interface in a web service).

Does anyone have any brilliant ideas on how to get around this?

Thank you very much!

Answers

Hi Andy,

There is a tool which could let you manipulate anything you want on the website. This agile HTML parser builds a read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams). More information, please check:

http://htmlagilitypack.codeplex.com/

Have a nice day.

Best regards

All replies

You are not telling if you are using a .NET Express edition or not

You are not telling which Framework

You are not realy saying what data you are getting from the web site.

So

I made an example of service that work on any Studio edition (including the Express)

to install it, I supposed that you have at least the Framework2, so you will use something similar to:

    %SystemRoot%\Microsoft.NET\Framework\v2.0.50727\installutil /i C:\Test\MyWindowService\MyWindowService\bin\Release\MyWindowService.exe

In the example, I supposed that you are downloading some file from the site

You will need a reference to Windows.Form for the timer


Imports System.ServiceProcess

Imports System.Configuration.Install

Public Class WindowsService : Inherits ServiceBase

  Private Minute As Integer = 60000

  Private WithEvents Timer As New Timer With {.Interval = 30 * Minute, .Enabled = True}

  Public Sub New()

    Me.ServiceName = "MyService"

    Me.EventLog.Log = "Application"

    Me.CanHandlePowerEvent = True

    Me.CanHandleSessionChangeEvent = True

    Me.CanPauseAndContinue = True

    Me.CanShutdown = True

    Me.CanStop = True

  End Sub


  Private Sub Timer_Tick(ByVal sender As Object, ByVal e As System.EventArgs) Handles Timer.Tick

    If IO.File.Exists("C:\MyPath.Data") Then IO.File.Delete("C:\MyPath.Data")

    My.Computer.Network.DownloadFile("http://MyURL.com", "C:\MyPath.Data", "MyUserName", "MyPassword")

    'Do Something with the data downloaded

  End Sub

End Class

<Microsoft.VisualBasic.HideModuleName()> _

Module MainModule

  Public TheServiceName As String

  Public Sub main()

    Dim TheServiceApplication As New WindowsService

    TheServiceName = TheServiceApplication.ServiceName

    ServiceBase.Run(TheServiceApplication)

  End Sub

End Module

<System.ComponentModel.RunInstaller(True)> _

Public Class WindowsServiceInstaller : Inherits Installer

  Public Sub New()

    Dim serviceProcessInstaller As ServiceProcessInstaller = New ServiceProcessInstaller()

    Dim serviceInstaller As ServiceInstaller = New ServiceInstaller()

    serviceProcessInstaller.Account = ServiceAccount.LocalSystem

    serviceProcessInstaller.Username = Nothing

    serviceProcessInstaller.Password = Nothing

    serviceInstaller.DisplayName = "My Windows Service"

    serviceInstaller.StartType = ServiceStartMode.Automatic

    serviceInstaller.ServiceName = TheServiceName

    Me.Installers.Add(serviceProcessInstaller)

    Me.Installers.Add(serviceInstaller)

  End Sub

End Class



 Hello Andy,

Thanks for your post.

What do you want to scrape from the page? HttpWebRequest class ans WebClient class may be what you need. More information, please check:

The HttpWebRequest class provides support for the properties and methods defined in WebRequest and for additional properties and methods that enable the user to interact directly with servers using HTTP.

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx

The WebClient class provides common methods for sending data to or receiving data from any local, intranet, or Internet resource identified by a URI

http://msdn.microsoft.com/en-us/library/system.net.webclient.aspx

If you have any concenrs, please feel free to follow up.

Best regards



Hi Andy,

What about this problem on your side now? If you have any concerns, please feel free to follow up.

Have a nice day.

Best regards



Hi Andy,

When you come back, if you need further assistance about this issue, please feel free to let us know. We will continue to work with this issue.

Have a nice day.

Best regards



Thank you for the reply. Sorry it has taken me so long to respond.  I did not receive any notification that someone had replied!

I am using Visual Studio 2010 Ultimate Edition and the .NET framework 4.0.  Actually, I am upgrading some old code written in VB 6.0, but I can use the latest and greatest thats available.

The application uses a browser control to go to the page, fill in values, click on UI elements, read the HTML that returns, etc.  The purpose of the application is to collection useful information regularily/automatically.

I know how to create a web service, but using the web control in such a service is problematic because the web browser control was meant to be placed on a windows form.  I am not able to create a new instance of it in a project designated as a windows service.



Andy

Thank you for the reply. Sorry it has taken me so long to respond.  I did not receive any notification that someone had replied!

I thought a web request was for web services (retrieving information from them).  I am trying to retreive useful information from a website designed for interaction by a human, such as selecting items from lists and clicking buttons.   I currently use a web browser control to programmatically do what a person would do and get the pages back which in turn get parsed.

Andy



Hi Andy,

There is a tool which could let you manipulate anything you want on the website. This agile HTML parser builds a read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams). More information, please check:

http://htmlagilitypack.codeplex.com/

Have a nice day.

Best regards



Thanks for the suggestion.  I will go to that link and see if it will work.  I will update this post with what I find.

I am writing to check the status of the issue on your side. Would you mind letting us know the result of the suggestions? If you have any concerns, please feel free to follow up.

Have a nice day.



Best regards

Hi Liliane

Thank for the follow up reply.  I don't have an answer as of yet.  Implementing this is going to take time and I haven't been given the go-ahead by my boss to spend the time to pursue it.



Hi Andy,

Never minde. You could have a try when you feel free. If you have any further questions about this issue, please feel free to let us know. We will continue to work with you on this issue.

Have a nice day.

Best regards

Source: https://social.msdn.microsoft.com/Forums/vstudio/en-US/f5d565b1-236b-43c2-90c7-f5cc3b2c341b/scraping-a-website-from-a-windows-service

Sunday, April 26, 2015

Methane levels as high as 2845ppb

Methane levels as high as 2845 parts per billion (ppb) were recorded on April 25, 2015, as illustrated by the image below.


This is an extremely high peak. The average daily peak in 2015 until May 1 was 2371 ppb, while the highest daily mean ranged from 1807 ppb (January 10) to 1829 ppb (April 22). Daily peaks and daily highest mean levels in 2015 are shown on the image below.



These peaks are much higher than they were in previous years, as illustrated by the image below, from an earlier post and showing the average highest peak readings in 2013 and 2014 at selected altitudes..


Peak readings in above image are averages over April 2013 and April 2014. On specific days, peak readings could be much higher, e.g. on April 28, 2014, methane levels were recorded as high as 2551 ppb at 469 mb. As said, methane levels as high as 2845 ppb were recorded on April 25, 2015, while the average peak for the first four months of 2015 was 2371 ppb, and this average was calculated from peaks across altitudes.

The table below shows the altitude equivalents in mb (millibar) and feet.
56,925 ft44,689 ft36,850 ft30,569 ft25,543 ft19,819 ft14,383 ft8,367 ft1,916 ft
74 mb147 mb218 mb293 mb367 mb469 mb586 mb742 mb945 mb

Peak levels in April appear to be rising strongly each year, following higher peak readings during previous months, especially at higher altitudes, i.e. especially the Arctic Ocean. It appears that much of the additional methane originating from the higher latitudes of the Northern Hemisphere has moved closer to the equator over the past few months, and is now accumulating at higher altitudes over the continents on the Northern Hemisphere, i.e. Asia, Europe, North America and north Africa.

Further analysis of the rise in global mean methane levels appears to confirm the above. The image below shows methane levels on April 22, over three years. While there appears to be little or no rise in mean methane levels at low altitudes, the rise is quite profound at higher altitudes.  

[ click on image to enlarge ]
Things look set to get worse. As shown by the image below, from an earlier post, global methane levels have risen sharply from a low of 723 ppb in 1755. Mean methane levels were as high as 1839 ppb in 2014. That's a rise of more than 254%.
As that post concluded a year ago, it appears that the rise of methane in the atmosphere is accelerating. What can we expect? As temperatures can be expected to continue to rise and as feedbacks start to kick in, this may well constitute a non-linear trend. The image below shows a polynomial trend that is contained in IPCC AR5 data from 1955 to 2011, pointing at methane reaching mean global levels higher than 3000 ppb by the year 2030. If methane starts to erupt in large quantities from clathrates underneath the seafloor of the Arctic Ocean, this may well be where we are heading. 




The situation is dire and calls for comprehensive and effective action, as discussed at the Climate Plan.



The 2845 ppb recorded on April 25, 2015, is an extremely high peak. The average daily peak in 2015 until now was 2372...
Posted by Sam Carana on Monday, April 27, 2015

Saturday, April 25, 2015

Data Mining and Market Research

Online market research attributes to success and growth of many businesses. Online market research in simple terms we can say it is the learning of current and the latest market situations which involve surveys, web and data mining modules. To date research by use of the internet it is very important since it depends on data gathered from internet services and then one can recognize that market research keeps the business successful.

A number of managers in small businesses have a mental deem in that online market research is obligatory to big or larger companies. For true you will understand that whether the businesses is medium, large or small actually need online marketing research and this is a reason why the significance of the process and allegation will approve the targeted and potential clients. In this case Data mining progression is employed to streamline on what targeted and potential clientele needs. Areas where data mining is used:

Preferences. In any given service or product, you will learn what a customer looking for and how your product or service is different from other competitors. By use of Data mining you will be able to determine the customer preferences and you will be able to modify your services and products meet the customer choice.

Buying patterns. What is known and created for purchasing patterns from different customers. A situation can be that customers try to spend a lot on certain products and little on others. But through Data mining it is easy to understand such purchasing patterns and finally plan the appropriate techniques to be used in the marketing.

Prices. To find out whether the company is selling its products to the clients or not prices are the key factors to take into account. One should understand on the right selling price of the products. Web scrapping is easier to find the suitable pricing.

Source: http://www.loginworks.com/blogs/web-scraping-blogs/data-mining-market-research/

Thursday, April 23, 2015

It's time to 'Do the math' again

By David Spratt

Have we gone mad? A new report released today explains why contemporary climate change policy-making should be characterised as increasingly delusional.

As the deadline approaches for submissions to the Australian government's climate targets process, there is a flurry of submissions and reports from advocacy groups and the Climate Change Authority.

Most of these reports are based on the twin propositions that two degrees Celsius (2°C) of global warming is an appropriate policy target, and that there is a significant carbon budget and an amount of "burnable carbon" for this target, and hence a scientifically-based escalating ladder of emission-reduction targets stretching to mid-century and beyond.

A survey of the relevant scientific literature by David Spratt, "Recount: It's time to 'Do the math' again", published today by Breakthrough concludes that the evidence does not support either of these propositions.

The catastrophic and irreversible consequences of 2°C of warming demand a strong risk-management approach, with a low rate of failure. We should not take risks with the climate that we would not take with civil infrastructure.

There is no carbon budget available if 2°C is considered a cap or upper boundary as per the Copenhagen Accord, rather than a hit-or-miss target which can be significantly exceeded; or if a low risk of exceeding 2°C is required; or if positive feedbacks such as permafrost and other carbon store losses are taken into account.

Effective policy making can only be based on recognising that climate change is already dangerous, and we have no carbon budget left to divide up. Big tipping-point events irreversible on human time scales such as in West Antarctica and large-scale positive feedbacks are already occurring at less than 1°C of warming. It is clear that 2°C of climate warming is not a safe cap.

In reality, 2°C is the boundary between dangerous and very dangerous climate change and 1°C warmer than human civilisation has ever experienced.

In the lead up to the forthcoming Paris talks, policy makers through their willful neglect of the evidence are in effect normalising a 2.5–3°C global warming target.

This evidence in "Recount: It's time to 'Do the math' again" demonstrates that action is necessary at a faster pace than most policy makers conceive is possible.



Related

- It's time to 'Do the math' again
http://www.climatecodered.org/2015/04/its-time-to-do-math-again.html

- RECOUNT - It's time to 'Do the math' again
http://media.wix.com/ugd/148cb0_938b5512abfa4d4e965ec8cc292893f7.pdf

- Two degrees of warming closer than you may think
http://arctic-news.blogspot.com/2015/02/two-degrees-of-warming-closer-than-you-may-think.html

- The real budgetary emergency and the myth of "burnable carbon"
http://arctic-news.blogspot.com/2014/05/the-real-budgetary-emergency-and-the-myth-of-burnable-carbon.html

It's time to 'Do the math' again | by David Spratt http://arctic-news.blogspot.com/2015/04/its-time-to-do-math-again.html

Posted by Sam Carana on Thursday, April 23, 2015

Saturday, April 18, 2015

The Great Unraveling


The great unraveling of how climate catastrophe is unfolding on land and in the oceans, in the atmosphere and the cryosphere, is becoming more and more clear every month.

March 2015 temperatures were the highest for March in the 136-year period of record. NOAA analysis shows that the average temperature across global land and ocean surface temperatures combined for March 2015 was 0.85°C (1.53°F) higher than the 20th century average of 12.7°C (54.9°F).

Ocean temperature anomalies on the Northern Hemisphere for March 2015 were the highest on record. In many ways, the situation looks set to get worse. For the 12-month period from April to March, data from 1880 contain a trendline that points at a rise of 2 degrees Celsius by the year 2032, as illustrated by the image below.

Click on image to enlarge
The rise in Northern Hemisphere ocean temperatures was especially profound in September and October 2014, when methane started to erupt from the Arctic Ocean seafloor in huge quantities.

The image below shows a polynomial trendline pointing at an October Northern Hemisphere sea surface temperature anomaly rise of 2°C (3.6°F) by 2030, and a rise of more than 5°C (9°F) by 2050, compared to the 20th century average, from an earlier post.
From: Ocean Temperature Rise continues
The images below give an idea of the current sea surface temperature anomalies around North America.


On April 11, 2015, a sea surface temperature of 22.2°C (71.96°F) was recorded off the North
American coast (green circle bottom), a 12.6°C (22.68°F) anomaly (green circle top).


Ocean heat is carried by the Gulf Stream from the North Atlantic into the Arctic Ocean. The huge amounts of energy entering the oceans translate into higher temperatures of the water and of the air over the water, as well as higher waves and stronger winds.

The image below highlights waves and winds, showing that waves as high as 12.06 m (39.57 ft) were recorded off the coast of North America in the path of the Gulf Stream, while winds with speeds as high as 115 km/h (71.46 mph) were recorded in that area on April 17, 2015.


The combination image below illustrates the threat. A sea surface temperature of 8°C (46.4°F, green circle left) was recorded near Svalbard on April 17, 2015, an anomaly of 6.2°C (11.16°F, green circle right).

Click on image to enlarge
A continued rise of ocean temperatures on the Northern Hemisphere threatens to unleash huge eruptions of methane from the seafloor of the Arctic Ocean, further accelerating the temperature rise in the Arctic and escalating into runaway global warming.

Malcolm Light comments: "The Pacific heating must be caused by the southward spreading Arctic methane global warming veil that is able to penetrate through a giant hole in the hydroxyl and ozone layer over the far east and is moving eastwards."

Current methane levels remain extremely high (see this recent post), on track to break the record mean level of 1839 ppb (parts per billion) reached in September 2014.


Above image shows that the highest mean methane levels ranged from 1815 ppb on March 30, 2015, to 1828 ppb on April 17, 2015. The highest peak level during this period was 2483 ppb, reached on April 15, 2015.

The extremely high methane levels are undoubtedly contributing to the high temperatures reached in March, especially at higher latitudes, on top of the dramatic global rise of greenhouse gases in general, as illustrated by above contribution by Peter Carter.


Above image shows that temperature anomalies over much of the Arctic Ocean were at the top end of the scale on April 17, 2015, i.e. 20°C or 36°F.

The image below gives an idea of the temperature differences on April 17, 215. While temperatures over the Sahara in Africa were as high as 32.1°C (89.78°F), temperatures over Greenland were as low as -41°C (-41.8°F). In between, temperatures of 2.8°C (37.04°) were recorded over the waters near Svalbard and of 6.1°C (42.98°F) closer to the coast of Norway.



Such wide temperature differences highlight the importance of looking at peaks, rather than at averages. The year-to-date maximum sea surface temperature anomaly, up to April 18, 2015, gives an idea of the peak anomalies that can be expected as the hot season approaches on the Northern Hemisphere.



Below are details for March 2015.



Temperature anomalies as high as 10.2°C (or 18.3°F) were recorded for March 2015 on Kolguyev Island in the Barents Sea.

A rise in ocean temperatures on the Northern Hemisphere of 2°C (3.6°F) by October 2030 looks set to go hand in hand with a 6°C (10.8°F) rise in Arctic temperatures by 2030, fueling runaway global warming, as illustrated by the image below, from another earlier post.

Without action, similar temperature rises look set to hit the globe at large a dozen years later, accompanied by huge temperature swings that threaten to cause depletion of supply of food and fresh water, as discussed by Guy McPherson in the video below and illustrated by the image further below.

Guy McPherson (left) in discussion with Paul Beckwith (right)

From Methane Levels Early 2015

In conclusion, the situation is dire and calls for comprehensive and effective action, as discussed at the Climate Plan blog.



Sources and Related

- Ocean temperatures, NOAA
http://www.ncdc.noaa.gov/sotc/global/2015/03

- Sea Surface Temperatures, from:
http://earth.nullschool.net
and from:
http://polar.ncep.noaa.gov/sst/ophi

- Kolguyev Island temperature anomaly, from:
http://data.giss.nasa.gov/tmp/gistemp/NMAPS/tmp_GHCN_GISS_ERSST_250km_Anom03_2015_2015_1951_1980/nmaps.txt

- Temperature anomaly April 17, 2015, Climate Reanalyzer
http://cci-reanalyzer.org

- Year-to-date maximum sea surface temperature anomaly April 18, 2015, from:
http://coralreefwatch.noaa.gov/satellite/bleaching5km/index_composites_5km.php

- Methane levels. NOAA IASI MetOp
http://www.ospo.noaa.gov/Products/atmosphere/soundings/iasi

- The Mechanism
http://arctic-news.blogspot.com/2015/02/the-mechanism.html

- Three kinds of warming (temperature trendlines), from: Methane levels Early 2015
http://arctic-news.blogspot.com/2015/03/methane-levels-early-2015.html

- Northern Hemisphere October Ocean Temperature Rise, from:
http://arctic-news.blogspot.com/2014/11/ocean-temperature-rise-continues.html


Ocean temperature anomalies on the Northern Hemisphere for March 2015 were the highest on record. In many ways, the...
Posted by Sam Carana on Saturday, April 18, 2015

Saturday, April 11, 2015

Arctic Sea Ice At Record Low On April 9 2015

On April 9, 2015, Arctic sea ice extent was only 14.051 square km, a record low for the time of the year, as illustrated by the image below.

Temperature anomalies at the top end of the scale (20°C, or 36°F) are hitting the Arctic Ocean in many places, as illustrated by the forecast below, showing an overall anomaly of +3.19°C for the Arctic for April 11, 2015, despite low temperatures over Greenland.


The situation is very worrying, the more so since a huge amount of ocean heat is lining up to be carried into the Arctic Ocean by the Gulf Stream. On April 10, 2015, sea surface temperatures of 24.1°C were recorded off the North American coast (green circle), a +12.5°C anomaly, as the image below shows.


Malcolm Light comments: In this inverted blowup of the high temperature region you can see the expanded effect of methane hydrate detabilization along the Gakkel Ridge and the high temperatures caused by the onshore methane eruption vents (image below).



The situation is dire and calls for comprehensive and effective action, as discussed at the Climate Plan blog.


On April 9, 2015, Arctic sea ice extent was only 14.051 square km, a record low for the time of the year. From the post ...

Posted by Sam Carana on Saturday, April 11, 2015