Friday, 31 May 2013

Data Mining, Visual Analytics, and The Human Component!

With all the massive amounts of data we are collecting from the Internet, well, it is just amazing the things we can do with it all. Of course, those concerned about privacy, well, you can understand why organizations like the Electronic Freedom Foundation is often fit to be tied. Still, think of all the good that can become of all this data? Let me explain.

You see, with the right use of visual analytics and various data mining strategies, we will be able to do nearly anything we need too. And, yes, I guess it goes without saying that I have a ton of thoughts on Visual Analytics of the Internet, Mobile Ad Hoc networking, and Social Networks along with some concepts for DARPAs plan for "crowd sourcing" innovation, it makes perfect sense to me, as each participant becomes basically a "neuron" and we use the natural neural network scheme.

What we need is a revolution in data mining visual analytics, so the other day I spent 20-minutes considering this and here are my thoughts. I propose an entirely new concept herein. Okay so let me explain my concept. But first let me briefly describe the bits and pieces of ideas and concepts I borrowed from to come up with this;

    There is an only UFO or Sci Fi tale I read, where the alien race said; "There is a whole new world waiting for you if you dare to take it,"
    Taking the "it" part of that line and calling "it" = "IT" as in Information Technologies.
    Next, combining that "IT" or "It entity" with that old Christian apocalyptic "mark of the beast" and the old computer system in Belgium 30-years ago claiming to be big enough to track every world transaction, also nick-named the beast.
    Then combining that concept with V. Bush's concept of "recording a life" or the later "life log theory" from Bell Labs.
    Then using the concept of the eRepublic, where government is nothing more than a networked website.
    Then considering the thought of Bill Gate's concepts in "the Road Ahead" where the digital nervous system of a corporation was completely and fully integrated.
    Combined with SAPs, and Oracles enterprise solutions
    Combined with Google's data bases
    Combined with the Pangaea Project for kids to collaborate in elementary school around the world and programming the AI computer, using a scheme designed by Carnegie Mellon to crowd source the teaching of an AI system. "eLearning Collaborative Networks like Quorum or Pangaea"
    Combined with IBMs newest mind map visualization recently in the news..
    Combined with these following thoughts of mine:

    My Book; "The Future of Truck Technologies," and 3D and 4D Transportation Computer Modeling; Page; 201.
    My Book; "Holographic Technologies," specifically; Data Visualization Schemes; Page 57 Chapter 5.
    My Article on 3D and 4D Mind Maps for Tracking and Analyzing.
    My Article on Mind Maps of the Future and Online style Think Tanks
    My Article on Stair Step Mentorship for Human Learning in the Future and Never Aging Societies.

Okay now let me explain the premise of my concept for Visual Analytics;

First, forget this whole idea of a 2D mind mapping concept or chart used to show links between terrorist players, cells, assets, acquaintances, etc., the way it is laid out currently - make it 3D, actually make it 4D and 5D where some layers can only be seen by a select few, and let's say a 6D level that can only be accessed by an AI super computer [why; because I don't trust humans, they can't be trusted, i.e. WikiLeak, leaker for instance].

Next ALL the data is stored within in the sphere. But to access the data on the outer side of the sphere, picture Earth's surface, the ball or sphere (with grids like a map of the globe) rolls around on a giant grid paper. When you want to look at a particular event, person, subject, or whatever, a particular point on the sphere's grid touches a corresponding point on the grid paper it rolls on, the grid paper it rolls on can wrap around and morph itself to the sphere or contour itself so the next corresponding piece of information on the surface can be accessed, rolling or spinning.

Picture a selectric typewriter ball on a shaft as a 2D model to consider this, now make it all 3D in your mind, and the paper molds around the sphere as it accesses, or in the case of a selectric typewriter it types. Now the Sphere is hollow inside containing layers, just like the earth, crust, mantel, and core. Information goes deep or across, every piece of information is connected, think about the earliest string theory models for this.

Great thing about my visualization concept is I believe all this math exists, even though in reality string theory is mostly bunk, but the math to get there makes this possible. As the information goes deep, think about the iPad touch screen, or the Microsoft restaurant "menu on a table" concept, or the depictions of Minority Reports, moving of the screens by way of motion gestures, I believe Lockheed also has this concept up and running for air-traffic control systems, prototype versions, perhaps the military is already using it, as it has massive applications for the net centric battlespace visualization too.

Okay so, some levels go through a frame-burst scenario taking you into another level, where the data generally stored at the almost infinite number of grid points and cross connected to every other is nothing more than a nucleus with additional data spinning around it. But the user cannot access all that information, without clearances, the AI system has access to all of it, while a sorting system is a series of search features within search features, with non-linked data also. You can't break into it; it's not connected to the users' interface at all, think of the hidden data as electrons unattached around the data. The data is known to exist but cannot be accessed that would be the 5D level, and 6D level no human may get too, but the data exists.

You know that surfer dude in Hawaii that came up with the "Grand Theory of the Universe" why not use his model for our visualization, in spherical form, again, the mathematics for all this already exists.

You see, what I need is a way to find people like me, I want to find these thinkers and innovators to take it all to the next level, and if the visualization is there, we can find; The Good Guys, Bad Guys, and the Future all at once. Why do I want a "Neural Network" visualization system in a sphere? It seems to me that this is how the brain does things, and what we are doing here is creating a Collective Brain, using each individual assigned to an "ever-expanding" unit of data, along a carrier or flow.

Remember when Microsoft Labs came out with that really cool way to travel through the Universe and look at all the celestial bodies along the way, using all the Hubble Pictures collected? It's kind of like that, you travel to the information, discover as you travel and it piques your curiosity as you go triggering your own brain waves, and splashing the users minds with chemical rewards as they go, as they discover more information, expanding their understanding as well, it just seems to me this is how it all works anyway.

Think of that old Sci Fiction concept where the Earth and our solar system are merely an atom of a chemical compound within a cell of the human body, all we can see is all the other compounds around us because everything is so small, thus, we cannot see the whole picture and what appears to be an entire universe would only be a few thousand cells close enough for us to see. And time itself is slow, as the electrons or planets moving around the atom appears to take a year to circle the nucleus instead of 10,000 times a second.

So, combining all these types of thoughts, this is how I envision how the future visualization tools would work.

Now then, using the whole concept of connecting the dots for information or even building an AI search feature scouring the system at speeds of terabytes a second, the AI computer can become the innovator, thanks to the user asking the question, and all the neurons (individual humans) with all their data putting in the information. You just need the best questions, you get instance answers.

Okay so, take this concept one step further; the AI super computer's operation is a "brain wave" and that brain wave is assigned a number, you can have as many brain waves, as the internet has IP addresses, with whatever scheme for that you choose. And your query can search the former queries too. The user's questions are as important as the data itself.

Thus, it helps us find the innovators, the question askers, once we know that, we have the opportunity for unlimited instant knowledge. Data visualization can take us there, and it removes all the fog of uncertainty, and answers most all the questions we could ever hope to ask, and comes up with its own questions as well. Does this make sense?

This is the type of visualization I need to faster access information, and I can solve all the problems, even the ones humans refuse to solve, or doom themselves to repeateth. That's my preliminary thought on this - may we start such a dialogue on the topic? If so, email me, and I hope you enjoyed today's dialogue?


Source: http://ezinearticles.com/?Data-Mining,-Visual-Analytics,-and-The-Human-Component!&id=4817019

Tuesday, 28 May 2013

Limitations and Challenges in Effective Web Data Mining

Web data mining and data collection is critical process for many business and market research firms today. Conventional Web data mining techniques involve search engines like Google, Yahoo, AOL, etc and keyword, directory and topic-based searches. Since the Web's existing structure cannot provide high-quality, definite and intelligent information, systematic web data mining may help you get desired business intelligence and relevant data.

Factors that affect the effectiveness of keyword-based searches include:
• Use of general or broad keywords on search engines result in millions of web pages, many of which are totally irrelevant.
• Similar or multi-variant keyword semantics my return ambiguous results. For an instant word panther could be an animal, sports accessory or movie name.
• It is quite possible that you may miss many highly relevant web pages that do not directly include the searched keyword.

The most important factor that prohibits deep web access is the effectiveness of search engine crawlers. Modern search engine crawlers or bot can not access the entire web due to bandwidth limitations. There are thousands of internet databases that can offer high-quality, editor scanned and well-maintained information, but are not accessed by the crawlers.

Almost all search engines have limited options for keyword query combination. For example Google and Yahoo provide option like phrase match or exact match to limit search results. It demands for more efforts and time to get most relevant information. Since human behavior and choices change over time, a web page needs to be updated more frequently to reflect these trends. Also, there is limited space for multi-dimensional web data mining since existing information search rely heavily on keyword-based indices, not the real data.

Above mentioned limitations and challenges have resulted in a quest for efficiently and effectively discover and use Web resources. Send us any of your queries regarding Web Data mining processes to explore the topic in more detail.

Source: http://ezinearticles.com/?Limitations-and-Challenges-in-Effective-Web-Data-Mining&id=5012994

Saturday, 25 May 2013

Find New and Used Cars for Sale or Sell Your Own

CarsDirect.com is a nationwide resource for new and used cars for sale. Our huge network of new and used car dealers, as well as private sellers, throughout the country makes it easy for you to find exactly the vehicle you want and then pick it up at a convenient location.

CarsDirect has thousands of new car dealers in our nationwide dealer network, so there's a good chance we'll find you the vehicle you're looking for at a dealership near you. In most areas, we offer you two options for buying a new car. You can either work with one of our Vehicle Specialists and get the CarsDirect guaranteed low price for your vehicle, or you can get a free quote from one of the premier dealers in our network. Either way, you will experience an easy and hassle-free way to buy a new car.

We also have a nationwide database of used cars for sale. Our giant directory of used vehicles includes listings from both private sellers and dealerships. And our handy Advanced Search function allows you to narrow down your list of cars for sale according to price, year and distance from your zip code. Additionally, we update our list of cars for sale daily to ensure it is accurate and up-to-date.

If you have a car of your own to sell, you'll find it's both easy and affordable to list it for sale on CarsDirect. We offer three listing packages. Our most affordable listings cost $25 and include premium placements for 30 days and one photo. Our Run 'til It Sells listings look the same but offer unlimited renewals for only $5 more, and our Combo Package includes unlimited renewals plus up to six photos for $37.50.

Whether you're in search of cars for sale or trying to sell your own, CarsDirect.com has you covered.

Source: http://www.carsdirect.com/help-center/buying-advice/find-new-used-cars-for-sale

Friday, 17 May 2013

New Software Detects Bots Scraping Website Data

IDG News Service — Web sites such as job boards face a persistent problem: their data is constantly pilfered by automated bots.

The data ends up on other competing job boards, which have stolen the content. It's a problem that plagues any Web site whose intellectual property must be publicly posted for free, or even those with subscription models.

But an Atlanta-based security company that specializes in detecting bots has developed software that can detect those screen-scraping and data-mining bots.

Pramana's main product, HumanPresent, detects automated bots that, for example, enter spam into Web-based forms or register for free e-mail accounts to be used for spam.

Pramana has now developed a module called "data mining and screen scraping prevention" for HumanPresent. It works on many of the same principles as its main product but has been modified for data-mining scenarios, said David Crowder, Pramana's CEO.

HumanPresent can detect bots by noticing differences in the way a human would normally interact with a Web page and contrasting that with how bots behave. It looks at more than 30 metrics, such as keyboard strokes, mouse clicks and the timing of those actions.

HumanPresent looks at single transactions, but the data-mining module has been modified to look at a timed period when either a bot or human is on the site, Crowder said.

Data-mining bots tend to entirely circumvent a browser's user interface. For example, a bot may request a Web page with lots and lots of data, but never scrolls or clicks on a page. If a series of pages are opened and viewed in that manner, it could mean a data-mining bot has arrived.

Pramana assigns a unique ID to the visitor, and after analyzing the visitor's behavior, can make a decision whether to label the visitor a bot or not. There are several different ways a Web site operator can then choose to deal with the situation.

The IP (Internet Protocol) address of the bot's computer can be block permanently. One car auction Web site that is testing Pramana's data mining module decided to move suspected bots into a "sandbox" where it is served completely false data.

"They're indeed data mining -- it's just dead wrong," Crowder said.

Other options include prompting the Web site visitor with a challenge or task, which some bots aren't capable of completing.

Data mining costs companies dearly. Companies that sell premium data will find that their competitors will buy a subscription and then use automated bots to steal the data for their own sites. In one example, a Web site that has gigabytes of data on used car prices found their data had been scraped and was for sale on eBay.

"They are actually competing with their own content," Crowder said.

Some Web sites have poor designs that make data scraping that much easier. The used car site had URLs (Uniform Resource Locators) could be sequentially modified to reveal more data, Crowder said.

The data-mining module will be wrapped into the HumanPresent product for now, but early next year Pramana plans to sell it separately, Crowder said. Pramana offers HumanPresent either as an on-premise appliance or as a software-as-as-service configuration.

For the SaaS (software as a service) offering, Pramana's technology is integrated into a Web application and session information is sent back to Pramana for analysis. Crowder said Pramana has been able to significantly cut down on the latency time in its latest version. For customers who need more speed, the appliance is available.

Source: http://www.cio.com/article/506726/New_Software_Detects_Bots_Scraping_Website_Data?page=2&taxonomyId=3089

Friday, 3 May 2013

Bernie Caps Auto Sales INC Selects Carsforsale.com® to Develop Dealer Marketing Solutions

Carsforsale.com has been selected by Bernie Caps Auto Sales INC, a Old Forge, Pennsylvania auto dealership, to help develop and optimize its dealer marketing solutions. Carsforsale.com has developed the online image, through web design, graphic design and web development, to increase its brand visibility, website traffic and build new customer relationships.

Bernie Caps Auto Sales INC in Old Forge also has a vast array of marketing solutions offered to them via the Carsforsale.com dealer system. The dealer system offers social media tools, template posting systems, Web positioning and search engine optimization, wholesale networking, customer relationship tools, data reporting and more.

In business since 1999, Carsforsale.com offers great opportunity to grow Bernie Caps Auto Sales INC's sales strategies, with millions of visitors viewing their inventory portal each month. This paired with dealer inventory postings on Scranton, Pennsylvania FreeClassifieds.com, an exclusive automotive partnership with Carsforsale.com, extends the dealer reach into further online consumer resources.

Sean Coffman of Carsforsale.com states, We pride ourselves in offering simple, powerful and effective marketing solutions to dealers throughout the nation and we are excited to add Bernie Caps Auto Sales INC to our dealer family.

About Bernie Caps Auto Sales INC:
Bernie Caps Auto Sales INC is located in Old Forge, Pennsylvania. Bernie Caps Auto Sales INC's inventory can be found on Carsforsale.com.

About Carsforsale.com:
Carsforsale.com created in 1999 and headquartered in Sioux Falls, SD, is one of the fastest-growing and most popular auto classified websites. Carsforsale.com offers a fast and effective way to connect buyers with sellers of used cars. Carsforsale.com reaches millions of unique visitors each month and is a privately held company.

Source: http://mediadesigner.digitalmedianet.com/article/Bernie-Caps-Auto-Sales-INC-Selects-Carsforsalecom174-to-Develop-Dealer-Marketing-Solutions-2507286