How&nbsp;<a href="https://www.toxicnatos.com/">ToxicNATOs</a>&nbsp;keeps an eye on its Competition.

ToxicNATOs is a small veteran-owned watch strap brand based in Colorado Springs, Co. They offer nylon, seatbelt, and leather NATO-styled straps to a small but fiercely loyal customer base. Being in a crowded market, they find it best to stay on top of their competitor brands&#39; pricing for like-kind products on the various marketplaces like Amazon and Walmart as well as niche retailer websites.

Using ToxicNATO-provided product inputs, FrigginYeah leveraged existing price monitoring integrations to price scrape various competitive brands and provide email alerts showing brands competitive to ToxicNATOS. These reports have helped optimize their product pricing as well as help identify new product opportunities.

&ldquo;It was awesome to start seeing competitive pricing data as it really allowed me to identify some areas of improvement in our catalog,&rdquo; said Terry, the owner/operator of ToxicNATOS. &ldquo;We are really looking forward to using FrigginYeah to monitor our product&#39;s <a href="https://frigginyeah.com/map-violation-monitoring">Minimum Advertised Price (MAP)</a> as we expand our distribution channels.&rdquo;

<a href="https://www.toxicnatos.com/">https://www.toxicnatos.com/</a>

ToxicNATOs, a veteran-owned watch strap brand, effectively monitors competitor pricing. With FrigginYeah's advanced price monitoring.

ToxicNatos keeps an eye on its Competition

toxicnatos keeps an eye on its competition

Why You Should Care About MAP Violations

If you&rsquo;re reading this you&rsquo;ve probably worked hard to develop a reputable online brand. Your brand has perceived customer value and you may have spent countless hours on maintenance in order to ensure balanced growth. But if you aren&rsquo;t supervising minimum advertised price (MAP) violations you are susceptible to losing massive revenue streams to unauthorized sellers, brand erosion, and perceived product value loss.

What is Minimum Advertised Price

Simply put, minimum advertised price or MAP is the lowest price a seller can advertise a good. However, minimum advertised pricing only relates to advertising and does not reflect the price at which a retailer can sell said item in their store or online shopping cart. MAP and is a form of manufacturer brand protection.

Imagine a product that has a manufacturer suggest retail price or MSRP of $500. But through an online reseller it is being advertised at $250. Your potential customers may start to think your prices are inflated. If, so they may refuse to pay MSRP for your product again. This perceived product value is representative of your brand&rsquo;s strength. If your customers will no longer pay MSRP your brand will erode.

What is a MAP Violation

A map violation is a breach in the minimum advertised price agreement between the manufacturer and reseller. The above examples demonstrate a classic MAP violation When MAP agreements are breeched your brand will be left weakened or devastated. Breaches regularly occur, especially within Amazon. This is due to Amazon not interfering with their seller&rsquo;s pricing policies and even Amazon breaking your policy directly. Unfortunately, pinpointing the sellers who are in violation and shutting them down is not an easy task. It is important that you use the correct tools to help save you time and make your case to Amazon or directly to your Sellers. It is imperative that you create MAP agreements to protect your brand.

How to Protect Your Brand

There are many eCommerce outlets that violate MAP agreements constantly. And of course, they will not take action because they do not want to disrupt their revenue streams. Monitoring your brand&rsquo;s presence on all of these outlets simultaneously requires a ton of time and effort. Fortunately, there are companies that have developed powerful software that mitigate this issue. FrigginYeah&rsquo;s platform allows users to consolidate this process and quickly monitor your product prices across whichever marketplaces or retailer you choose. This helps pinpoint and target MAP violations and shut them down. Saving you countless hours and potential revenue.

MAP Agreements

To protect your brand from any future MAP violations it is crucial that you create a MAP agreement with resellers. A MAP agreement will promote fair competition across distribution channels, defend brand identity and value, and protect your margins. Smaller retailers will also be in favor of MAP agreements because it allows them to compete fairly with larger brands.

Ultimately, MAP violation enforcement and monitoring should be taken seriously. It is important to arm your brand with the correct tools to protect, enhance, and repair your revenue streams now and in the future.

Protect your brand and revenue with MAP enforcement: avoid brand erosion and ensure fair pricing with strategic monitoring and agreements.

Why You Should Care About MAP Violations

why you should care about map violations

Continuing from my previous posts on Web Data Extraction, I&rsquo;d like to explore what I see as the main cost drivers for every Web Data Extraction project. These drivers are relevant both in a Data-as-a-Service application and a Platform play, however there are a few additional factors that come into the picture when purchasing a platform. Each of these add multiples to the cost of delivering a project, so be measured when considering what you really &ldquo;need&rdquo; versus what your available budget is.

Volume&ndash; How much data is expected? Is this millions of rows? Hundreds of thousands? Tens of thousands?

The volume of a web scraping project will be a huge physical cost driver. The higher the volume of data, the more careful a provider will need to be about how they go about harvesting. They may need to bring to bear 3rd party services, like a Proxy Provider, or leverage a more expensive IP set. Also, depending on your Frequency, they also may need to bring more infrastructure online to support your extraction. These are both challenges you would need to overcome if this project is running as a platform in your enterprise. Why? Simple. The larger your footprint on the source, the more attention you draw. The more attention you draw, the more likely they are to block you. Once you are blocked, you can&rsquo;t access the data you need without going about it in a whole new way. There are other implications that come with this as well, but that is an entirely different post!

Finally, the business of web harvesting can be murky on occasion, but this is an aspect of it that should not be. It is pretty easy to estimate the amount of data that the project hopes to produce. If you have a high quantity of sources, take a sample of them and explore the amount of data that might exist on each site. Then, multiply that across your full site list. Or, if you have just one or two high volume sources, it is more than likely that they will display the number of records that exist. If not, just take the number of records on a page and multiply that by the number of pages, or whatever similar logic exists based on the page structure.

Frequency&ndash; Will this be an every minute, hourly, daily, weekly, or monthly harvest?

Yet another large cost driver is the frequency with which sources need to be scraped. This creates a multiple when applied to either Volume or Quantity, or both! Much like volume, how frequently you extract from a source increases the likelihood of detection and blocking by that source. Plus, the more frequent an extraction, usually the more hardware it will take. So finding a sweet spot between getting the data you need and leaving a large footprint or having servers up the wazoo is key.

Quantity&ndash; How many sites are being targeted? Is it just one? Tens? Hundreds?

This is a no brainer! The more sources that are being targeted, the more it is going to cost. Why? Well, depending on the types of sources and the structure of the data, there is a very high likelihood that each source will need some extra level of customization. When talking about customization you have to think labor, and not just labor but skilled labor. I had a customer come to me not too long ago who wanted to extract data from 17,000 sources and only had $10K to spend&hellip; which basically means he was willing to spend $0.58 a source.&nbsp;Assuming only light customization is needed, say 15 minutes per source, the provider would spend roughly a little over 2-People-Years, or 4,250 hours, of effort to bring those 17,000 sources online&hellip; and this does not even address the low-quality data you should expect out of a system like this.&nbsp;&nbsp;So that shows you how much he valued the data versus what the actual cost to get that data would be.

Long story really short, we had to develop a new way to attack the same problem that did not take 15 years to pay back his original investment.

Final thought

Too keep costs down, no matter if you are building a prototype or an MVP, consider how you can do more with less. Can you do this with a small amount of data to start with? If so, then think about hiring an individual off of Upwork, or the like, for a one-time extraction&hellip; even some smaller DaaS shops will pick up one-time work if the price is right. Once you have what you need and have proved out your product, move on to a more robust solution. The same goes for Frequency and Quantity. If your use-case calls for daily, try to get by with weekly or even less frequently until you prove out your product. On the flip side, if you want to build 17,000 sites, maybe you can bite off a smaller amount until the value prop is proven. Think, is there a 20/80? Or maybe a 5/95? Or even broaden your horizons, maybe this data exists in a smaller field of sources&hellip;

Key cost drivers in Web Data Scraping: Volume, Frequency, and Quantity impact project costs. Strategic planning and scaling.

Top 3 Cost Drivers For Every Web Scraping Project

top 3 cost drivers for every web scraping project

Well it was a wrap last week and just now getting time to post about it.

This was our first event exhibiting our new price monitoring product &ndash; It was also my first event exhibiting ever. I have attended a few conferences, absentmindedly, over the years and thought to myself going into this &ldquo;How hard can it be?&rdquo;

Now as I sit down to net it all out I just have to say I have left Shop.Org with a new found respect for the logistics, sales, and marketing skills that the folks who work these events have.

A couple of my key takeaways were:

1) Proper Prior Planning &ndash; This goes from understanding the event layout and where you are located, to local services support, to showing up and setting up on the first day available (definitely not on the last day!), to product pitch prep, to taking advantage of all the great marketing/media potential that comes with this type of event. We experienced some self-inflicted wounds as well as some from the event support. However most, if not all, of those could have been prevented with proper prior planning. You can wing a lot in life but conference exhibiting I have learned is not one of those.

2) Know your event &ndash;

<ul>
	<li>We have a product that is purpose built with Brands in mind but there are a lot of analogies between Retailers and Brands when price tracking comes into play &ndash; its the same data after all. So tailoring our message from a Brand being able to price monitor for minimum advertised price violations or buybox ownership to that of a Retailer looking to do competitive price intelligence or helped us have more productive conversations.</li>
	<li>Also in this category is having an understanding of event turn-outs. As far as attendees go I was pretty disappointed in the overall amount of foot traffic for the exhibitors&hellip; and I was not alone. Ultimately we had some great quality conversations and I am sure one or two will convert, but I expected to talk to (or at least have the opportunity to) 5x the amount of people based on the event attendee forecasts. Beyond attendee forecasts how the event was structured really meant folks had to have some predefined interest in your product and peel away, not a lot of random foot traffic (lesson learned around pre-marketing for an event). We will probably still go to&nbsp;<a href="http://shop.org/">Shop.Org</a>&nbsp;2018 but it will be with eyes wide open!</li>
</ul>

3) Network, Network, Network &ndash; This may seem obvious, but sometimes theres a gap between whats obvious and how you execute. I was impressed with a lot of people that I met on the floor and its infectious to be around folks that are out in front.

4) Presentation &ndash; while there were a great number of awesome sales and marketing folks at the event. I was put off with how many folks working their booths spent the majority of time glued to their cell phone or laptop. One booth was so bad that all their folks were huddled around the bar top table in a semi-circle with their backs to the floor and I watched at least one interested party walk away when they were not acknowledged. I am glad to say that this was not a problem at Booth #2112!

All-in-all this was a great new experience for me and a little outside of my comfort zone -but that&rsquo;s exactly how it should be. Unfortunately we are little late for the Big NRF event in January (not to mention its on the tail end of my Honeymoon), but ShopTalk 2018 and IRCE 2018 here we come!

About FrigginYeah:

FrigginYeah&rsquo;s PricingIQ is a Data-as-a-Service (DaaS) platform that provides Brands and Retailers with a reliable, high-quality, and scalable brand, product content and price monitoring solution. PricingIQ is the ideal solution when trying to identify BuyBox ownership for marketplaces, dynamically identifying/tracking product catalogue changes, alerting for Minimum Advertised Price (MAP) violations, product catalogue quality scrubs, product variant (sizing, colors, etc.) tracking, and full exposure to all 3rd&nbsp;party Marketplace sellers. Our constantly expanding library of over than 150 retailer sites provide comprehensive coverage of the top online US Ecommerce retailers as well as Canada, United Kingdom, France, and Italy.

Reflecting on our first exhibit with FrigginYeah: Key lessons on planning, knowing your event, and networking. MAP Monitoring was a hot topic.

Shop.Org – It’s a wrap! 

shop.org – it’s a wrap! 

Continuing on my previous post, &ldquo;You Have a Product Idea That Includes Extracted Web Data&hellip; Now What?&ldquo; (<a href="http://tinyurl.com/pr9x5vg">http://tinyurl.com/pr9x5vg</a>), I want to move on to explore the four main ways with which you can acquire the data necessary to fuel your product.

Much like every situation in business, not all solutions for a given problem are the right ones. What you need today for a product prototype may be drastically different in terms of time and dollars invested than what you may need when your product goes live (or if you&rsquo;re lucky, not until your product is realizing some returns on your investment). It could even be that your idea cannot stand up for itself unless you go &ldquo;all-in&rdquo;, which makes what you want to do even riskier.

Below I have listed the four main ways that you can go about obtaining the web data necessary to fuel your product. How you move forward is dependent on many factors, including where you are at with your product, the total $&rsquo;s you have to invest, the amount of control and predictability you may want, and in some cases, the desire to focus on your core product while allowing others to do the commodity work of web harvesting.

Programmer&nbsp;($ &ndash; $$$) &ndash; C# / Perl / Python / PHP, and what have you to build scripts accessing source content. Your primary costs here are wages and whatever you need to keep your infrastructure up and humming along.

Pros&nbsp;

<ul>
	<li>You control your sources.</li>
	<li>Extracting from new sources can be as easy as available keyboard time.</li>
	<li>The build and maintenance process is more transparent. (This does not denote easier)</li>
	<li>Everything is in-house.</li>
</ul>

Cons&nbsp;

<ul>
	<li>Will not leverage the power of Machine Learning (ML) algorithms that some software tools have built in that can increase the resilience of an extraction.</li>
	<li>When a script breaks, more than likely hours will be spent debugging it.</li>
	<li>The likelihood of investing similar amounts of time fixing a broken script as the original build is high.</li>
	<li>Enterprise features such as scheduling, data cleaning, data transport, throttling, anonymization (proxy services), etc. will need to be cobbled together, where possible, to create a true data pipe from the web.</li>
</ul>

_

DaaS&nbsp;($$ &ndash; $$$$) &ndash; No muss, no fuss. Tell them what you want, when you want it, and in what structure; then go back to building your core product and wait for the data to start coming in. Sounds great, no? Well it&rsquo;s not always that rosy&mdash;it is web data after all&mdash;but it sure beats doing it yourself when you can afford a reputable DaaS provider. &ldquo;No Names&rdquo;, BPOs, and software companies that offer up their service teams As alluded to, this one can be all over the map; not just in regard to pricing but also in regard to reputable providers and data quality. &nbsp;Be mindful that you will truly get what you pay for when going the DaaS route.

Pros&nbsp;

<ul>
	<li>Turn on the data spigot and just sit back with your feet up.</li>
	<li>Someone else has the problem of building and maintaining access.</li>
	<li>Normally higher-end services will offer a consolidated data schema (e.g. all data from all sources will be in one format and similar data types).</li>
	<li>Staff only what you need to build your product, and you can forget about experts in web technologies, C#, JavaScript, Ajax, PHP, etc&hellip;</li>
</ul>

Cons&nbsp;

<ul>
	<li>You are not their only customer, so if &ldquo;it&rdquo; hits the proverbial fan then it may take some time to get the spigot flowing again.</li>
	<li>In a lot of cases your service is only as good as your SLA, so build in a good one!</li>
	<li>Hard to find a real, true proactive service when problems arise.</li>
	<li>No real transparency into issues, what caused them, or how they are rectified. Let&rsquo;s face it, this is still a dark art in some respects, and when things break it is usually easier to fix them then to figure out why they broke. So more often than not, you will be left with few answers and that is definitely frustrating.</li>
</ul>

_

Software&nbsp;($$ &ndash; $$$$) &ndash; Don&rsquo;t want to deal with building a fragile, cobbled-together infrastructure based solely on a programmer&rsquo;s discretion that may or may not outlast their tenure, (breathe) but want to empower your team with a tool that potentially allows the power of markup, automation, ML, enterprise scheduling, data typing, error handling, throttling, reusability, proxy services, and survivability? Then purchasing a software tool may just be the route you need to go&mdash;but don&rsquo;t expect the lightweight tools like Visual Web Ripper ($) to have the same suite of tools as a #Kapow or #Connotate.

Pros&nbsp;

<ul>
	<li>No matter where you fall in the spend spectrum, you are more than likely buying a tool that has had hundreds of thousands of hours of developer seat time.</li>
	<li>Will bring to bear advance functions like; API Connectivity, Enterprise Scheduling, Data Transformations, Error Notification, Logins, CAPTCHA Breaking Connectors, Throttling (speed up or slow down connection requests) , Anonymization Proxy Services (Think: Hide me!)</li>
	<li>Ease of Use, which requires either a lower programing skill set or none at all. But be warned, this not always a good thing!</li>
	<li>Repeatability</li>
	<li>Vendor Support: a lot of vendors in the space provide some level of no-cost support either by way of their support desk, discussion boards, or even Webinars on how to use their tools in different use-cases or against different web technologies. #Import.IO is not only one of the youngest platforms out there, but they are also one of the best in the business when it comes to supporting their user base.</li>
</ul>

Cons&nbsp;

<ul>
	<li>How much of the web do they purport to cover? Can they verify that claim? But more importantly, can they demonstrate the coverage you need for your use-case? I can&rsquo;t stress this enough: if you are buying an enterprise piece of software that will cost you $$$$ then run them through their paces. See the sites, functionality, or target source technologies in action. Ask to speak to current references who have similarities to what you want to do. Don&rsquo;t take a content aggregator reference when you are looking to build an on-demand application.</li>
	<li>Why is this a con when buying an enterprise platform? Because it&rsquo;s a risk. As much as I&rsquo;d like to sit here and tell you that web data harvesting is a science, I can&rsquo;t. Sure, it&rsquo;s based on extracting programming code, but the sheer plethora of ways information is bolted together moves this from the science realm and plants it firmly into one of Art.</li>
	<li>Support for un-harvestable, one-off sources lives in the world of &ldquo;it&rsquo;s on the road map&rdquo;&hellip;a road map that, odds have it, will be completely new, and conspicuously different, from the last time you saw it. On that same point, be sure to understand what their release schedule looks like. No harm in asking for a snapshot of the previous year or two highlighting major and minor releases with dates.</li>
</ul>

_

Licensing&nbsp;($$$$) &ndash; drinking right from the source may be just the data manna your product needs to solidify its future. This is the most costly course, but it is also the one that has the biggest rubber stamp of approval on it. That might not be all that big of a deal when you are targeting hundreds of sources, but if you are relying on one or two streams of data, this is the only way to build a viable long-term product.

Pros&nbsp;

<ul>
	<li>If you have a limited number of sources, this is option that is least likely to have the top of your funnel shut down. Huge!</li>
	<li>There could potentially be partnership opportunities with the source(s), providing a little more street cred to your product idea.</li>
</ul>

Cons&nbsp;

<ul>
	<li>Costs! Depending on the source, and where their data comes from, you could be in for a rude awakening when you hear their price tag.</li>
	<li>Quality: API data may not be the same quality as what they present on the web. This could be due to variations in structure, the data elements, or even the cleanliness of the data.</li>
	<li>Terms of Use: There will be restrictions on how, where, and with whom you can use the data&hellip;sometimes even more restrictive than what they present on their site.</li>
	<li>Throttling: They will control the spigot and deliver on a frequency that is best for them.</li>
	<li>Potentially competitive situation. They might respond to your request, post your conversation with &ldquo;we already have something like this in the works&rdquo; or something similar to that tune. Or even worse yet, they leapfrog you and go further up the value chain to completely negate any value you were going to provide in the first place</li>
</ul>

Series Note: This is the second post in a series of publications on web data extraction. The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions feel free to contact me&nbsp;<a href="mailto:tom@frigginyeah.com">tom@frigginyeah.com</a>&nbsp;or visit my website&nbsp;<a href="https://frigginyeah.com/">https://frigginyeah.com</a>

Choose the right data acquisition method, Programmer, DaaS, Software, or Licensing, to power your product effectively. 

Programmer, Data as a Service, Software, or Licensing… what is right for my Web Data based product?

programmer, data as a service, software, or licensing… what is right for my web data based product?

Hard to notice but Amazon and Office Depot were about the same pricing for days until Walmart breached that price level and as you can see Amazon started pegging its price with Walmart&rsquo;s.

We are price monitoring school products and seeing Amazon pegging its price to Walmart. It has always been interesting doing this for our customers, but now price tracking competitor prices for our own purposes has added a new angle at how we are viewing the challenge. This also has us cooking up a new internal project to start monitoring these price movements at a more substantial level.

Amazon and Walmart Price Tracking

amazon and walmart price tracking

In the process of doing MAP violation tracking for our customers we also provide all the data for products that we find a MAP violation for or not. This is the result of MAP violation monitoring being a function of price monitoring in general. What that allows our users to do is peel back more layers on changes they might be making on a marketplace (like Amazon, Walmart, Home Depot) or on their retailers (Wayfair, Hayneedle, Target, etc).

For the example I looked at one product, of several thousands, we track for one of our clients over a ~30 day time span and there are 18 different price changes, 10 by Walmart and eight by Amazon. Six of those days Amazon had the premium price whereas the bulk of the days that was held by Walmart. However, it is slightly entertaining to see that pricing on Amazon would drop by a mere penny from day-t0-day to stay in front pricing on Walmart. This actually happened over and 11 day span before both were advertising a new higher price that maintained a price parity with each other.

A couple of side notes:

<ul>
	<li>None of this pricing action actually violated our client&rsquo;s MAP policies with either marketplace. In fact they do not have MAP set for this product as you might have noticed that &ldquo;US&rdquo; and &ldquo;CA&rdquo; are grayed out, these selectors allow a user to overlap the MAP price with extracted data.</li>
	<li>Both data points on this price chart rely on Amazon and Walmart as a retailer and does not include marketplace sellers. Those sellers are price monitored as well, but are displayed slightly different. Like so:</li>
	<li>Price tracking websites like Amazon, Walmart, or Wayfair are not always about some shadowy competitor price tracking another competitor in some questionably dubious price fixing scheme. This use-case is a perfect example where a Brand is able to view their advertised product pricing on retailers and make correlations between changes they make behind the scenes on their side and how it impacts the retailers and marketplaces they sell on.</li>
	<li>Walmart and Amazon price monitoring is not easy, but finding the right partner can make it seem so at times.</li>
</ul>

Discover how MAP violation tracking offers in-depth product pricing insights across marketplaces like Amazon, Walmart, and Wayfair.

Amazon & Walmart Price Monitoring

amazon & walmart price monitoring

You have struggled with the pain of a problem ripe to be solved, or you have identified an opportunity that needs to be seized and have now conceptualized the surefire way to solve it. There is just one problem: that idea will require you to utilize data from the Web in combination with your smarts to make it valuable.

Let&rsquo;s run down the list of terms that you&rsquo;ve Googled up until this point:&nbsp;Web Scraping,&nbsp;Screen Scraping,&nbsp;Web Crawling,&nbsp;Web Extraction,&nbsp;and if you really want to church it up,&nbsp;Web Harvesting. They all really mean the same thing, which is targeting a source of data on the web, capturing it by some means, and returning it to you in a form or fashion that is acceptable. Sure, each of the tools and services that exist out there go about it in different ways that span the spectrum from brute force to intelligent precision extraction, but at the end of the day, all that should matter is that you are receiving high-quality, accurate data on the frequency and schedule you expect, and, if it is mission critical, that the company/programmer/source does not go&nbsp;*poof*&nbsp;in the night.

These &ldquo;Top 3&rdquo; considerations I have found to be critical when assessing product ideas and their risks before engaging a vendor, bringing in your team, or buying software.

Creating Your Extraction Profile&nbsp;

1)&nbsp;# of Sources&nbsp;&ndash; I will just get in front of this at the get-go. If you are planning to extract data from only one source, then your project will be in the high risk category. Some of the riskiest choices are aggregators like Expedia, Kayak, Shopzilla, and Pronto, and moving sources like Zillow, Trulia, Autotrader, Cars.com, Yelp, and Craigslist. All these sources make their money based upon their data, usually pay-for-use and advertising $&rsquo;s (sometimes they&rsquo;ll even license their data set), so they spend a lot of money to acquire and ultimately protect that data from people of our ilk. Therefore it is a risky proposition to build a viable long-term product based on data from these sources. (Read: proceed with caution.)

On the flip side, if you need hundreds or thousands of sources before your product is viable then you might just be on to something. The more sources = the higher barrier to entry for new competition&hellip; of course that means a longer runway before you product hits its MVP, maybe. My suggestion here is to survey the list of sources and stack rank them based on ROI (think: the # of records available, or even the quality of content) and come up with your 80/20, or in this case in this case the 20/80. Can you come up with 20% of the sources that will give you 80% of the data functionality you need?

When moving the scale from 1-to-100&rsquo;s, building the extraction from these sources stops being viable for a programming team and starts being a candidate to bring automation to bear. This decision should not be made in a vacuum, though, and it should include the frequency, the number of sources, their build times, break rates, and fix times &ndash; all of which I will cover in a future post.

2) Frequency &ndash; I will shamelessly proclaim that I have helped build every Web Data project variation that has ever existed, and a key component to understand before moving forward is how frequently you need the data in order to make your product idea valuable. While it might seem obvious, I will state it anyway: what it takes to extract data from 100 sources once a month can take drastically different resources than extracting from one source near-real-time on-demand. Resources can be several things: people, hardware, software, 3rd party support services, bandwidth, etc&hellip; I recommend mapping out exactly what you need to make your product viable and then build your tiers of value:frequency:costs from there.

3)&nbsp;Amount of Data&nbsp;&ndash; Finally some serious thought should be logged on the amount of data you will be extracting from every source. You should have a good idea upfront what you should expect from each source. Not only will this go a long way into projecting storage and DB configurations, it will also allow you to establish a baseline, troubleshoot if your data rates start falling below a set threshold, and even start calculating how much you can expand with your current processing power.

Bonus:&nbsp;When thinking about the&nbsp;Frequency&nbsp;and the&nbsp;Amount of Data&nbsp;you plan on extracting, just note as the time frequency increases and the amount of data you need to download continues to expand, footprint will only grow that much larger and with that you risk being identified and shut down by the source.

Finally, before moving forward with attempting to engage a vendor or purchase software, knowing your&nbsp;Extraction Profile&nbsp;will only save you time and money. If you don&rsquo;t need to build a clone of the internet, don&rsquo;t, Start small, prove your product, and expand organically.

Series Note: This is the first in a series of publications on web data extraction. The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions feel free to contact me tom@frigginyeah.com or visit my website <a href="https://frigginyeah.com/">Frigginyeah</a>

Harness web data effectively: Avoid relying on risky single sources, plan data frequency, and consider data volume. Start small and scale wisely.

You Have a Product Idea that includes extracted Web Data… Now what?

you have a product idea that includes extracted web data… now what?

Whew! It&rsquo;s been awhile &amp; loads of good things have happened over the past few months! Lots of folks needing all kinds of help with their web data solutions. With projects ranging across the board from workflow automation challenges, to product pricing intelligence platforms, and even some higher level data curation and analytical components. So needless to say, but I will any who, it&rsquo;s been fun in exciting times and with that in mind I bring you this new short blog posting.

In every space there are a few terms that are bandied about, and the world of Web Scraping is no exception. You frequently hear people chirp about their&nbsp;Scale&nbsp;and that their solution is&nbsp;Resilient&nbsp;or that there is some level of&nbsp;Intelligent&nbsp;detection and processing, but what exactly is this and does it work&hellip; or better yet, does it matter? Unfortunately the answer to all these questions is &ldquo;&hellip;it depends.&rdquo; There is just not a one-size fit all approach when it comes to buying software, data, or a service.

What does scale mean?

I won&rsquo;t sit here and posit that all platform companies (enterprise, cloud, desktop) out there are in the business of pulling the wool over your eyes, but they will definitely make truthful claims that have many meanings. &ldquo;Oh yes we can scale&hellip;&rdquo; can mean your configuration (license, servers, network) can go from one website to 20 with no real troubles, or it can mean that the software has no known bottlenecks when adding five additional servers to support those 20 additional sites. I am sure you can see where I am going here: it boils down to costs. &nbsp;What does licensing look like when you have to add five additional servers, or what is the cost for the infrastructure, bandwidth, or even human capital? That idea of scale can quickly end up costing you more $$$ then you had ever planned to spend on your solution.

So find out what it means to scale. Have your solution providers demonstrate this against relevant sources, content, and data flow. As the $$$ of a deal goes up so should your proof points and their reception to those. Don&rsquo;t feel uncomfortable for asking for a protracted evaluation period. If at any point things start becoming wishy washy throw your hands up and let them know you are calling BS and give them the latitude to respond. &nbsp;After all they did just expend time, resources, and opportunity to bring you along this far.

What does it mean to be resilient?

Can you take a punch? It&rsquo;s that simple: can data move around on the page or in the source code and can the software still programmatically sniff this out? This isn&rsquo;t a &ldquo;Yes, it can!&rdquo; sort of verification process from your part. &nbsp;Much like testing for scalability, determining the resilience of a solution is something that needs to be demonstrated. Ask for this just like you would ask for a demonstration of programmatic extraction. &nbsp;After all, it is one of the hot topics in the space.

What is Intelligent web extraction?

So you have a smart thinking machine that is going to do the work for me? Tell me more&hellip; Artificial Intelligence (AI) and Machine Learning (ML) is not a myth! But does your dear vendor actually have algorithms that demonstrate an autonomous level of sophistication able to make a judgement call between a transposed Sales Price and List Price? Or better yet, what does it really mean to you on a practical level? I would hazard to bet that almost every data extraction platform all the way up to analytics stacks have some such quantity of machine learning algorithms built-in to help process the identification and flow of data. But how does that translate into real $$$ with what you are buying? Is this something your should pay more for? Are you seeing benefit from it because there is a measured increase in resilience, performance, or&hellip;? Its up to you whether or not this is just a bright shiny button that does or does not launch global thermonuclear war.

Finally, and my favorite, where does it break?

This is the fun proof point and where you really start determining if you are just another customer. A true solution seller should be walking down the aisle with you hand-in-hand and that means that your prospective vendor should not be trying to &ldquo;sell&rdquo; you anything but instead helping demonstrate and build a future with their product cozily encapsulated in yours. As such there should be no hesitation from their end, once you have earned the trust/value, to help tease apart your use-cases, sites, or workflow and point out not only where potential problems may arise but also solutions and/or workarounds to them. Unfortunately you usually only find this behavior true to form when there are some serious $$$ involved.

 
Series Note: This is the fifth in a series of publications on web data extraction. The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions feel free to contact me tom@frigginyeah.com or visit my website<a href="https://frigginyeah.com" target="_blank">&nbsp;https://frigginyeah.com</a>

The nuances of web data solutions: Scale, resilience, intelligent extraction. Understand and push vendors to demonstrate real value.

Intelligently Scale your Resilience to BS

intelligently scale your resilience to bs

Continuing on my previous post, &ldquo; Programmer,&nbsp;Data&nbsp;as&nbsp;a&nbsp;Service&nbsp;(DaaS),&nbsp;Software,&nbsp;or&nbsp;Licensing&hellip;&nbsp;what&nbsp;is&nbsp;right&nbsp;for&nbsp;my&nbsp;Web&nbsp;Data&nbsp;product? &ldquo; (<a href="http://tinyurl.com/qavxcqw" target="_blank">http://tinyurl.com/qavxcqw</a>), Let&rsquo;s explore what I believe are two main components in the decision process when choosing between DaaS and a Platform.

Is it your core?&nbsp;&ndash; Will this be part of your core product? If the service goes down or if the quality starts to suffer, will it hurt your credibility with your customers or, worse yet, your pocketbook? Do you have punitive SLAs between yourself and your vendors and expect the same to be in place upstream from your data provider? If the answer to all these is &ldquo;Yes,&rdquo; then bring the software in-house. I have never been a fan of outsourcing key components of your vertical so owning the entire pipe, source &mdash;&gt;&nbsp; post-process cleanup &mdash;&gt; consumption, will be essential in ensuring you are providing the quality service your customer(s) expects.

There is a valid argument against doing so which usually runs along the lines of &rdquo;you will never be able to use the software as well as we can, we are the experts&hellip;&rdquo;. I completely agree with it, but I propose that you look at a hybrid relationship, one where the software vendor provides the professional services in support of their platform which will be installed in your Enterprise. This can be done in parallel with bringing your own team up to speed, if eventually you want to own the proprietary building components, or you can explore contracting FTEs from the vendor or their partnership network. It will cost a little more than buying the software outright, but you get an increased level of ownership from your vendor while still coming in a little bit cheaper than an all-out DaaS model and affording yourself the comfort of owning your entire data acquisition workflowl.

Remember: Don&rsquo;t outsource your vertical! It will bite, and its not a matter of if, but when.

What&rsquo;s it going to Cost?&nbsp;&ndash; &ldquo;If cost was not an issue&hellip;&rdquo; we&rsquo;d all be cruising around on 200&prime; mega yachts sipping Dom. But seriously, cost will always be a critical component. No matter how it&rsquo;s sliced, the product or the value built off of the extracted data will come from whatever the post-process (after extraction) secret sauce is. This is true no matter if the product is for the street or internal consumption. If you find yourself confused by this, I suggest taking a hard look at your value prop because, as served, web data is commodity data with very little inherent value of its own.

But back on topic, buying a platform will normally come at a discount compared to a DaaS offering. This logic model holds when dealing with the top tier vendors and starts to become paradoxical as you move further down the ladder to less sophisticated vendors (platforms and BPOs that provide something akin to a DaaS offering without the maturity of Enterprise software). Where the product is at &ndash;be it a prototype, alpha, or general release with paying customers &ndash; should help guide the decision. Don&rsquo;t buy a platform when what&rsquo;s needed is a few thousands rows of data to help frame out your prototype; there are plenty of vendors out there who can do that for you on the cheap (sub-$500). Conversely, don&rsquo;t rely on a BPO who leverages scripts when what&rsquo;s needed is a reliable solution with high uptime and good quality data. This will end up costing a fair sum of money and I am almost positive it will never end well.

If it sounds too good to be true, then more than likely it is.

Series Note: This is the third post in a series of publications on web data extraction (#webscraping). The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions, feel free to contact me at&nbsp;tom@frigginyeah.comor visit my website:&nbsp;<a href="https://frigginyeah.com/" target="_blank">https://frigginyeah.com</a>

Decide wisely between DaaS and a platform for your web data needs. Assess if it's core to your business and understand the true cost implications. 

Two Things to Consider When Deciding Between Web Scraping Software or Data as a Service (DaaS)

two things to consider when deciding between web scraping software or data as a service (daas)

Data is fun and sometimes messy, but not today. Here&rsquo;s a pretty clean example of price monitoring competition SKU pricing on Amazon for the lowest priced seller. Does this help you making pricing decisions on your own products, maybe. Can you make those decisions without this type of data, I wouldn&rsquo;t recommend it!

Competitor price monitoring can be difficult as there are many different ways to slice the data, especially when considering Marketplaces. Do you care about the lowest seller of that product? Or how about only when Amazon is the seller? Or just the average price over all the sellers? We&#39;ve got you No matter how you want to slice the data.

Our process is pretty simple for monitoring your competitor&#39;s prices.

<ol>
	<li>Let us know the sites (Amazon, Walmart, Jet, etc..)</li>
	<li>Please provide us with the SKU&rsquo;s</li>
	<li>Sit back and enjoy the data&hellip;</li>
</ol>

Our product is an eCommerce price monitoring tool for Brands. The most common use-case is Brands trying to identify minimum advertised price violators (MAP Violators, as you will see us use sometimes). But MAP violation tracking is just a function of price monitoring. So beyond just looking for MAP violators we can also help a Brand track their competition, identify poor product listings (like ones with 2 images or less), provide all the sellers, and even custom reports like &ldquo;Tell me what products have dropped in price since our previous price scrape.&rdquo;

From there, we have a myriad of ways to provide the data to you. We can serve it up on our Web App, send you email alerts, get you hooked into our API, drop it off on an FTP, or even upload it to DropBox. Your call, we are flexible.

Check out&nbsp;<a href="https://frigginyeah.com/competitive-price-monitoring">https://frigginyeah.com/competitive-price-monitoring</a>&nbsp;for more details on how we can help.

If you are interested in learning more about MAP Policy checkout our&nbsp;<a href="https://frigginyeah.com/map-violation-monitoring">MAP Policy FAQ</a>

Finally, if you&rsquo;d like to read about&nbsp;<a href="https://frigginyeah.com/reseller-agreements">Authorized Reseller Agreements</a>, we&rsquo;ve got you covered.

Optimize your pricing strategy with comprehensive price monitoring. Track competitors, manage MAP violations, and receive tailored data. 

Price Monitoring Your Competition SKUs

price monitoring your competition skus

Have you ever thought about the fact that you might be paying someone to monitor your Brands Minimum Advertised Price (MAP) only to have them take and resell your data downstream to a reseller for repricing?

On its face is just wrong.

You come to a MAP monitoring solution because you have a problem. More than likely that problem is resellers (1P and 3P) are driving your prices down.

Then only to have a reseller tell you they are just following someone else. 
&nbsp;

To us that just doesn&rsquo;t make sense. We made what we considered an ethical decision and threw our lot in with Brands&hellip; and we are happy about it!

When you sign up with us you don&rsquo;t have to think about the fact that your MAP provider might be contributing to your violation problem by providing repricing data to resellers. Be it Amazon, Walmart, or any host of resellers that are out there.

Choosing a MAP monitoring service? Ensure they don't resell your data for reseller repricing, undermining your efforts to maintain price integrity.

Is your MAP Monitoring Solution selling your data?

is your map monitoring solution selling your data?

We get this question a lot. Sometimes, it is from a Retailer who clicked through looking for competitive pricing, which we do provide, but only to brands. Other times it is when we are at conferences advertising <a href="https://frigginyeah.com/map-violation-monitoring">MAP monitoring</a> to the Brands who happen to be in attendance. The latter of which we always receive a surprising amount of incredulity over our business position.&nbsp;

But we think it&#39;s pretty simple&hellip;&nbsp;

If we provide retailer repricing data that is used in a competitive fashion, then we are not helping the Brands that we set out to in the first place. So, a Brand ends up paying us for a service monitoring price movement that we would, in effect, be causing. It would definitely be a self-fulfilling business venture, but not one that we would feel happy about founding.&nbsp;

Was it tough making this decision?&nbsp;

Not from an ethical standpoint, no. It was pretty easy in our early days to make heavily principled decisions and cast our lot in with Brands looking for a MAP monitoring solution.&nbsp;

What is rewarding for our team is to hear feedback on our product, not just from the aspect of the data we provide, but also the emotions it elicits&hellip; &ldquo;Wow, we now have people walking around the office screaming &lsquo;FrigginYeah!&rsquo;&rdquo; from a brand that just signed up from one of our competitors.&nbsp;

Or another &ldquo;We are equally happy and upset now that we have signed up. We never knew how bad our channels were because our previous provider missed so much&hellip;&rdquo;&nbsp;

The latter is the classic &ldquo;known, unknowns&rdquo; scenario. This brand new they had a MAP problem but did not know the severity of its minimum advertised price violations and unauthorized sellers until they switched from a competitor to us.&nbsp;

Our strength lies in price scraping. With hundreds of sites currently in our library, we can access any consumer-facing product detail page and provide you with product pricing. Imagine finally hearing from a provider, &ldquo;We can do that!&rdquo;.&nbsp;

<a href="https://frigginyeah.com/contact-us">Contact us</a> today to learn more.&nbsp;

Dedicated to brands, we provide ethical MAP monitoring, not competitive repricing, ensuring brand integrity and revealing crucial market insights. 

Why we only work with Brands

It took a Pandemic to really let retailers and 3p sellers get out of control. We saw two trends happen over the past few months as it relates to inbound prospects.

March &ndash; April&nbsp;&ndash; 3p sellers and Retailers started blowing up our emails and phones. They all wanted one thing, access to more price monitoring data to be able to chase their competitors to the bottom. The theme remained the same across the board: faster, cheaper, and more frequent access to price checks.

May &ndash; June&nbsp;&ndash; Brands trying to figure out what the hell just happened to their online channels?

Price Monitoring is an interesting space in that we appeal to both sides of the equation. But appealing and partnering are two different animals. So March and April were terrible months for us after an insane finish last year and equally jubilant beginning of the year.

Why was it terrible? Because we only partner with Brands. It has never made much since to me that some of our competitors sit on the fence selling Minimum Advertised Price violation data to a Brand and then turning right around and selling that exact same MAP data to a 3p seller to do repricing.

How&rsquo;s that not talking out both sides of your mouth?

Any who I digress a bit. Another trend that has shown itself over the past few months, and this is not a shocker, is Brick &amp; Mortar retail has been irreversibly changed from sales being 50/50 between B&amp;M and eTailers to almost all sales on eTail channels.

The writing has been on the wall for years here and while I do not think traditional retail is dead it will need to carve out a new niche going forward.

<h2>Looking forward</h2>

But what does that mean for price monitoring? Well I don&rsquo;t see this need going away. eTailers are not going to magically decide to all play ball and not violate MAP. In fact I think it is going to become increasingly more competitive as eTailers compete for eyeballs, clicks, and check outs. Nothing is going to help solidify that more than them hawking your wares for way below MAP. It will be the savvy Brands who can consume pricing monitoring data and act upon that information who profit and come out the other side stronger.

Pandemic shifts in retail: Brands seek to navigate e-commerce challenges with focused MAP monitoring and strategic partnerships.

Blog

ToxicNatos keeps an e ...

Why You Should Care A ...

Top 3 Cost Drivers Fo ...

Shop.Org – It’s a wrap!

Programmer, Data as a ...

Amazon and Walmart Pr ...

Amazon & Walmart Pric ...

You Have a Product Id ...

Intelligently Scale y ...

Two Things to Conside ...

Price Monitoring Your ...

Is your MAP Monitorin ...

Why we only work with ...

3p Sellers Running Amok

Subscribe to our newsletter

Solutions

Sites We Track

Company

Resources

Contact

Community