&nbsp;

<img alt="Thrive Market Digital Shelf Analysis " src="https://cloud.frigginyeah.com/s/AZXwbs2ojFnffyt/preview" style="height:2578px; width:900px" />

Analyze the digital shelf on ThriveMarket.com with targeted monitoring for strategic product visibility and growth.

ThriveMarket.com's Digital Shelf - January 2023

I am hoping to share a little trick to get rid of some of the small sharks out there messing up your channels. Unauthorized Sellers are really a PITA and can cause a lot of damage through price erosion do to automatic repricing. Our <a href="https://frigginyeah.com/map-violation-monitoring">minimum advertised price monitoring software</a>&nbsp;routinely uncovers&nbsp;these unauthorized sellers through the course of monitoring product prices and 3p sellers. Those brands who sell on <a href="https://frigginyeah.com/amazon-price-tracking">Amazon</a> know the pain these 3p sellers cause, and the havoc they can wreak if they take over the buybox.&nbsp;

First, lets cover the basics.&nbsp;

<hr />
<h2>What is an Unauthorized Seller?&nbsp;</h2>

An unauthorized seller&nbsp;is a business or individual that is not officially recognized by the manufacturer or brand owner to sell their products online. These sellers do not have permission to sell the brand&#39;s products and are not allowed to sell new, authentic products from the brand. They may also not have access to special pricing and promotions, and may not be able to offer warranties and returns on products. Unauthorized sellers are often known as counterfeit sellers, or gray market sellers, and they often sell products that are not genuine or are not covered under the manufacturer&#39;s warranty. These sellers may sell goods at lower prices than authorized sellers, but the authenticity and quality of the products can be questionable.

<hr />
<h2>Ok, so what&#39;s the trick?</h2>

First we will start with the premise that most of your brand&#39;s unauthorized sellers on Amazon are acquiring their stock through close-outs or similar unauthorized channels. If that premise is accepted then the trick is to purchase the limited stock that is available from these gray market sellers to get them off of your product listing.&nbsp;

Another benefit is that if it is FBM (fulfilled by merchant) then you can start backing into who that unauthorized seller is.&nbsp;

<h2>Go to the Amazon product detail page and example the seller list</h2>

<img alt="Amazon Unauthorized 3P Sellers" src="https://cloud.frigginyeah.com/s/Ztww3oBQswJxYx9/preview" style="height:623px; width:500px" />

<h2>Option 1 - Add to cart on the seller popout.&nbsp;</h2>

For some sellers you will be able to see if they have anywhere from 1 - 5 available to be purchased. But, if you see &quot;5&quot; then add it to your cart and look at the below.&nbsp;

<img alt="Amazon Unauthorized 3P Sellers Add to Cart" src="https://cloud.frigginyeah.com/s/58eNNTajoHLyyTE/preview" style="height:116px; width:300px" />

<h2>Option 2 - in Cart add additional quantity</h2>

When it&#39;s in your cart you are presented with the same drop down. But here you will get an accurate representation of how many products the unauthorized seller has.&nbsp;

<img alt="Amazon Cart" src="https://cloud.frigginyeah.com/s/YAq9f7yBsPH8tJ8/preview" style="height:93px; width:300px" />

<h2>Add 100 and see what Amazon shares back of what they have in stock</h2>

<img alt="amazon a lot of available products" src="https://cloud.frigginyeah.com/s/WFgk7MFMYpf3fai/preview" style="width:300px" />

If the unauthorized seller does not have that number available you will see something like this warning message.&nbsp;

<img alt="stock availability warning message " src="https://cloud.frigginyeah.com/s/iCFq8Fi5JDzPMzs/preview" style="height:70px; width:300px" />

If they have a lot of stock then you might see something like this. Which is successfully adding the 100 to your cart.&nbsp;

<img alt="amazon in stock" src="https://cloud.frigginyeah.com/s/ACGKFWxXSfN3nnr/preview" style="height:91px; width:300px" />

It&#39;s pretty easy to buy up all the folks that have 1-3 products available. But when going through this process if you come across someone like the example above that has over 100 products in stock and ready to ship then you need to treat those sellers a bit differently. They represent a leaky distrbution channel and that will require a different process to figure out who they are and start clawing them back. We will cover that in another post.&nbsp;

&nbsp;

A clever strategy to tackle unauthorized sellers on Amazon, ensure brand integrity and prevent price erosion with FrigginYeah's monitoring software.

Unauthorized Sellers Getting You Down?

<h2>What is MAP?</h2>

MAP, or Minimum Advertised Price, refers to the lowest price a retailer is allowed to advertise a product for sale. It&#39;s a policy that manufacturers or distributors set to maintain brand identity and to ensure that retailers do not advertise or sell their products at excessively low prices, which can lead to several issues such as:

<ol>
	<li>Brand Devaluation: Constantly low prices can tarnish a brand&#39;s image, making it seem less &#39;premium&#39; or of lower quality.</li>
	<li>Unfair Competition: It helps level the playing field among retailers, preventing larger retailers from using their buying power to offer unsustainably low prices, which smaller retailers can&#39;t compete with.</li>
	<li>Maintaining Profit Margins: It ensures that all retailers have a fair margin on the products, encouraging them to carry and promote the entire range of a manufacturer&#39;s products.</li>
</ol>

While MAP only dictates the advertised price and does not tell retailers the actual selling price, it&#39;s a crucial tool for manufacturers in controlling their products&#39; positioning in the market and maintaining a healthy relationship with their retail partners. Breaking a MAP policy can lead to penalties from the manufacturer, including ceasing to supply products to the retailer in question.

<h2>What does that mean for eCommerce retailers or marketplaces?&nbsp;</h2>

If the price is below your MAP agreement or <a href="https://frigginyeah.com/reseller-agreements">Reseller Agreement</a> and is shown on the list page or on the detail page then that would be a MAP violation. How you enforce your MAP agreement will largely depend on what MAP policy you have in place between yourselves and the seller.&nbsp;

<h2>But Amazon is a <a href="https://frigginyeah.com/map-violation-monitoring">MAP violator</a>, what can I do?&nbsp;</h2>

Not much. At least as far as Amazon is concerned. Amazon will not adhere or comply with your MAP policy, even if you threaten them. Instead they will point the finger back at you and tell you that someone else is selling at or below their price.&nbsp;

Amazon uses price automation software, just like we do here at FrigginYeah, to monitor the various channels your products are sold on. So one of the first things you should do is take stock of all your channels and see if you can find out where Amazon might be repricing to and force that retailer to comply with your MAP agreement.&nbsp;

<h2>But it&rsquo;s a 3rd Party on Amazon, what can I do?&nbsp;</h2>

This is a tough one. If the 3rd party seller is violating your MAP agreement and you do not know who they are or how they are getting your product then you will need to research them and see what you can find out. A couple of options are:&nbsp;

<ul>
	<li>If it&rsquo;s Fulfilled By Merchant - buy the product and then request a return to get their return address.&nbsp;</li>
	<li>If it&rsquo;s Fulfilled by Amazon - see how many products they have in stock. Most unauthorized or gray market sellers have shallow stock and your best/easiest course of action is to buy all the stock that is available. Then you can inspect the products to see if you can find out how they got them. If you are lucky you acquire all the excess stock and your pricing normalizes. If you are not then you will need to start working backwards to figure out how they are getting access to your product.&nbsp;</li>
</ul>

<h2>The retailer says they didn&rsquo;t drop the price first, what now?&nbsp;</h2>

If it&rsquo;s Amazon violating MAP then like mentioned above you need to figure out who else has a low price and enforce your MAP policy there first.&nbsp;

If it&rsquo;s not Amazon, and your MAP policy doesn&rsquo;t specifically have some call out about first movers (most do not), then it doesn&rsquo;t matter who dropped price first. The seller has signed your MAP agreement in good faith that they would adhere to it, now they are violating your MAP. So no matter who dropped price first

<h2>Why will retailers not want to sign your MAP policy?&nbsp;</h2>

Firstly, retailers may feel that they are being restricted in their ability to price products competitively, which can limit their ability to attract customers and increase sales. They may also feel that MAP policies limit their ability to offer discounts or promotions to customers.

Secondly, retailers may feel that they are being put at a disadvantage compared to other retailers who are not bound by a MAP policy. For example, if you do not have tight distribution methods an unauthorized reseller who is not bound by a MAP policy may be able to offer lower prices than a retailer who is bound by a MAP policy, which can give the unauthorized reseller an advantage.

Thirdly, retailers may feel that they are being asked to agree to a policy that is not in their best interest. They may feel that they are being asked to accept a policy that is primarily intended to protect the interests of the manufacturer or distributor, rather than their own interests.

Finally, retailers may feel that enforcement of MAP policy is difficult to enforce and often arbitrary. Retailers may feel that they are being held to a standard that is difficult to meet, and that the enforcement of MAP policy is not consistent across different retailers.

Overall, retailers may see MAP policies as a restriction on their ability to compete in the marketplace and may be hesitant to agree to them.

This will be an ongoing&nbsp;series of posts about What MAP is and what Brands may face when trying to track, identify, and enforce their MAP policies.&nbsp;

&nbsp;

Explore MAP (Minimum Advertised Price): the lowest advertised price for products, crucial for maintaining brand value and fair competition among retailers.

What is MAP? Part 1

Monitoring Minimum Advertised Price (MAP) on Amazon is a must&nbsp;for businesses looking to protect their brand and pricing strategy. MAP is the lowest price at which a manufacturer or distributor allows a product to be advertised. By enforcing a MAP policy, businesses can ensure that their products are being advertised at fair and competitive prices, which helps to protect the integrity of their brand, fight price erosion, and reduce overall channel conflict.&nbsp;&nbsp;

There are three main ways to monitor MAP on Amazon: manual monitoring, automated monitoring with a web scraping tool, and using a <a href="https://frigginyeah.com/map-violation-monitoring">MAP monitoring service</a>. In this blog post, we will lightly explore each of these methods in more detail and discuss the pros and cons of each.

Manual Monitoring

Manual monitoring of MAP on Amazon involves manually searching for and checking the prices of your products on the platform. This can be done by manually searching for your products on Amazon and checking the prices against your established MAP policy.

The main advantage of manual monitoring is that it gives you complete control over the process. However, it also has several disadvantages. Firstly, it can be time-consuming, especially if you have a large number of products or if you have a lot of 3P sellers on Amazon. Secondly, it can be difficult to keep track of all the prices and is prone to human error, even more so if you have a lot of products or if prices change frequently. Finally, it can be quite difficult to detect when a price changes. A lot of savvy, albeit unscrupulous sellers, will make the conscious decision to violate your MAP policy after hours therefore negating your ability to catch them in the act. This in turn will cause other sellers who use automated repricing solutions to drop and thus begins the cycle of &ldquo;...we didn&rsquo;t drop price first&hellip;&rdquo;.&nbsp;

Automated Monitoring with a Web Scraping Tool&nbsp;

Automated monitoring of MAP on Amazon with a web scraping tool involves using scraping software (on-premise or a SaaS solution) to automate the process of searching (list pages)&nbsp; and checking prices (on product detail pages, if the tool or proxy it uses is worth its weight in salt).. There are several software options available that can automatically check prices on Amazon.&nbsp;

The main advantage of price monitoring with a web scraping tool is it can be cheaper to build and maintain (we will get into Total Cost of Ownership in another blog post) than using a service and it can scale better than undertaking price monitoring manually. However, it also has disadvantages. Firstly, it can initially be time consuming to configure and setup. The setup requires you to manage the web scraping process/team/platform, sometimes even build the web scrapers, as well as manage your products and the ASINs (Amazon Standard Identification Number) for each product. Secondly, you will then need to monitor the scraping performance as well as review the data for accuracy before you ever begin to start your MAP compliance process with your resellers.&nbsp; Secondly, it may not be able to detect all violations, this occurs when you have not set up the software correctly (junk in / junk out) or if the tool you are using is being blocked by Amazon. Finally, it may not track prices for new listings that appear on Amazon with different ASINs if you have not configured a proper lookup process to feed the detail page process.&nbsp;

Using a MAP Monitoring Service

Using a service&nbsp;for <a href="https://frigginyeah.com/amazon-price-tracking">Amazon MAP monitoring</a>&nbsp;involves outsourcing the process to a third-party company, like us here at FrigginYeah. Of course there are several companies that offer MAP monitoring services beyond us and those services can include monitoring, alerts, and reporting. All these services are not created equal. Some are super expensive and offer other services beyond just MAP monitoring / Price Monitoring which may get confusing and expensive (that&rsquo;s their goal). While others can be dirt cheap but fall down with scale, be it the number of products or number of websites you are looking to monitor.&nbsp;

The main advantage of using a service is that it can save time and make it easier to keep track of prices. Additionally, a service is responsible for the full lifecycle of your MAP monitoring project. So if a website changes, or fields a new blocking solution, it&rsquo;s on the service to rebuild the code to accommodate these changes. If you are currently being charged for maintenance + a subscription <a href="https://frigginyeah.com/contact-us">we should talk!</a>&nbsp; The main disadvantage is the cost of service, which can seem higher than that of a web scraping tool, or manually monitoring. However here at FrigginYeah our Amazon MAP monitoring software is an industry leader in providing quality, accurate, and on-time price monitoring for an economical price.&nbsp;

In conclusion, monitoring MAP on Amazon is essential for businesses looking to protect their brand and pricing strategy. There are several methods available, including manual monitoring, automated monitoring with a web scraping tool, and using a MAP monitoring service. Each method has its own advantages and disadvantages, and businesses should consider their specific needs and resources before choosing a method. We are always happy to chat and advise what we believe is the best solution for your business. We were consultants before we built our service and always work to ensure you are getting the right solution for your business.

&nbsp;

Protect your brand on Amazon with MAP monitoring strategies: manual, automated tools, or services. Discover the best fit for your brand's needs.

How to Monitor MAP on Amazon

The gray market, also known as the parallel market or unofficial market, refers to the buying and selling of goods through channels that are not authorized by the manufacturer, brand or trademark owner. These goods are often genuine products that have been manufactured for the legitimate market, but they are sold outside of the manufacturer&#39;s approved distribution channels.

One way that gray market goods can enter the market is through parallel imports, you may also have heard of this as product diversion, which occurs when a product is legally purchased in one country and then sold in another without the authorization of the trademark owner. This can happen when a product is in high demand and the price is lower in one country compared to another. For example, a product that is sold for a lower price in a country with a weaker currency may be imported and sold for a higher price in a country with a stronger currency.

The gray market can have a significant impact on brands and manufacturers. One of the main concerns is that gray market goods can dilute the value of a brand by undercutting the prices of legitimate products. This can lead to confusion among consumers, who may not be able to tell the difference between a gray market product and a legitimate product, and can also lead to lost sales for manufacturers and authorized distributors.&nbsp;

Another concern is that gray market goods may not be subject to the same quality control standards as legitimate products, which can lead to safety concerns and damage to a brand&#39;s reputation. This can also damage the manufacturer&#39;s image when a consumer seeks a warranty claim that cannot / will not be honored because it was purchased from a gray market seller. Additionally, gray market goods can also impact the ability of manufacturers to control their distribution networks, which can make it difficult to ensure that products are being sold in the intended markets and at the intended prices.

One strategy manufacturers can use to combat the gray market is setting and enforcing <a href="https://frigginyeah.com/map-violation-monitoring">Minimum Advertised Price (MAP) policies</a>. A MAP policy is a manufacturer&#39;s suggested minimum price for a product, and it is meant to prevent retailers from advertising a product at a price that is below the manufacturer&#39;s suggested minimum.

<a href="https://frigginyeah.com/map-faq">MAP policies</a> can help prevent retailers from undercutting the prices of legitimate products and can also help to preserve the value of a brand. However, it&#39;s important to note that MAP policies can be difficult to enforce, as retailers can still sell products at a lower price, they just can&#39;t advertise it. Additionally, some countries have regulations that can limit the ability of manufacturers to enforce these policies.

In conclusion, the gray market refers to the buying and selling of goods through channels that the manufacturer or trademark owner does not authorize. This can happen through parallel imports or the diversion of goods.The gray market can have a negative impact on brands and manufacturers as it can dilute the value of a brand, lead to safety concerns and damage reputation. One strategy to combat the gray market is setting and enforcing<a href="https://frigginyeah.com/map-violation-monitoring"> Minimum Advertised Price (MAP) policies </a>and <a href="https://frigginyeah.com/reseller-agreements">Reseller Agreements</a>. But we will be honest: dealing with the gray market is no simple task and cannot be solved with agreements alone. It requires diligence, good data, and constant pressure by the manufacturer.

Manage gray market challenges with our expertise on parallel imports and diversion. Safeguard your brand and enforce MAP with strategy. Get expert advice.

What's the Gray Market?

Enforcing a minimum advertised price (MAP) policy can be a challenging task, and brands need to approach MAP enforcement with care. Here are the top three things NOT to do when enforcing your MAP policy:

<ol>
	<li>Do not be overly aggressive or confrontational from the get go: It is important to approach <a href="https://frigginyeah.com/map-violation-monitoring">MAP enforcement</a> in a professional and respectful manner. From feedback from a lot of the top brands on our platform the majority of MAP violations are honest mistakes and will usually be quickly fixed. Do not be aggressive or confrontational with resellers, as this can damage relationships and create more problems than it solves. Introducing additional channel conflict (e.g. MAP violations + damaged reseller relationships) are not a combination for a winning scenario.&nbsp;</li>
	<li>Do not be inconsistent: It is important to be consistent in the way that you enforce your MAP policy. Do not apply different standards to different resellers, as this can create confusion, resentment, and may even legally put your brand at risk or invalidate any legal standing you have in litigation.&nbsp;</li>
	<li>Do not ignore MAP violations: Ignoring MAP violations will only make the problem worse. It is important to take swift and decisive action when a violation is discovered, in order to protect the value and reputation of your products and the future enforceability of your MAP agreement or <a href="https://frigginyeah.com/reseller-agreements">Reseller agreements</a>. This is one of the reasons we recommend brands monitor their resellers at least once a day.&nbsp;</li>
</ol>

It is important to approach MAP enforcement with care. By avoiding these three things, brands can better protect the value and reputation of their products and maintain healthy relationships with resellers.

Avoid confrontation, ensure consistent MAP enforcement, and promptly address violations to safeguard your brand's value and reputation.

Top 3 things NOT to do when enforcing your MAP agreement.

top 3 things not to do when enforcing your map agreement. 

<a href="https://frigginyeah.com/map-violation-monitoring">Minimum advertised price (MAP) monitoring</a> is a crucial aspect of ecommerce brand management, and this is especially true during times of economic recession. Here are a few reasons why MAP monitoring is important in a recession:

<ol>
	<li>
	Protecting brand reputation and customer loyalty: During a recession, consumers may be more price-sensitive and more likely to shop around for the best deal. If your brand&#39;s products are being advertised and sold at significantly discounted prices by unauthorized resellers, it can hurt the brand&#39;s reputation and lead to a loss of customer loyalty. By enforcing a MAP policy and ensuring that resellers are adhering to it, brand managers can help to protect the brand&#39;s reputation and encourage customer loyalty.
	</li>
	<li>
	Maintaining control over pricing: In a recession, it&#39;s especially important for brand managers to maintain control over pricing in order to protect the brand&#39;s bottom line. Allowing resellers to advertise and sell products at significantly discounted prices can lead to price erosion, where customers come to expect lower prices and are less likely to pay full price for the product. By enforcing a MAP policy, brand managers can help to prevent price erosion and ensure that customers are willing to pay a fair price for the brand&#39;s products.
	</li>
	<li>
	Promoting a level playing field: During a recession, it&#39;s common for businesses to become more competitive as they try to maintain profits and stay afloat. Allowing some resellers to advertise and sell products at significantly discounted prices can create an uneven playing field, with some resellers undercutting others and potentially driving them out of business. By enforcing a MAP policy, brand managers can help to level the playing field and ensure that all resellers are able to compete fairly.
	</li>
	<li>
	Ensuring compliance with MAP agreements: Many brands enter into MAP agreements with their resellers, which outline the minimum price at which the brand&#39;s products can be advertised and sold. It&#39;s especially important to enforce these agreements during a recession, as it can help to protect the brand&#39;s legal interests and avoid potential legal issues.
	</li>
	<li>
	Maintaining control over the brand&#39;s image: During a recession, it&#39;s important for brand managers to maintain control over the brand&#39;s image in order to maintain customer loyalty and drive sales. Allowing resellers to advertise and sell products at significantly discounted prices can lead to a loss of control over the brand&#39;s image, as customers may perceive the brand as being of lower quality or less desirable if they can get the same product for a much lower price from another seller. By enforcing a MAP policy, brand managers can help to maintain control over the brand&#39;s image and ensure that customers have a consistent and positive experience with the brand.
	</li>
</ol>

Overall, MAP monitoring is an important aspect of ecommerce brand management, and this is especially true during times of economic recession. By establishing and enforcing a clear and consistent MAP policy, monitoring advertised prices, and taking action when necessary, brand managers can help to protect the brand&#39;s reputation, maintain customer loyalty, promote a level playing field, prevent price erosion, ensure compliance with MAP agreements, and maintain control over the brand&#39;s image. All of these factors can ultimately help to drive sales and increase profits for the brand, even in challenging economic times.

Discover why MAP monitoring is essential for e-commerce brands during recessions: safeguard reputation, ensure fair pricing, and promote competition.

MAP Monitoring, more important than ever.

map monitoring, more important than ever. 

ThriveMarket is an online grocery store whose original mission of providing high-quality, shelf-stable products at a fraction of the price has continued to expand. This expansion has made more and more 3rd parties interested in monitoring its <a href="https://frigginyeah.com/custom-development">digital shelf</a>.&nbsp;

Now, they offer not only the existing brands they&#39;ve launched with but have continued to expand their categories through the addition of other known brands, and they have started to white label under their own branding. Furthermore, they have moved into frozen goods. However, the latter is of less interest at the moment.

We leveraged our advanced scraping technologies to automatically scrape the hundreds of brands and thousands of products that are available. We&#39;ve even pinpointed on a large chunk of products that have <a href="https://frigginyeah.com/map-violation-monitoring">Minimum Advertised Price violations</a>, but that wasn&#39;t the goal here.&nbsp;

Our plan is to continue to monitor Thrive Market month over month doing product and price scraping. We will then levreage that data and provide these free high-level reports on what we believe are interesting Thrive Market KPI&#39;s.&nbsp;

If you might be interested in more detailed data please drop us a <a href="https://frigginyeah.com/contact-us">note</a>.<img alt="" src="https://cloud.frigginyeah.com/s/J5bWGkTWQTpGLKo" />

Monitor ThriveMarket's digital shelf and MAP trends with our detailed monthly reports. Dive into data-driven insights

ThriveMarket.com's Digital Shelf - June 2022

thrivemarket.com's digital shelf - june 2022

We are still at it and in our 7th week of price monitoring back to school supplies there continues to be more interesting things that are coming to light. Using our PricingIQ product we are price tracking 24 must have back to school products products across Amazon, Walmart, Office Depot, and Target. (See the full product list here: https://www.frigginyeah.com/2017/06/30/back-to-school-shopping-with-price-monitoring-sorry-parents/)

Retailers reselling cheaper on Marketplaces

Oddly &ndash; Office Depot sells some products cheaper through the Amazon Marketplace than they do on their own website. Our price tracking shows several products that have cheaper daily pricing anywhere from 2%-8% sold by Office Depot on Amazon.&nbsp;I am not sure why this happens &ndash; but have a few theories that I want to vet before sharing.

Bad Product Pricing Data

Even more Odd &ndash; Some Marketplace sellers just have horrible data loaded for products they are hawking.

I see this as primarily an Amazon problem. The Mega Deals (<a href="https://www.amazon.com/s?me=AY2ANYVC1YIBT&amp;merchant=AY2ANYVC1YIBT&amp;qid=1501010590" target="_blank">https://www.amazon.com/s?me=AY2ANYVC1YIBT&amp;merchant=AY2ANYVC1YIBT&amp;qid=1501010590</a>) seems to be one of the worst offenders based on the data we are extracting. I am not sure what measures Amazon has in place, if any, to protect a reseller against themselves &ndash; but I do believe (anecdotally) that Walmart does a better job at policing this.

Price Tracking

We continue to see what we believe is price tracking and price matching between the marketplaces (Amazon and Walmart) across several products. Though in one case we saw Target and Office Depot get into the action, still not directly price competitive mind you, after big price drops on Amazon and Walmart. Texas Instrument&rsquo;s graphing calculator (TI-84) is a perfect example of what might fall under price matching and price tracking as well as maybe a Brand just reducing its minimum advertised price policy.

Bad Website | Unavailable Products?

Walmart seems the biggest offender here where products are listed on its list page, or as a suggested product in a twister but when you click on that product it is not available. This is not a one-off problem as Walmart will, and has been, return this page for Dixon&rsquo;s Ticonderoga 30PK of pencils for two months (https://www.walmart.com/ip/Ticonderoga-Pre-Sharpened-Pencil-HB-2-Yellow-Barrel-30-Pack), it does not work not matter how hard you try yet its prominently displayed on the site and has actually had price movements from $5.97 to $5.77 over the past two months. I guess if all else fails its always available from The Mega Deals on Amazon for $647!

&nbsp;

The Bad Website functionality and Bad Data are two challenges that make our lives difficult from a product perspective and for our customers as data consumer. The first reaction is the product is broke and we pull our hair trying to figure out what is going on only to see that the old proverb &ldquo;Junk in / Junk out&rdquo; still remains true.

We have another 30 days left running this little experiment and we are hoping to learn more. So far we have taken several points we have identified and used those to improve our product from a consumer of data angle opposed to a provider of data &ndash; its a win/win so I am hoping for a lot more over this last bit before we move on to our next data run.

7 weeks into price monitoring back-to-school supplies reveals surprising findings across Amazon, Walmart, and more. Insights and anomalies ahead.

Back to School Monitoring Week 7

back to school monitoring week 7

I was surprised to find that while looking at the product pricing data we are monitoring for MAP violations an odd occurrence.

I kept seeing products priced higher on eBay than on the Retailer&rsquo;s site. At first I thought there was probably a shipping component that I was missing, but that wasn&rsquo;t the case. Both product pages offer free shipping. So the only difference is that if you bought that product off of eBay instead of Wayfair then 6 out of the 7 days you were paying a premium.

While this is definitely not anything conclusive it&rsquo;s interesting enough to compel me to pull back the onion layers even more. So more to follow soon on that&hellip;

As a side note I am happy to report that in our price monitoring all of these products I looked at were in compliance with the Brand&rsquo;s minimum advertised price agreements! We&rsquo;ve helped reduce these MAP violations from over 1k a day down to less than around 75 daily.

Discovering higher eBay prices than retailer sites, even with free shipping. Investigating further, but our MAP monitoring shows compliance success.

Is eBay really cheaper?

is ebay really cheaper?

How&nbsp;<a href="https://www.toxicnatos.com/">ToxicNATOs</a>&nbsp;keeps an eye on its Competition.

ToxicNATOs is a small veteran-owned watch strap brand based in Colorado Springs, Co. They offer nylon, seatbelt, and leather NATO-styled straps to a small but fiercely loyal customer base. Being in a crowded market, they find it best to stay on top of their competitor brands&#39; pricing for like-kind products on the various marketplaces like Amazon and Walmart as well as niche retailer websites.

Using ToxicNATO-provided product inputs, FrigginYeah leveraged existing price monitoring integrations to price scrape various competitive brands and provide email alerts showing brands competitive to ToxicNATOS. These reports have helped optimize their product pricing as well as help identify new product opportunities.

&ldquo;It was awesome to start seeing competitive pricing data as it really allowed me to identify some areas of improvement in our catalog,&rdquo; said Terry, the owner/operator of ToxicNATOS. &ldquo;We are really looking forward to using FrigginYeah to monitor our product&#39;s <a href="https://frigginyeah.com/map-violation-monitoring">Minimum Advertised Price (MAP)</a> as we expand our distribution channels.&rdquo;

<a href="https://www.toxicnatos.com/">https://www.toxicnatos.com/</a>

ToxicNATOs, a veteran-owned watch strap brand, effectively monitors competitor pricing. With FrigginYeah's advanced price monitoring.

ToxicNatos keeps an eye on its Competition

toxicnatos keeps an eye on its competition

Why You Should Care About MAP Violations

If you&rsquo;re reading this you&rsquo;ve probably worked hard to develop a reputable online brand. Your brand has perceived customer value and you may have spent countless hours on maintenance in order to ensure balanced growth. But if you aren&rsquo;t supervising minimum advertised price (MAP) violations you are susceptible to losing massive revenue streams to unauthorized sellers, brand erosion, and perceived product value loss.

What is Minimum Advertised Price

Simply put, minimum advertised price or MAP is the lowest price a seller can advertise a good. However, minimum advertised pricing only relates to advertising and does not reflect the price at which a retailer can sell said item in their store or online shopping cart. MAP and is a form of manufacturer brand protection.

Imagine a product that has a manufacturer suggest retail price or MSRP of $500. But through an online reseller it is being advertised at $250. Your potential customers may start to think your prices are inflated. If, so they may refuse to pay MSRP for your product again. This perceived product value is representative of your brand&rsquo;s strength. If your customers will no longer pay MSRP your brand will erode.

What is a MAP Violation

A map violation is a breach in the minimum advertised price agreement between the manufacturer and reseller. The above examples demonstrate a classic MAP violation When MAP agreements are breeched your brand will be left weakened or devastated. Breaches regularly occur, especially within Amazon. This is due to Amazon not interfering with their seller&rsquo;s pricing policies and even Amazon breaking your policy directly. Unfortunately, pinpointing the sellers who are in violation and shutting them down is not an easy task. It is important that you use the correct tools to help save you time and make your case to Amazon or directly to your Sellers. It is imperative that you create MAP agreements to protect your brand.

How to Protect Your Brand

There are many eCommerce outlets that violate MAP agreements constantly. And of course, they will not take action because they do not want to disrupt their revenue streams. Monitoring your brand&rsquo;s presence on all of these outlets simultaneously requires a ton of time and effort. Fortunately, there are companies that have developed powerful software that mitigate this issue. FrigginYeah&rsquo;s platform allows users to consolidate this process and quickly monitor your product prices across whichever marketplaces or retailer you choose. This helps pinpoint and target MAP violations and shut them down. Saving you countless hours and potential revenue.

MAP Agreements

To protect your brand from any future MAP violations it is crucial that you create a MAP agreement with resellers. A MAP agreement will promote fair competition across distribution channels, defend brand identity and value, and protect your margins. Smaller retailers will also be in favor of MAP agreements because it allows them to compete fairly with larger brands.

Ultimately, MAP violation enforcement and monitoring should be taken seriously. It is important to arm your brand with the correct tools to protect, enhance, and repair your revenue streams now and in the future.

Protect your brand and revenue with MAP enforcement: avoid brand erosion and ensure fair pricing with strategic monitoring and agreements.

Why You Should Care About MAP Violations

why you should care about map violations

Continuing from my previous posts on Web Data Extraction, I&rsquo;d like to explore what I see as the main cost drivers for every Web Data Extraction project. These drivers are relevant both in a Data-as-a-Service application and a Platform play, however there are a few additional factors that come into the picture when purchasing a platform. Each of these add multiples to the cost of delivering a project, so be measured when considering what you really &ldquo;need&rdquo; versus what your available budget is.

Volume&ndash; How much data is expected? Is this millions of rows? Hundreds of thousands? Tens of thousands?

The volume of a web scraping project will be a huge physical cost driver. The higher the volume of data, the more careful a provider will need to be about how they go about harvesting. They may need to bring to bear 3rd party services, like a Proxy Provider, or leverage a more expensive IP set. Also, depending on your Frequency, they also may need to bring more infrastructure online to support your extraction. These are both challenges you would need to overcome if this project is running as a platform in your enterprise. Why? Simple. The larger your footprint on the source, the more attention you draw. The more attention you draw, the more likely they are to block you. Once you are blocked, you can&rsquo;t access the data you need without going about it in a whole new way. There are other implications that come with this as well, but that is an entirely different post!

Finally, the business of web harvesting can be murky on occasion, but this is an aspect of it that should not be. It is pretty easy to estimate the amount of data that the project hopes to produce. If you have a high quantity of sources, take a sample of them and explore the amount of data that might exist on each site. Then, multiply that across your full site list. Or, if you have just one or two high volume sources, it is more than likely that they will display the number of records that exist. If not, just take the number of records on a page and multiply that by the number of pages, or whatever similar logic exists based on the page structure.

Frequency&ndash; Will this be an every minute, hourly, daily, weekly, or monthly harvest?

Yet another large cost driver is the frequency with which sources need to be scraped. This creates a multiple when applied to either Volume or Quantity, or both! Much like volume, how frequently you extract from a source increases the likelihood of detection and blocking by that source. Plus, the more frequent an extraction, usually the more hardware it will take. So finding a sweet spot between getting the data you need and leaving a large footprint or having servers up the wazoo is key.

Quantity&ndash; How many sites are being targeted? Is it just one? Tens? Hundreds?

This is a no brainer! The more sources that are being targeted, the more it is going to cost. Why? Well, depending on the types of sources and the structure of the data, there is a very high likelihood that each source will need some extra level of customization. When talking about customization you have to think labor, and not just labor but skilled labor. I had a customer come to me not too long ago who wanted to extract data from 17,000 sources and only had $10K to spend&hellip; which basically means he was willing to spend $0.58 a source.&nbsp;Assuming only light customization is needed, say 15 minutes per source, the provider would spend roughly a little over 2-People-Years, or 4,250 hours, of effort to bring those 17,000 sources online&hellip; and this does not even address the low-quality data you should expect out of a system like this.&nbsp;&nbsp;So that shows you how much he valued the data versus what the actual cost to get that data would be.

Long story really short, we had to develop a new way to attack the same problem that did not take 15 years to pay back his original investment.

Final thought

Too keep costs down, no matter if you are building a prototype or an MVP, consider how you can do more with less. Can you do this with a small amount of data to start with? If so, then think about hiring an individual off of Upwork, or the like, for a one-time extraction&hellip; even some smaller DaaS shops will pick up one-time work if the price is right. Once you have what you need and have proved out your product, move on to a more robust solution. The same goes for Frequency and Quantity. If your use-case calls for daily, try to get by with weekly or even less frequently until you prove out your product. On the flip side, if you want to build 17,000 sites, maybe you can bite off a smaller amount until the value prop is proven. Think, is there a 20/80? Or maybe a 5/95? Or even broaden your horizons, maybe this data exists in a smaller field of sources&hellip;

Key cost drivers in Web Data Scraping: Volume, Frequency, and Quantity impact project costs. Strategic planning and scaling.

Top 3 Cost Drivers For Every Web Scraping Project

top 3 cost drivers for every web scraping project

Well it was a wrap last week and just now getting time to post about it.

This was our first event exhibiting our new price monitoring product &ndash; It was also my first event exhibiting ever. I have attended a few conferences, absentmindedly, over the years and thought to myself going into this &ldquo;How hard can it be?&rdquo;

Now as I sit down to net it all out I just have to say I have left Shop.Org with a new found respect for the logistics, sales, and marketing skills that the folks who work these events have.

A couple of my key takeaways were:

1) Proper Prior Planning &ndash; This goes from understanding the event layout and where you are located, to local services support, to showing up and setting up on the first day available (definitely not on the last day!), to product pitch prep, to taking advantage of all the great marketing/media potential that comes with this type of event. We experienced some self-inflicted wounds as well as some from the event support. However most, if not all, of those could have been prevented with proper prior planning. You can wing a lot in life but conference exhibiting I have learned is not one of those.

2) Know your event &ndash;

<ul>
	<li>We have a product that is purpose built with Brands in mind but there are a lot of analogies between Retailers and Brands when price tracking comes into play &ndash; its the same data after all. So tailoring our message from a Brand being able to price monitor for minimum advertised price violations or buybox ownership to that of a Retailer looking to do competitive price intelligence or helped us have more productive conversations.</li>
	<li>Also in this category is having an understanding of event turn-outs. As far as attendees go I was pretty disappointed in the overall amount of foot traffic for the exhibitors&hellip; and I was not alone. Ultimately we had some great quality conversations and I am sure one or two will convert, but I expected to talk to (or at least have the opportunity to) 5x the amount of people based on the event attendee forecasts. Beyond attendee forecasts how the event was structured really meant folks had to have some predefined interest in your product and peel away, not a lot of random foot traffic (lesson learned around pre-marketing for an event). We will probably still go to&nbsp;<a href="http://shop.org/">Shop.Org</a>&nbsp;2018 but it will be with eyes wide open!</li>
</ul>

3) Network, Network, Network &ndash; This may seem obvious, but sometimes theres a gap between whats obvious and how you execute. I was impressed with a lot of people that I met on the floor and its infectious to be around folks that are out in front.

4) Presentation &ndash; while there were a great number of awesome sales and marketing folks at the event. I was put off with how many folks working their booths spent the majority of time glued to their cell phone or laptop. One booth was so bad that all their folks were huddled around the bar top table in a semi-circle with their backs to the floor and I watched at least one interested party walk away when they were not acknowledged. I am glad to say that this was not a problem at Booth #2112!

All-in-all this was a great new experience for me and a little outside of my comfort zone -but that&rsquo;s exactly how it should be. Unfortunately we are little late for the Big NRF event in January (not to mention its on the tail end of my Honeymoon), but ShopTalk 2018 and IRCE 2018 here we come!

About FrigginYeah:

FrigginYeah&rsquo;s PricingIQ is a Data-as-a-Service (DaaS) platform that provides Brands and Retailers with a reliable, high-quality, and scalable brand, product content and price monitoring solution. PricingIQ is the ideal solution when trying to identify BuyBox ownership for marketplaces, dynamically identifying/tracking product catalogue changes, alerting for Minimum Advertised Price (MAP) violations, product catalogue quality scrubs, product variant (sizing, colors, etc.) tracking, and full exposure to all 3rd&nbsp;party Marketplace sellers. Our constantly expanding library of over than 150 retailer sites provide comprehensive coverage of the top online US Ecommerce retailers as well as Canada, United Kingdom, France, and Italy.

Reflecting on our first exhibit with FrigginYeah: Key lessons on planning, knowing your event, and networking. MAP Monitoring was a hot topic.

Shop.Org – It’s a wrap! 

shop.org – it’s a wrap! 

Continuing on my previous post, &ldquo;You Have a Product Idea That Includes Extracted Web Data&hellip; Now What?&ldquo; (<a href="http://tinyurl.com/pr9x5vg">http://tinyurl.com/pr9x5vg</a>), I want to move on to explore the four main ways with which you can acquire the data necessary to fuel your product.

Much like every situation in business, not all solutions for a given problem are the right ones. What you need today for a product prototype may be drastically different in terms of time and dollars invested than what you may need when your product goes live (or if you&rsquo;re lucky, not until your product is realizing some returns on your investment). It could even be that your idea cannot stand up for itself unless you go &ldquo;all-in&rdquo;, which makes what you want to do even riskier.

Below I have listed the four main ways that you can go about obtaining the web data necessary to fuel your product. How you move forward is dependent on many factors, including where you are at with your product, the total $&rsquo;s you have to invest, the amount of control and predictability you may want, and in some cases, the desire to focus on your core product while allowing others to do the commodity work of web harvesting.

Programmer&nbsp;($ &ndash; $$$) &ndash; C# / Perl / Python / PHP, and what have you to build scripts accessing source content. Your primary costs here are wages and whatever you need to keep your infrastructure up and humming along.

Pros&nbsp;

<ul>
	<li>You control your sources.</li>
	<li>Extracting from new sources can be as easy as available keyboard time.</li>
	<li>The build and maintenance process is more transparent. (This does not denote easier)</li>
	<li>Everything is in-house.</li>
</ul>

Cons&nbsp;

<ul>
	<li>Will not leverage the power of Machine Learning (ML) algorithms that some software tools have built in that can increase the resilience of an extraction.</li>
	<li>When a script breaks, more than likely hours will be spent debugging it.</li>
	<li>The likelihood of investing similar amounts of time fixing a broken script as the original build is high.</li>
	<li>Enterprise features such as scheduling, data cleaning, data transport, throttling, anonymization (proxy services), etc. will need to be cobbled together, where possible, to create a true data pipe from the web.</li>
</ul>

_

DaaS&nbsp;($$ &ndash; $$$$) &ndash; No muss, no fuss. Tell them what you want, when you want it, and in what structure; then go back to building your core product and wait for the data to start coming in. Sounds great, no? Well it&rsquo;s not always that rosy&mdash;it is web data after all&mdash;but it sure beats doing it yourself when you can afford a reputable DaaS provider. &ldquo;No Names&rdquo;, BPOs, and software companies that offer up their service teams As alluded to, this one can be all over the map; not just in regard to pricing but also in regard to reputable providers and data quality. &nbsp;Be mindful that you will truly get what you pay for when going the DaaS route.

Pros&nbsp;

<ul>
	<li>Turn on the data spigot and just sit back with your feet up.</li>
	<li>Someone else has the problem of building and maintaining access.</li>
	<li>Normally higher-end services will offer a consolidated data schema (e.g. all data from all sources will be in one format and similar data types).</li>
	<li>Staff only what you need to build your product, and you can forget about experts in web technologies, C#, JavaScript, Ajax, PHP, etc&hellip;</li>
</ul>

Cons&nbsp;

<ul>
	<li>You are not their only customer, so if &ldquo;it&rdquo; hits the proverbial fan then it may take some time to get the spigot flowing again.</li>
	<li>In a lot of cases your service is only as good as your SLA, so build in a good one!</li>
	<li>Hard to find a real, true proactive service when problems arise.</li>
	<li>No real transparency into issues, what caused them, or how they are rectified. Let&rsquo;s face it, this is still a dark art in some respects, and when things break it is usually easier to fix them then to figure out why they broke. So more often than not, you will be left with few answers and that is definitely frustrating.</li>
</ul>

_

Software&nbsp;($$ &ndash; $$$$) &ndash; Don&rsquo;t want to deal with building a fragile, cobbled-together infrastructure based solely on a programmer&rsquo;s discretion that may or may not outlast their tenure, (breathe) but want to empower your team with a tool that potentially allows the power of markup, automation, ML, enterprise scheduling, data typing, error handling, throttling, reusability, proxy services, and survivability? Then purchasing a software tool may just be the route you need to go&mdash;but don&rsquo;t expect the lightweight tools like Visual Web Ripper ($) to have the same suite of tools as a #Kapow or #Connotate.

Pros&nbsp;

<ul>
	<li>No matter where you fall in the spend spectrum, you are more than likely buying a tool that has had hundreds of thousands of hours of developer seat time.</li>
	<li>Will bring to bear advance functions like; API Connectivity, Enterprise Scheduling, Data Transformations, Error Notification, Logins, CAPTCHA Breaking Connectors, Throttling (speed up or slow down connection requests) , Anonymization Proxy Services (Think: Hide me!)</li>
	<li>Ease of Use, which requires either a lower programing skill set or none at all. But be warned, this not always a good thing!</li>
	<li>Repeatability</li>
	<li>Vendor Support: a lot of vendors in the space provide some level of no-cost support either by way of their support desk, discussion boards, or even Webinars on how to use their tools in different use-cases or against different web technologies. #Import.IO is not only one of the youngest platforms out there, but they are also one of the best in the business when it comes to supporting their user base.</li>
</ul>

Cons&nbsp;

<ul>
	<li>How much of the web do they purport to cover? Can they verify that claim? But more importantly, can they demonstrate the coverage you need for your use-case? I can&rsquo;t stress this enough: if you are buying an enterprise piece of software that will cost you $$$$ then run them through their paces. See the sites, functionality, or target source technologies in action. Ask to speak to current references who have similarities to what you want to do. Don&rsquo;t take a content aggregator reference when you are looking to build an on-demand application.</li>
	<li>Why is this a con when buying an enterprise platform? Because it&rsquo;s a risk. As much as I&rsquo;d like to sit here and tell you that web data harvesting is a science, I can&rsquo;t. Sure, it&rsquo;s based on extracting programming code, but the sheer plethora of ways information is bolted together moves this from the science realm and plants it firmly into one of Art.</li>
	<li>Support for un-harvestable, one-off sources lives in the world of &ldquo;it&rsquo;s on the road map&rdquo;&hellip;a road map that, odds have it, will be completely new, and conspicuously different, from the last time you saw it. On that same point, be sure to understand what their release schedule looks like. No harm in asking for a snapshot of the previous year or two highlighting major and minor releases with dates.</li>
</ul>

_

Licensing&nbsp;($$$$) &ndash; drinking right from the source may be just the data manna your product needs to solidify its future. This is the most costly course, but it is also the one that has the biggest rubber stamp of approval on it. That might not be all that big of a deal when you are targeting hundreds of sources, but if you are relying on one or two streams of data, this is the only way to build a viable long-term product.

Pros&nbsp;

<ul>
	<li>If you have a limited number of sources, this is option that is least likely to have the top of your funnel shut down. Huge!</li>
	<li>There could potentially be partnership opportunities with the source(s), providing a little more street cred to your product idea.</li>
</ul>

Cons&nbsp;

<ul>
	<li>Costs! Depending on the source, and where their data comes from, you could be in for a rude awakening when you hear their price tag.</li>
	<li>Quality: API data may not be the same quality as what they present on the web. This could be due to variations in structure, the data elements, or even the cleanliness of the data.</li>
	<li>Terms of Use: There will be restrictions on how, where, and with whom you can use the data&hellip;sometimes even more restrictive than what they present on their site.</li>
	<li>Throttling: They will control the spigot and deliver on a frequency that is best for them.</li>
	<li>Potentially competitive situation. They might respond to your request, post your conversation with &ldquo;we already have something like this in the works&rdquo; or something similar to that tune. Or even worse yet, they leapfrog you and go further up the value chain to completely negate any value you were going to provide in the first place</li>
</ul>

Series Note: This is the second post in a series of publications on web data extraction. The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions feel free to contact me&nbsp;<a href="mailto:tom@frigginyeah.com">tom@frigginyeah.com</a>&nbsp;or visit my website&nbsp;<a href="https://frigginyeah.com/">https://frigginyeah.com</a>

Choose the right data acquisition method, Programmer, DaaS, Software, or Licensing, to power your product effectively. 

Programmer, Data as a Service, Software, or Licensing… what is right for my Web Data based product?

programmer, data as a service, software, or licensing… what is right for my web data based product?

Hard to notice but Amazon and Office Depot were about the same pricing for days until Walmart breached that price level and as you can see Amazon started pegging its price with Walmart&rsquo;s.

We are price monitoring school products and seeing Amazon pegging its price to Walmart. It has always been interesting doing this for our customers, but now price tracking competitor prices for our own purposes has added a new angle at how we are viewing the challenge. This also has us cooking up a new internal project to start monitoring these price movements at a more substantial level.

Amazon and Walmart Price Tracking

amazon and walmart price tracking

In the process of doing MAP violation tracking for our customers we also provide all the data for products that we find a MAP violation for or not. This is the result of MAP violation monitoring being a function of price monitoring in general. What that allows our users to do is peel back more layers on changes they might be making on a marketplace (like Amazon, Walmart, Home Depot) or on their retailers (Wayfair, Hayneedle, Target, etc).

For the example I looked at one product, of several thousands, we track for one of our clients over a ~30 day time span and there are 18 different price changes, 10 by Walmart and eight by Amazon. Six of those days Amazon had the premium price whereas the bulk of the days that was held by Walmart. However, it is slightly entertaining to see that pricing on Amazon would drop by a mere penny from day-t0-day to stay in front pricing on Walmart. This actually happened over and 11 day span before both were advertising a new higher price that maintained a price parity with each other.

A couple of side notes:

<ul>
	<li>None of this pricing action actually violated our client&rsquo;s MAP policies with either marketplace. In fact they do not have MAP set for this product as you might have noticed that &ldquo;US&rdquo; and &ldquo;CA&rdquo; are grayed out, these selectors allow a user to overlap the MAP price with extracted data.</li>
	<li>Both data points on this price chart rely on Amazon and Walmart as a retailer and does not include marketplace sellers. Those sellers are price monitored as well, but are displayed slightly different. Like so:</li>
	<li>Price tracking websites like Amazon, Walmart, or Wayfair are not always about some shadowy competitor price tracking another competitor in some questionably dubious price fixing scheme. This use-case is a perfect example where a Brand is able to view their advertised product pricing on retailers and make correlations between changes they make behind the scenes on their side and how it impacts the retailers and marketplaces they sell on.</li>
	<li>Walmart and Amazon price monitoring is not easy, but finding the right partner can make it seem so at times.</li>
</ul>

Discover how MAP violation tracking offers in-depth product pricing insights across marketplaces like Amazon, Walmart, and Wayfair.

Amazon & Walmart Price Monitoring

amazon & walmart price monitoring

You have struggled with the pain of a problem ripe to be solved, or you have identified an opportunity that needs to be seized and have now conceptualized the surefire way to solve it. There is just one problem: that idea will require you to utilize data from the Web in combination with your smarts to make it valuable.

Let&rsquo;s run down the list of terms that you&rsquo;ve Googled up until this point:&nbsp;Web Scraping,&nbsp;Screen Scraping,&nbsp;Web Crawling,&nbsp;Web Extraction,&nbsp;and if you really want to church it up,&nbsp;Web Harvesting. They all really mean the same thing, which is targeting a source of data on the web, capturing it by some means, and returning it to you in a form or fashion that is acceptable. Sure, each of the tools and services that exist out there go about it in different ways that span the spectrum from brute force to intelligent precision extraction, but at the end of the day, all that should matter is that you are receiving high-quality, accurate data on the frequency and schedule you expect, and, if it is mission critical, that the company/programmer/source does not go&nbsp;*poof*&nbsp;in the night.

These &ldquo;Top 3&rdquo; considerations I have found to be critical when assessing product ideas and their risks before engaging a vendor, bringing in your team, or buying software.

Creating Your Extraction Profile&nbsp;

1)&nbsp;# of Sources&nbsp;&ndash; I will just get in front of this at the get-go. If you are planning to extract data from only one source, then your project will be in the high risk category. Some of the riskiest choices are aggregators like Expedia, Kayak, Shopzilla, and Pronto, and moving sources like Zillow, Trulia, Autotrader, Cars.com, Yelp, and Craigslist. All these sources make their money based upon their data, usually pay-for-use and advertising $&rsquo;s (sometimes they&rsquo;ll even license their data set), so they spend a lot of money to acquire and ultimately protect that data from people of our ilk. Therefore it is a risky proposition to build a viable long-term product based on data from these sources. (Read: proceed with caution.)

On the flip side, if you need hundreds or thousands of sources before your product is viable then you might just be on to something. The more sources = the higher barrier to entry for new competition&hellip; of course that means a longer runway before you product hits its MVP, maybe. My suggestion here is to survey the list of sources and stack rank them based on ROI (think: the # of records available, or even the quality of content) and come up with your 80/20, or in this case in this case the 20/80. Can you come up with 20% of the sources that will give you 80% of the data functionality you need?

When moving the scale from 1-to-100&rsquo;s, building the extraction from these sources stops being viable for a programming team and starts being a candidate to bring automation to bear. This decision should not be made in a vacuum, though, and it should include the frequency, the number of sources, their build times, break rates, and fix times &ndash; all of which I will cover in a future post.

2) Frequency &ndash; I will shamelessly proclaim that I have helped build every Web Data project variation that has ever existed, and a key component to understand before moving forward is how frequently you need the data in order to make your product idea valuable. While it might seem obvious, I will state it anyway: what it takes to extract data from 100 sources once a month can take drastically different resources than extracting from one source near-real-time on-demand. Resources can be several things: people, hardware, software, 3rd party support services, bandwidth, etc&hellip; I recommend mapping out exactly what you need to make your product viable and then build your tiers of value:frequency:costs from there.

3)&nbsp;Amount of Data&nbsp;&ndash; Finally some serious thought should be logged on the amount of data you will be extracting from every source. You should have a good idea upfront what you should expect from each source. Not only will this go a long way into projecting storage and DB configurations, it will also allow you to establish a baseline, troubleshoot if your data rates start falling below a set threshold, and even start calculating how much you can expand with your current processing power.

Bonus:&nbsp;When thinking about the&nbsp;Frequency&nbsp;and the&nbsp;Amount of Data&nbsp;you plan on extracting, just note as the time frequency increases and the amount of data you need to download continues to expand, footprint will only grow that much larger and with that you risk being identified and shut down by the source.

Finally, before moving forward with attempting to engage a vendor or purchase software, knowing your&nbsp;Extraction Profile&nbsp;will only save you time and money. If you don&rsquo;t need to build a clone of the internet, don&rsquo;t, Start small, prove your product, and expand organically.

Series Note: This is the first in a series of publications on web data extraction. The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions feel free to contact me tom@frigginyeah.com or visit my website <a href="https://frigginyeah.com/">Frigginyeah</a>

Harness web data effectively: Avoid relying on risky single sources, plan data frequency, and consider data volume. Start small and scale wisely.

You Have a Product Idea that includes extracted Web Data… Now what?

you have a product idea that includes extracted web data… now what?

Whew! It&rsquo;s been awhile &amp; loads of good things have happened over the past few months! Lots of folks needing all kinds of help with their web data solutions. With projects ranging across the board from workflow automation challenges, to product pricing intelligence platforms, and even some higher level data curation and analytical components. So needless to say, but I will any who, it&rsquo;s been fun in exciting times and with that in mind I bring you this new short blog posting.

In every space there are a few terms that are bandied about, and the world of Web Scraping is no exception. You frequently hear people chirp about their&nbsp;Scale&nbsp;and that their solution is&nbsp;Resilient&nbsp;or that there is some level of&nbsp;Intelligent&nbsp;detection and processing, but what exactly is this and does it work&hellip; or better yet, does it matter? Unfortunately the answer to all these questions is &ldquo;&hellip;it depends.&rdquo; There is just not a one-size fit all approach when it comes to buying software, data, or a service.

What does scale mean?

I won&rsquo;t sit here and posit that all platform companies (enterprise, cloud, desktop) out there are in the business of pulling the wool over your eyes, but they will definitely make truthful claims that have many meanings. &ldquo;Oh yes we can scale&hellip;&rdquo; can mean your configuration (license, servers, network) can go from one website to 20 with no real troubles, or it can mean that the software has no known bottlenecks when adding five additional servers to support those 20 additional sites. I am sure you can see where I am going here: it boils down to costs. &nbsp;What does licensing look like when you have to add five additional servers, or what is the cost for the infrastructure, bandwidth, or even human capital? That idea of scale can quickly end up costing you more $$$ then you had ever planned to spend on your solution.

So find out what it means to scale. Have your solution providers demonstrate this against relevant sources, content, and data flow. As the $$$ of a deal goes up so should your proof points and their reception to those. Don&rsquo;t feel uncomfortable for asking for a protracted evaluation period. If at any point things start becoming wishy washy throw your hands up and let them know you are calling BS and give them the latitude to respond. &nbsp;After all they did just expend time, resources, and opportunity to bring you along this far.

What does it mean to be resilient?

Can you take a punch? It&rsquo;s that simple: can data move around on the page or in the source code and can the software still programmatically sniff this out? This isn&rsquo;t a &ldquo;Yes, it can!&rdquo; sort of verification process from your part. &nbsp;Much like testing for scalability, determining the resilience of a solution is something that needs to be demonstrated. Ask for this just like you would ask for a demonstration of programmatic extraction. &nbsp;After all, it is one of the hot topics in the space.

What is Intelligent web extraction?

So you have a smart thinking machine that is going to do the work for me? Tell me more&hellip; Artificial Intelligence (AI) and Machine Learning (ML) is not a myth! But does your dear vendor actually have algorithms that demonstrate an autonomous level of sophistication able to make a judgement call between a transposed Sales Price and List Price? Or better yet, what does it really mean to you on a practical level? I would hazard to bet that almost every data extraction platform all the way up to analytics stacks have some such quantity of machine learning algorithms built-in to help process the identification and flow of data. But how does that translate into real $$$ with what you are buying? Is this something your should pay more for? Are you seeing benefit from it because there is a measured increase in resilience, performance, or&hellip;? Its up to you whether or not this is just a bright shiny button that does or does not launch global thermonuclear war.

Finally, and my favorite, where does it break?

This is the fun proof point and where you really start determining if you are just another customer. A true solution seller should be walking down the aisle with you hand-in-hand and that means that your prospective vendor should not be trying to &ldquo;sell&rdquo; you anything but instead helping demonstrate and build a future with their product cozily encapsulated in yours. As such there should be no hesitation from their end, once you have earned the trust/value, to help tease apart your use-cases, sites, or workflow and point out not only where potential problems may arise but also solutions and/or workarounds to them. Unfortunately you usually only find this behavior true to form when there are some serious $$$ involved.

 
Series Note: This is the fifth in a series of publications on web data extraction. The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions feel free to contact me tom@frigginyeah.com or visit my website<a href="https://frigginyeah.com" target="_blank">&nbsp;https://frigginyeah.com</a>

The nuances of web data solutions: Scale, resilience, intelligent extraction. Understand and push vendors to demonstrate real value.

Intelligently Scale your Resilience to BS

intelligently scale your resilience to bs

Continuing on my previous post, &ldquo; Programmer,&nbsp;Data&nbsp;as&nbsp;a&nbsp;Service&nbsp;(DaaS),&nbsp;Software,&nbsp;or&nbsp;Licensing&hellip;&nbsp;what&nbsp;is&nbsp;right&nbsp;for&nbsp;my&nbsp;Web&nbsp;Data&nbsp;product? &ldquo; (<a href="http://tinyurl.com/qavxcqw" target="_blank">http://tinyurl.com/qavxcqw</a>), Let&rsquo;s explore what I believe are two main components in the decision process when choosing between DaaS and a Platform.

Is it your core?&nbsp;&ndash; Will this be part of your core product? If the service goes down or if the quality starts to suffer, will it hurt your credibility with your customers or, worse yet, your pocketbook? Do you have punitive SLAs between yourself and your vendors and expect the same to be in place upstream from your data provider? If the answer to all these is &ldquo;Yes,&rdquo; then bring the software in-house. I have never been a fan of outsourcing key components of your vertical so owning the entire pipe, source &mdash;&gt;&nbsp; post-process cleanup &mdash;&gt; consumption, will be essential in ensuring you are providing the quality service your customer(s) expects.

There is a valid argument against doing so which usually runs along the lines of &rdquo;you will never be able to use the software as well as we can, we are the experts&hellip;&rdquo;. I completely agree with it, but I propose that you look at a hybrid relationship, one where the software vendor provides the professional services in support of their platform which will be installed in your Enterprise. This can be done in parallel with bringing your own team up to speed, if eventually you want to own the proprietary building components, or you can explore contracting FTEs from the vendor or their partnership network. It will cost a little more than buying the software outright, but you get an increased level of ownership from your vendor while still coming in a little bit cheaper than an all-out DaaS model and affording yourself the comfort of owning your entire data acquisition workflowl.

Remember: Don&rsquo;t outsource your vertical! It will bite, and its not a matter of if, but when.

What&rsquo;s it going to Cost?&nbsp;&ndash; &ldquo;If cost was not an issue&hellip;&rdquo; we&rsquo;d all be cruising around on 200&prime; mega yachts sipping Dom. But seriously, cost will always be a critical component. No matter how it&rsquo;s sliced, the product or the value built off of the extracted data will come from whatever the post-process (after extraction) secret sauce is. This is true no matter if the product is for the street or internal consumption. If you find yourself confused by this, I suggest taking a hard look at your value prop because, as served, web data is commodity data with very little inherent value of its own.

But back on topic, buying a platform will normally come at a discount compared to a DaaS offering. This logic model holds when dealing with the top tier vendors and starts to become paradoxical as you move further down the ladder to less sophisticated vendors (platforms and BPOs that provide something akin to a DaaS offering without the maturity of Enterprise software). Where the product is at &ndash;be it a prototype, alpha, or general release with paying customers &ndash; should help guide the decision. Don&rsquo;t buy a platform when what&rsquo;s needed is a few thousands rows of data to help frame out your prototype; there are plenty of vendors out there who can do that for you on the cheap (sub-$500). Conversely, don&rsquo;t rely on a BPO who leverages scripts when what&rsquo;s needed is a reliable solution with high uptime and good quality data. This will end up costing a fair sum of money and I am almost positive it will never end well.

If it sounds too good to be true, then more than likely it is.

Series Note: This is the third post in a series of publications on web data extraction (#webscraping). The goal of this series is to help shed some light on the full life cycle of web data extraction. If you have any specific questions, feel free to contact me at&nbsp;tom@frigginyeah.comor visit my website:&nbsp;<a href="https://frigginyeah.com/" target="_blank">https://frigginyeah.com</a>

Decide wisely between DaaS and a platform for your web data needs. Assess if it's core to your business and understand the true cost implications. 

Blog

ThriveMarket.com's Di ...

Unauthorized Sellers ...

What is MAP? Part 1

How to Monitor MAP on ...

What's the Gray Market?

Top 3 things NOT to d ...

MAP Monitoring, more ...

ThriveMarket.com's Di ...

Back to School Monito ...

Is eBay really cheaper?

ToxicNatos keeps an e ...

Why You Should Care A ...

Top 3 Cost Drivers Fo ...

Shop.Org – It’s a wrap!

Programmer, Data as a ...

Amazon and Walmart Pr ...

Amazon & Walmart Pric ...

You Have a Product Id ...

Intelligently Scale y ...

Two Things to Conside ...

Subscribe to our newsletter

Solutions

Sites We Track

Company

Resources

Contact

Community