Robots.txt and SEO: What you need to know in 2026
A practical look at modern robots.txt use, from allow and disallow logic to wildcards, crawl-rate control and avoiding common pitfalls.
Contributor
Elmer Boutin's latest articles
The agentic web is here: Why NLWeb makes schema your greatest SEO asset
Microsoft’s NLWeb bridges websites and AI agents. Learn how to make your schema work harder — powering smarter discovery and visibility.
How to create product demos that convert and differentiate your brand
Great demos require empathy, practice, and competitive clarity. See how to turn presentations into powerful growth assets.
Your brand in the age of generative search: How to show up and be cited
In the age of AI search, visibility depends on citations, not just rankings. Here's how to make your brand trusted, structured, and seen.
WordPress pages or posts: Which should you use?
Confused about WordPress content types? Find out when to use Pages or Posts to create a site that's easy to manage and SEO-friendly.
Optimizing for AI search: Why classic SEO principles still apply
AI-powered search is reshaping SEO, but traditional tactics still hold power. Here's how to optimize for both AI and search engines.
Why branding matters for in-house SEO teams
Your SEO team’s brand can impact collaboration and executive support. Find out how to create a valuable team brand that stands out.
How should SEO change in the new AI age?
SEO is evolving as it always has, maybe a little faster lately. Learn to leverage AI appropriately, and you can stay ahead of the game.
International SEO: How to avoid common translation and localization pitfalls
Are you doing business in more than one language or more than one country? If you are, you'll want to avoid these common pitfalls.
JavaScript rendering and indexing: Cautionary tales and how to avoid them
The results of a JavaScript rendering and indexing experiment highlight some challenges of running JS-dependent content.
Marketing leadership: The dangers of ego
Confidence is an important leadership trait. But when it crosses egotistical behavior, it harms the team and limits leadership effectiveness.
A 3-phased approach to proactive online reputation management
It's better to take steps to build a better online reputation now instead of waiting for a crisis to blow up your business. Here’s how.
Bad website design: Bad for SEO, UX and business
Are these design elements slowing your download times and keeping your website from doing what you need it to do: earn you more business?
5 ways content helps create a better user experience
If you give users a good content experience, they will stay longer – and come back when they’re ready to buy.
Marketing leadership: 4 lessons learned from military service
Here are four valuable lessons learned from the military that you can use to help lead your team to success.
4 ways to own your own career path in digital marketing
If you find yourself going with the flow or doing a lot of hoping for your career, here’s how to start taking charge today.