CXL Scholarship — 8th Week Review

It is officially the 8th week of my program. I have been there and can tell you it was huge! Quantity- and Quality-wise. Are you ready to read about it? Let’s commence!

14- Google Analytics Audit by Fred Pike

• This seems to be a great lesson for me, because it is talking about developer tools in Google Chrome and I like it a lot!
• Right in the beginning, the tutor started talking about chrome extensions that are specifically made for tracking GA and GTM. One of them which is very powerful is the one that I got to know in this program. It is named as Adswerve DataLayer Inspector+ and it is awesome. The remaining ones are Google Tag Assistant, GA Debugger, and GA/GTM Debug.
• One of the things that has always been a question for me and I haven’t looked for its answer yet, is the difference between kinds of tracking codes. The problem is that I have never really understood what is ga.js, gtag.js, analytics.js, and other methods if there are any.
• Fred just talked about how to find which version a particular site is using, but didn’t mention the difference between them.
• You know, I have found something that seems to be a rule of thumb: anybody who knows something about Google Analytics or Google Tag Manager is somehow respecting Simo Ahava as a Grand Master! That’s awesome that CXL has courses that are taught by Simo.
• I watched a video on the course that was talking about checking every setting in Google Analytics account, property, and view. It wasn’t that bad but wasn’t very good either.
• One good point is that I thought you shouldn’t mess with Raw Data view at all. But now I see that it should have access to data sources like Google Search Console, Google Ads and Site Search as well. Thank you Fred for teaching me this great lesson.
• Other parts of the video were about filters, alerts and custom dimensions. Fred again referenced Simo Ahava’s article on defining custom dimensions in GA and reminded me about what a great man Simo is.
• When Fred is trying to audit a website, he uses a model that he said has borrowed from the book of “7 habits of highly effective people”. He categorizes every issue as being important or not and urgent or not. This is simple and highly effective. I liked it too.
• He uses a good technique which is auditing other websites using the DataLayer Inspector+. This is actually very fun and exciting.
• Working on problems like overcounting or undercounting is another thing that I learned. A very common mistake that a lot of analysts might make is using GA and GTM tracking codes simultaneously on a website.
• I seriously like to know more about Single Page Apps (SPAs) and how to track their performance.
• The next topic was about hostname filters and how to verify them. Truth be told, I didn’t understand what was the point and how should I verify that it is being reported correctly. The only thing that I got was that in the GA/GTM debug, you can see the hostname as dh (domain hostname). Just this one and nothing more.
• I thought that excluding IP filters in GA is one of the easiest things in the world. But according to Fred, there are a lot of ways of getting it wrong in the GA.
• He used Google Tag Assistant to record a session and verified if it works or not. It was very fun to look at a recorded session and see if the internal IPs are actually blocked or not.

Fred Pike

• Using a simple RegEx rule, a range of IP addresses can simply be filtered in GA. I just learned that. One pro tip: RegEx rules won’t work if you are using a predefined section in setting filters, You have to instead use the custom settings to apply RegEx rules.
• Oh, I forget to say this: if you have a ton of organic or paid search and don’t want your data to be sampled due to its large amount, you can have a separate view for each of them. You just need to assign different sources and mediums to these views in GA.
• How do you make sure that your incoming traffic is being precisely put in the right bucket? You most certainly have seen default channel grouping in the google analytics acquisition report. I had this question for a long time: how does Google put different types of traffic in its relevant bucket?
• Fred just answered my question. GA uses the source and medium of the traffic as the main determinators. If they have been seen correctly, the traffic can be categorized in the correct bucket. If it isn’t as you intend them to be, you have to use UTM parameters to tag your traffic correctly.
• Google Analytics has a setting in the view section. Using channel settings, you can simply adjust default channel groupings based on the guidance that google has officially made accessible to everyone.
• What do you think is the relationship between the site crawl and this course? Well, I have always thought that crawling is related to SEO, especially when it comes to Screaming Frog. But this time, I was mistaken.
• Fred used the screaming frog tool to audit a website and searched for different analytics tags. It turned out that the tool can also be useful in these kinds of instances and it was very joyful to know that. Man, you can also audit internal UTMs (which are false all the time).
• So, we’ve come across content grouping. You can group your content pages in specific groups and identify them accordingly. If you do so, you can look at the big picture more efficiently. This one should be audited.
• Another thing is related to URLs. Trailing slash is the one that comes at the end of the web URL and is better not to be there. You can remove them using filters. You can also change uppercase letters in the URL to lowercase so that it doesn’t mess the data you have collected.
• Query parameters are the ones that come after the question mark in the URLs. some browsers block these parameters in the first place, so you have to know this.
• The next part was about auditing events in google analytics. As Fred truthfully said, events are the heart and soul of the analytics and they should be tracked with a bit of caution.
• Events can easily be set up misleadingly and you have to know how to use the event category, event action, and event label like a pro. The less you put diversity in your naming, the better it generally is.
• I now know that ec, ea, and el are acronyms and they refer to three main dimensions of events. Excellent!
• We have been talking about events, how can we not talk about goals? These two are really important things to master. Because they measure the real value of a website.
• Just like events, you have to check if they are being used, are they meaningful and if there are too many or too few of those.
• I learned another great thing from Fred Pike. that you have to be friends with events and use them all the time. But don’t mess too much with goals. Whatever change you apply to the goals, they can’t be undone. You can just repurpose them in the future, but the previous data remains there. It is better to use goals just to track the most important actions on your site.
• Personally Identifiable Information or simply put, PII is something dangerous in google analytics. You should be wary about the fact that collecting PII is against google’s rules and if it catches your hand in collecting this kind of info, your account will be deleted. How dangerous is that?
• You have to be aware that storing any kind of name, email address, social security number or things like them is against google’s rules and generally accepted the privacy rules. Try to distance yourself from collecting this kind of intelligence.
• The next part was about enhanced eCommerce (EEC), but not about how to set it up correctly. Instead, it was about how to check if everything is being tracked and measured in the correct way. Fred used a lot of his chrome extensions to check if all major event attributes are set for eCommerce sites.

15- How to Run Tests by Peep Laja

• Peep started this lesson with a strong statement. He first said you should never do something to please your boss or stakeholder. Anybody who forces you to do otherwise is full of herself.
• The next thing was about A/B testing and Multivariate testing. We actually have A/B/C/D/… testing. The big point was that if you have less than 30K traffic websites, don’t think about anything big and stick to the A/B testing thing.
• The next thing is all about prioritization. When you are optimizing or trying to optimize a website, it is better to categorize issues in different buckets.
• After giving a score to different issues, you will understand what needs to be done and what needs to be done prior to others.
• You can use wireframes to show others what is on your mind. Using wireframes should be fast and easy. But if you see that others don’t understand what you want to show them or insist on seeing the final design, you’d better not waste your time and energy.
• Peep says it is better to use the tool Balsamiq and use the desktop version instead of the web version because it is way faster.
• When you want to do A/B testing, you have to have enough sample size and you need the test to run for a good amount of time.
• Don’t stop the test once you see one of the variations seems to be the winner. Give it time so that the data is fully cooked and can be served.
• One thing that was mentioned here was statistical significance. This term compares the results of the control version and variations to see which one has the upper hand. The kind that can be easily witnessed.

Peep Laja

• Always track the money. Look for ways to affect the bottom line of the company. Even if you are tracking the engagement, go after those that will finally bring money to the bank account. When you are doing an A/B test, test the funnel steps that are important from this point of view.
• Always consider the element of seasonality in your test. Never stop a test after a few days, just because you think one variation is the obvious winner. Users show different behaviors on different days of the week. So always take this into account that each test has to run for at least a week. But never stop a test unless you reach a confidence level.
• You might need to run multiple A/B tests simultaneously. In these cases, if the pages might overlap or be parts of a funnel, make sure that the traffic has been evenly divided.
• If you use Optimizely, VWO, or Monetate, you need to send your data to google analytics. It is always better to look at your data in two places and check if they match up or not.
• You are running a test with one control version and multiple variations. After running the test for a few days, you see that one of the variations is performing much lower than the others. The matter of fact is that you shouldn’t stop that variation, because it might skew your data. If you want to get rid of it, first stop the test completely, remove the variation, and then start the whole test from the beginning.
• The next topic is about a concept that everyone seems to know about, but in my opinion, nobody really knows what it is all about. I am talking about buyer persona, avatar, or whatever you name them. It is so scary to me that I really don’t know how to come up with them in the first place. It seems to be in the hands of a small group of elites.
• Peep says that we have two things: a customer theory and a buyer persona. The theory part is about what we think we know about the customer. It also consists of our learnings about what has worked in the past, what hasn’t, and what can be done. Actually buyer persona is a subset of customer theory.
• Customer theory needs to change after every test. You learn new things about your customers and change your assumptions about the customers.
• Forming a good hypothesis is rather hard. You need to know what is the problem, what you think is the solution for that problem and after that, how are you going to measure the results.
• A good and educated hypothesis will lead you through the test. The better the hypothesis is, the better the results would be and consequently, you will learn more after the test. All of these will lead you to form a better hypothesis again and this loop continues itself.
• When there are a lot of problems that need addressing, what will you do? How do you prioritize these problems and how will you find which one is the most important one? Peep introduced three models for prioritization. The PIE (Potential, Importance, Ease) and ICE (Impact, Confidence, Ease) models were almost the same. But we had another model that is developed by CXL itself which is named as PXL.
• A lot of things might skew your test results. For example we have something called history effect. In the history effect, other elements outside of your control will have an impact on your results. For example a part of your business is related to operations in the middle east, some weird country like Iran decides to run an atomic missile program and the whole world goes crazy about it!
• Another thing is the instrumentation effect. You choose a test to run, but you don’t check the cross device or cross browser. The test might not work as planned on a specific device or browser and the whole thing will blow up!
• The next thing is selection effect. You run a ppc campaign and get good results with it. After spending all of your ppc budget, you think that kind of result is also achievable using SEO or social media. But it doesn’t work that way.
• In running an A/B test, there is some common error known as the Flicker Effect. It happens when the user sees the control version before the variation for a few seconds. This is bad news for you, because it will open your hand before the users’ eyes. There are a lot of factors that can have an effect on the flicker effect.
• If you want to implement the A/B test tag using tag managers like GTM, don’t do this. This is a bad practice and is prone to result in a flicker effect.

16- Testing Strategies by Peep Laja

• So, what should I test? What should I change first and then measure the effectiveness? This is a big and also important question that you need an answer to.
• You know, you have to start the whole testing thing after a hypothesis. You have to know what is wrong or what might be wrong and then try to be creative and think about possible solutions to that problem.
• If your landing page doesn’t convert enough traffic, maybe it is because of a weak CTA or a long page. Maybe the users get confused and can’t find her path on the page. Maybe your imagery isn’t good enough. You have to guess different things based on the problem that you know is there.
• Well, I am officially confused. Peep says if you want to run A/B tests, you’d better not only change one thing on the page unless you have a website with millions of visitors. Is this rational?
• He says you can change multiple items together and then look if the changes made an impact or not. If it did, then you can go back and turn off those changes one by one to see which one actually did the job.
• I have always thought that in A/B testing, we are only allowed to change one thing at a time, because only then we can attribute the results to the change. But I think Peep said this because startups usually are short in time and don’t have the luxury of time bi enterprises possess.
• Should I use A/B test or MVT? This is heavily dependent on the traffic volume you have. If you have a low traffic website, start with A/B tests and stick with it. But if you have a decent amount of traffic (>100,000 / month), you can think about MVT.
• Most agencies run 1 MVT after every 10 A/B tests. This isn’t a general rule, but it can give you a hint.
• I just learned about a notion and I just love it. It appears that we have something called a Bandit test in some testing tools that allow us to dynamically allocate bigger traffic to the variation that seems to be performing better.
• In bandit tests, you don’t need to personally supervise the process. It is based on constant learning and implementation and the outlier will be awarded with more traffic automatically.
• You might have a bunch of sections on your website or on a specific landing page. By using Existence Testing, you are going to test if a section is necessary to be or not. You simply delete that section off the page and run an A/B test. If the conversion rate didn’t change, it simply means that the specific section didn’t play any roles on the page.
• You know what? Peep talked about two concepts that I didn’t get very much. One was Iterative testing and the other was Innovative testing. All of those concepts seemed very like the ones that I had previously known here in this course.
• Right at the last part of the course which normally doesn’t convey new messages, Peep introduced a new concept called Split Path testing. In this method, you take the user to a completely different path. You can suppose that the user has gone through a whole different funnel and the designer intends to find out which path leads to more conversions.

17- Statistics for A/B Testing by Georgi Georgiev

• This lesson is all about statistics, at least based on the preview I just watched. The tutor has a new accent and you have to focus to understand what he is saying, but he is likable.
• He says marketers just look at the numbers and don’t know what to do with them. For example they don’t know how to do risk management or making an estimation. So are we going to get some data-related knowledge? I hope so.
• To be honest, I didn’t understand what the guys talked about. He said that data needs to be used for making business decisions and this seemed understandable to me. But after that, All those mean, deviations, p-value and stuff came in and I couldn’t take my eyes off the subtitles to understand what he was talking about.
• He also talked about the difference between correlation and causation. I previously knew about this topic and hence, I could barely understand the topic.
• He promised that everything is going to be meaningful in the next lessons. He’d better be right, because this course has a final exam! Scary!
• The next part wasn’t any different. He tried to explain something called z-score and p-value using mathematical formulas. But I swear to god I didn’t understand a single one of them.
• The Only thing that I understood was the relationship between p-value and statistical significance. The smaller the p-value is, the bigger or better the statistical significance will be.
• He also talkd about something called statistical intervals and I didn’t understand it either. No surprises!

Giorgi Giorgiev

• You can’t believe how boring and difficult it was going through this course. I can definitely tell you it was the worst part of my experience with this whole program. I just wanted to finish it.
• Believe me or not, I didn’t learn anything. It is better to tell you that it was kind of not understandable at all! So as a sign of objection, I decided no to write any points of it down.

Wrap up

I don’t know what to say. It is a saying that people tend to evaluate their experience based on the last thing they have had. So I don’t want to fall into that trap. I know that the last thing I watched was terrible in quality and ruined my experience, But now I am over it. It is gone and there are a lot of things to come. I am eager to learn more practical stuff.

Let’s move on.